The SO_REUSEPORT option allows sockets to listen on the same port and to
accept connections evenly. However, there is a defect in the current
implementation[1]. When a SYN packet is received, the connection is tied to
a listening socket. Accordingly, when the listener is closed, in-flight
requests during the three-way handshake and child sockets in the accept
queue are dropped even if other listeners on the same port could accept
such connections.
This situation can happen when various server management tools restart
server (such as nginx) processes. For instance, when we change nginx
configurations and restart it, it spins up new workers that respect the new
configuration and closes all listeners on the old workers, resulting in the
in-flight ACK of 3WHS is responded by RST.
The SO_REUSEPORT option is excellent to improve scalability. On the other
hand, as a trade-off, users have to know deeply how the kernel handles SYN
packets and implement connection draining by eBPF[2]:
1. Stop routing SYN packets to the listener by eBPF.
2. Wait for all timers to expire to complete requests
3. Accept connections until EAGAIN, then close the listener.
or
1. Start counting SYN packets and accept syscalls using eBPF map.
2. Stop routing SYN packets.
3. Accept connections up to the count, then close the listener.
In either way, we cannot close a listener immediately. However, ideally,
the application need not drain the not yet accepted sockets because 3WHS
and tying a connection to a listener are just the kernel behaviour. The
root cause is within the kernel, so the issue should be addressed in kernel
space and should not be visible to user space. This patchset fixes it so
that users need not take care of kernel implementation and connection
draining. With this patchset, the kernel redistributes requests and
connections from a listener to others in the same reuseport group at/after
close() or shutdown() syscalls.
Although some software does connection draining, there are still merits in
migration. For some security reasons such as replacing TLS certificates, we
may want to apply new settings as soon as possible and/or we may not be
able to wait for connection draining. The sockets in the accept queue have
not started application sessions yet. So, if we do not drain such sockets,
they can be handled by the newer listeners and could have a longer
lifetime. It is difficult to drain all connections in every case, but we
can decrease such aborted connections by migration. In that sense,
migration is always better than draining.
Moreover, auto-migration simplifies userspace logic and also works well in
a case where we cannot modify and build a server program to implement the
workaround.
Note that the source and destination listeners MUST have the same settings
at the socket API level; otherwise, applications may face inconsistency and
cause errors. In such a case, we have to use eBPF program to select a
specific listener or to cancel migration.
Link:
[1] The SO_REUSEPORT socket option
https://lwn.net/Articles/542629/
[2] Re: [PATCH 1/1] net: Add SO_REUSEPORT_LISTEN_OFF socket option as drain mode
https://lore.kernel.org/netdev/1458828813.10868.65.camel@edumazet-glaptop3.roam.corp.google.com/
Changelog:
v2:
* Do not save closed sockets in socks[]
* Revert 607904c357c61adf20b8fd18af765e501d61a385
* Extract inet_csk_reqsk_queue_migrate() into a single patch
* Change the spin_lock order to avoid lockdep warning
* Add static to __reuseport_select_sock
* Use refcount_inc_not_zero() in reuseport_select_migrated_sock()
* Set the default attach type in bpf_prog_load_check_attach()
* Define new proto of BPF_FUNC_get_socket_cookie
* Fix test to be compiled successfully
* Update commit messages
v1:
https://lore.kernel.org/netdev/20201201144418.35045-1-kuniyu@amazon.co.jp/
* Remove the sysctl option
* Enable migration if eBPF progam is not attached
* Add expected_attach_type to check if eBPF program can migrate sockets
* Add a field to tell migration type to eBPF program
* Support BPF_FUNC_get_socket_cookie to get the cookie of sk
* Allocate an empty skb if skb is NULL
* Pass req_to_sk(req)->sk_hash because listener's hash is zero
* Update commit messages and coverletter
RFC:
https://lore.kernel.org/netdev/20201117094023.3685-1-kuniyu@amazon.co.jp/
Kuniyuki Iwashima (13):
tcp: Allow TCP_CLOSE sockets to hold the reuseport group.
bpf: Define migration types for SO_REUSEPORT.
Revert "locking/spinlocks: Remove the unused spin_lock_bh_nested()
API"
tcp: Introduce inet_csk_reqsk_queue_migrate().
tcp: Set the new listener to migrated TFO requests.
tcp: Migrate TCP_ESTABLISHED/TCP_SYN_RECV sockets in accept queues.
tcp: Migrate TCP_NEW_SYN_RECV requests.
bpf: Introduce two attach types for BPF_PROG_TYPE_SK_REUSEPORT.
libbpf: Set expected_attach_type for BPF_PROG_TYPE_SK_REUSEPORT.
bpf: Add migration to sk_reuseport_(kern|md).
bpf: Support BPF_FUNC_get_socket_cookie() for
BPF_PROG_TYPE_SK_REUSEPORT.
bpf: Call bpf_run_sk_reuseport() for socket migration.
bpf: Test BPF_SK_REUSEPORT_SELECT_OR_MIGRATE.
include/linux/bpf.h | 1 +
include/linux/filter.h | 4 +-
include/linux/spinlock.h | 8 +
include/linux/spinlock_api_smp.h | 2 +
include/linux/spinlock_api_up.h | 1 +
include/net/inet_connection_sock.h | 12 ++
include/net/request_sock.h | 13 ++
include/net/sock_reuseport.h | 15 +-
include/uapi/linux/bpf.h | 25 +++
kernel/bpf/syscall.c | 13 ++
kernel/locking/spinlock.c | 8 +
net/core/filter.c | 56 +++++-
net/core/sock_reuseport.c | 96 +++++++---
net/ipv4/inet_connection_sock.c | 99 +++++++++-
net/ipv4/inet_hashtables.c | 9 +-
net/ipv4/tcp_ipv4.c | 9 +-
net/ipv6/tcp_ipv6.c | 9 +-
tools/include/uapi/linux/bpf.h | 25 +++
tools/lib/bpf/libbpf.c | 5 +-
.../bpf/prog_tests/select_reuseport_migrate.c | 173 ++++++++++++++++++
.../bpf/progs/test_select_reuseport_migrate.c | 53 ++++++
21 files changed, 590 insertions(+), 46 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/select_reuseport_migrate.c
create mode 100644 tools/testing/selftests/bpf/progs/test_select_reuseport_migrate.c
--
2.17.2 (Apple Git-113)
This patch is a preparation patch to migrate incoming connections in the
later commits and adds a field (num_closed_socks) to the struct
sock_reuseport to allow TCP_CLOSE sockets to access to the reuseport group.
When we close a listening socket, to migrate its connections to another
listener in the same reuseport group, we have to handle two kinds of child
sockets. One is that a listening socket has a reference to, and the other
is not.
The former is the TCP_ESTABLISHED/TCP_SYN_RECV sockets, and they are in the
accept queue of their listening socket. So, we can pop them out and push
them into another listener's queue at close() or shutdown() syscalls. On
the other hand, the latter, the TCP_NEW_SYN_RECV socket is during the
three-way handshake and not in the accept queue. Thus, we cannot access
such sockets at close() or shutdown() syscalls. Accordingly, we have to
migrate immature sockets after their listening socket has been closed.
Currently, if their listening socket has been closed, TCP_NEW_SYN_RECV
sockets are freed at receiving the final ACK or retransmitting SYN+ACKs. At
that time, if we could select a new listener from the same reuseport group,
no connection would be aborted. However, it is impossible because
reuseport_detach_sock() sets NULL to sk_reuseport_cb and forbids access to
the reuseport group from closed sockets.
This patch allows TCP_CLOSE sockets to hold sk_reuseport_cb while any child
socket references to them. The point is that reuseport_detach_sock() is
called twice from inet_unhash() and sk_destruct(). At first, it decrements
num_socks and increments num_closed_socks. Later, when all migrated
connections are accepted, it decrements num_closed_socks and sets NULL to
sk_reuseport_cb.
By this change, closed sockets can keep sk_reuseport_cb until all child
requests have been freed or accepted. Consequently calling listen() after
shutdown() can cause EADDRINUSE or EBUSY in reuseport_add_sock() or
inet_csk_bind_conflict() which expect that such sockets should not have the
reuseport group. Therefore, this patch also loosens such validation rules
so that the socket can listen again if it has the same reuseport group with
other listening sockets.
Reviewed-by: Benjamin Herrenschmidt <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
include/net/sock_reuseport.h | 5 +++--
net/core/sock_reuseport.c | 39 +++++++++++++++++++++++----------
net/ipv4/inet_connection_sock.c | 7 ++++--
3 files changed, 35 insertions(+), 16 deletions(-)
@@ -13,8 +13,9 @@ extern spinlock_t reuseport_lock;structsock_reuseport{structrcu_headrcu;-u16max_socks;/* length of socks */-u16num_socks;/* elements in socks */+u16max_socks;/* length of socks */+u16num_socks;/* elements in socks */+u16num_closed_socks;/* closed elements in socks *//* The last synq overflow event timestamp of this*reuse->socks[]group.*/
As noted in the preceding commit, there are two migration types. In
addition to that, the kernel will run the same eBPF program to select a
listener for SYN packets.
This patch defines three types to signal the kernel and the eBPF program if
it is receiving a new request or migrating ESTABLISHED/SYN_RECV sockets in
the accept queue or NEW_SYN_RECV socket during 3WHS.
Signed-off-by: Kuniyuki Iwashima <redacted>
---
include/uapi/linux/bpf.h | 14 ++++++++++++++
tools/include/uapi/linux/bpf.h | 14 ++++++++++++++
2 files changed, 28 insertions(+)
@@ -22,6 +22,8 @@ int in_lock_functions(unsigned long addr);void__lockfunc_raw_spin_lock(raw_spinlock_t*lock)__acquires(lock);void__lockfunc_raw_spin_lock_nested(raw_spinlock_t*lock,intsubclass)__acquires(lock);+void__lockfunc_raw_spin_lock_bh_nested(raw_spinlock_t*lock,intsubclass)+__acquires(lock);void__lockfunc_raw_spin_lock_nest_lock(raw_spinlock_t*lock,structlockdep_map*map)__acquires(lock);
This patch defines a new function to migrate ESTABLISHED/SYN_RECV sockets.
Listening sockets hold incoming connections as a linked list of struct
request_sock in the accept queue, and each request has reference to its
full socket and listener. In inet_csk_reqsk_queue_migrate(), we only unlink
the requests from the closing listener's queue and relink them to the head
of the new listener's queue. We do not process each request and its
reference to the listener, so the migration completes in O(1) time
complexity.
Moreover, if TFO requests caused RST before 3WHS has completed, they are
held in the listener's TFO queue to prevent DDoS attack. Thus, we also
migrate the requests in the TFO queue in the same way.
After 3WHS has completed, there are three access patterns to incoming
sockets:
(1) access to the full socket instead of request_sock
(2) access to request_sock from access queue
(3) access to request_sock from TFO queue
In the first case, the full socket does not have a reference to its request
socket and listener, so we do not need the correct listener set in the
request socket. In the second case, we always have the correct listener and
currently do not use req->rsk_listener. However, in the third case of
TCP_SYN_RECV sockets, we take special care in the next commit.
Reviewed-by: Benjamin Herrenschmidt <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
include/net/inet_connection_sock.h | 1 +
net/ipv4/inet_connection_sock.c | 68 ++++++++++++++++++++++++++++++
2 files changed, 69 insertions(+)
@@ -992,6 +992,74 @@ struct sock *inet_csk_reqsk_queue_add(struct sock *sk,}EXPORT_SYMBOL(inet_csk_reqsk_queue_add);+voidinet_csk_reqsk_queue_migrate(structsock*sk,structsock*nsk)+{+structrequest_sock_queue*old_accept_queue,*new_accept_queue;+structfastopen_queue*old_fastopenq,*new_fastopenq;+spinlock_t*l1,*l2,*l3,*l4;++old_accept_queue=&inet_csk(sk)->icsk_accept_queue;+new_accept_queue=&inet_csk(nsk)->icsk_accept_queue;+old_fastopenq=&old_accept_queue->fastopenq;+new_fastopenq=&new_accept_queue->fastopenq;++l1=&old_accept_queue->rskq_lock;+l2=&new_accept_queue->rskq_lock;+l3=&old_fastopenq->lock;+l4=&new_fastopenq->lock;++/* sk is never selected as the new listener from reuse->socks[],+*soinversiondeadlockdoesnothappenhere,+*butchangetheordertoavoidthewarningoflockdep.+*/+if(sk<nsk){+swap(l1,l2);+swap(l3,l4);+}++spin_lock(l1);+spin_lock_nested(l2,SINGLE_DEPTH_NESTING);++if(old_accept_queue->rskq_accept_head){+if(new_accept_queue->rskq_accept_head)+old_accept_queue->rskq_accept_tail->dl_next=+new_accept_queue->rskq_accept_head;+else+new_accept_queue->rskq_accept_tail=old_accept_queue->rskq_accept_tail;++new_accept_queue->rskq_accept_head=old_accept_queue->rskq_accept_head;+old_accept_queue->rskq_accept_head=NULL;+old_accept_queue->rskq_accept_tail=NULL;++WRITE_ONCE(nsk->sk_ack_backlog,nsk->sk_ack_backlog+sk->sk_ack_backlog);+WRITE_ONCE(sk->sk_ack_backlog,0);+}++spin_unlock(l2);+spin_unlock(l1);++spin_lock_bh(l3);+spin_lock_bh_nested(l4,SINGLE_DEPTH_NESTING);++new_fastopenq->qlen+=old_fastopenq->qlen;+old_fastopenq->qlen=0;++if(old_fastopenq->rskq_rst_head){+if(new_fastopenq->rskq_rst_head)+old_fastopenq->rskq_rst_tail->dl_next=new_fastopenq->rskq_rst_head;+else+old_fastopenq->rskq_rst_tail=new_fastopenq->rskq_rst_tail;++new_fastopenq->rskq_rst_head=old_fastopenq->rskq_rst_head;+old_fastopenq->rskq_rst_head=NULL;+old_fastopenq->rskq_rst_tail=NULL;+}++spin_unlock_bh(l4);+spin_unlock_bh(l3);+}+EXPORT_SYMBOL(inet_csk_reqsk_queue_migrate);+structsock*inet_csk_complete_hashdance(structsock*sk,structsock*child,structrequest_sock*req,boolown_req){
A TFO request socket is only freed after BOTH 3WHS has completed (or
aborted) and the child socket has been accepted (or its listener has been
closed). Hence, depending on the order, there can be two kinds of request
sockets in the accept queue.
3WHS -> accept : TCP_ESTABLISHED
accept -> 3WHS : TCP_SYN_RECV
Unlike TCP_ESTABLISHED socket, accept() does not free the request socket
for TCP_SYN_RECV socket. It is freed later at reqsk_fastopen_remove().
Also, it accesses request_sock.rsk_listener. So, in order to complete TFO
socket migration, we have to set the current listener to it at accept()
before reqsk_fastopen_remove().
Reviewed-by: Benjamin Herrenschmidt <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
net/ipv4/inet_connection_sock.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
@@ -500,6 +500,16 @@ struct sock *inet_csk_accept(struct sock *sk, int flags, int *err, bool kern)tcp_rsk(req)->tfo_listener){spin_lock_bh(&queue->fastopenq.lock);if(tcp_rsk(req)->tfo_listener){+if(req->rsk_listener!=sk){+/* TFO request was migrated to another listener so+*thenewlistenermustbeusedinreqsk_fastopen_remove()+*toholdrequestswhichcauseRST.+*/+sock_put(req->rsk_listener);+sock_hold(sk);+req->rsk_listener=sk;+}+/* We are still waiting for the final ACK from 3WHS*socan'tfreereqnow.Instead,wesetreq->skto*NULLtosignifythatthechildsocketistaken
This patch lets reuseport_detach_sock() return a pointer of struct sock,
which is used only by inet_unhash(). If it is not NULL,
inet_csk_reqsk_queue_migrate() migrates TCP_ESTABLISHED/TCP_SYN_RECV
sockets from the closing listener to the selected one.
By default, the kernel selects a new listener randomly. In order to pick
out a different socket every time, we select the last element of socks[] as
the new listener. This behaviour is based on how the kernel moves sockets
in socks[]. (See also [1])
Basically, in order to redistribute sockets evenly, we have to use an eBPF
program called in the later commit, but as the side effect of such default
selection, the kernel can redistribute old requests evenly to new listeners
for a specific case where the application replaces listeners by
generations.
For example, we call listen() for four sockets (A, B, C, D), and close()
the first two by turns. The sockets move in socks[] like below.
socks[0] : A <-. socks[0] : D socks[0] : D
socks[1] : B | => socks[1] : B <-. => socks[1] : C
socks[2] : C | socks[2] : C --'
socks[3] : D --'
Then, if C and D have newer settings than A and B, and each socket has a
request (a, b, c, d) in their accept queue, we can redistribute old
requests evenly to new listeners.
socks[0] : A (a) <-. socks[0] : D (a + d) socks[0] : D (a + d)
socks[1] : B (b) | => socks[1] : B (b) <-. => socks[1] : C (b + c)
socks[2] : C (c) | socks[2] : C (c) --'
socks[3] : D (d) --'
Here, (A, D), or (B, C) can have different application settings, but they
MUST have the same settings at the socket API level; otherwise, unexpected
error may happen. For instance, if only the new listeners have
TCP_SAVE_SYN, old requests do not hold SYN data, so the application will
face inconsistency and cause an error.
Therefore, if there are different kinds of sockets, we must attach an eBPF
program described in later commits.
Link: https://lore.kernel.org/netdev/CAEfhGiyG8Y_amDZ2C8dQoQqjZJMHjTY76b=KBkTKcBtA=dhdGQ@mail.gmail.com/
Reviewed-by: Benjamin Herrenschmidt <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
include/net/sock_reuseport.h | 2 +-
net/core/sock_reuseport.c | 16 +++++++++++++---
net/ipv4/inet_hashtables.c | 9 +++++++--
3 files changed, 21 insertions(+), 6 deletions(-)
This patch renames reuseport_select_sock() to __reuseport_select_sock() and
adds two wrapper function of it to pass the migration type defined in the
previous commit.
reuseport_select_sock : BPF_SK_REUSEPORT_MIGRATE_NO
reuseport_select_migrated_sock : BPF_SK_REUSEPORT_MIGRATE_REQUEST
As mentioned before, we have to select a new listener for TCP_NEW_SYN_RECV
requests at receiving the final ACK or sending a SYN+ACK. Therefore, this
patch also changes the code to call reuseport_select_migrated_sock() even
if the listening socket is TCP_CLOSE. If we can pick out a listening socket
from the reuseport group, we rewrite request_sock.rsk_listener and resume
processing the request.
Link: https://lore.kernel.org/bpf/202012020136.bF0Z4Guu-lkp@intel.com/
Reported-by: kernel test robot <redacted>
Reviewed-by: Benjamin Herrenschmidt <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
include/net/inet_connection_sock.h | 11 ++++++++
include/net/request_sock.h | 13 ++++++++++
include/net/sock_reuseport.h | 8 +++---
net/core/sock_reuseport.c | 40 ++++++++++++++++++++++++------
net/ipv4/inet_connection_sock.c | 13 ++++++++--
net/ipv4/tcp_ipv4.c | 9 +++++--
net/ipv6/tcp_ipv6.c | 9 +++++--
7 files changed, 86 insertions(+), 17 deletions(-)
@@ -170,7 +170,7 @@ int reuseport_add_sock(struct sock *sk, struct sock *sk2, bool bind_inany)}reuse->socks[reuse->num_socks]=sk;-/* paired with smp_rmb() in reuseport_select_sock() */+/* paired with smp_rmb() in __reuseport_select_sock() */smp_wmb();reuse->num_socks++;rcu_assign_pointer(sk->sk_reuseport_cb,reuse);
@@ -743,8 +743,17 @@ static void reqsk_timer_handler(struct timer_list *t)structrequest_sock_queue*queue=&icsk->icsk_accept_queue;intmax_syn_ack_retries,qlen,expire=0,resend=0;-if(inet_sk_state_load(sk_listener)!=TCP_LISTEN)-gotodrop;+if(inet_sk_state_load(sk_listener)!=TCP_LISTEN){+sk_listener=reuseport_select_migrated_sock(sk_listener,+req_to_sk(req)->sk_hash,NULL);+if(!sk_listener){+sk_listener=req->rsk_listener;+gotodrop;+}+inet_csk_reqsk_queue_migrated(req->rsk_listener,sk_listener,req);+icsk=inet_csk(sk_listener);+queue=&icsk->icsk_accept_queue;+}max_syn_ack_retries=icsk->icsk_syn_retries?:net->ipv4.sysctl_tcp_synack_retries;/* Normally all the openreqs are young and become mature
@@ -1978,8 +1978,13 @@ int tcp_v4_rcv(struct sk_buff *skb)gotocsum_error;}if(unlikely(sk->sk_state!=TCP_LISTEN)){-inet_csk_reqsk_queue_drop_and_put(sk,req);-gotolookup;+nsk=reuseport_select_migrated_sock(sk,req_to_sk(req)->sk_hash,skb);+if(!nsk){+inet_csk_reqsk_queue_drop_and_put(sk,req);+gotolookup;+}+inet_csk_reqsk_queue_migrated(sk,nsk,req);+sk=nsk;}/* We own a reference on the listener, increase it again*aswemightloseittoosoon.
This commit adds new bpf_attach_type for BPF_PROG_TYPE_SK_REUSEPORT to
check if the attached eBPF program is capable of migrating sockets.
When the eBPF program is attached, the kernel runs it for socket migration
only if the expected_attach_type is BPF_SK_REUSEPORT_SELECT_OR_MIGRATE.
The kernel will change the behaviour depending on the returned value:
- SK_PASS with selected_sk, select it as a new listener
- SK_PASS with selected_sk NULL, fall back to the random selection
- SK_DROP, cancel the migration
Link: https://lore.kernel.org/netdev/20201123003828.xjpjdtk4ygl6tg6h@kafai-mbp.dhcp.thefacebook.com/
Suggested-by: Martin KaFai Lau <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
include/uapi/linux/bpf.h | 2 ++
kernel/bpf/syscall.c | 13 +++++++++++++
tools/include/uapi/linux/bpf.h | 2 ++
3 files changed, 17 insertions(+)
This commit introduces a new section (sk_reuseport/migrate) and sets
expected_attach_type to two each section in BPF_PROG_TYPE_SK_REUSEPORT
program.
Signed-off-by: Kuniyuki Iwashima <redacted>
---
tools/lib/bpf/libbpf.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
This patch adds u8 migration field to sk_reuseport_kern and sk_reuseport_md
to signal the eBPF program if the kernel calls it for selecting a listener
for SYN or migrating sockets in the accept queue or an immature socket
during 3WHS.
Note that this field is accessible only if the attached type is
BPF_SK_REUSEPORT_SELECT_OR_MIGRATE.
Link: https://lore.kernel.org/netdev/20201123003828.xjpjdtk4ygl6tg6h@kafai-mbp.dhcp.thefacebook.com/
Suggested-by: Martin KaFai Lau <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
include/linux/bpf.h | 1 +
include/linux/filter.h | 4 ++--
include/uapi/linux/bpf.h | 1 +
net/core/filter.c | 15 ++++++++++++---
net/core/sock_reuseport.c | 2 +-
tools/include/uapi/linux/bpf.h | 1 +
6 files changed, 18 insertions(+), 6 deletions(-)
@@ -4462,6 +4462,7 @@ struct sk_reuseport_md {__u32ip_protocol;/* IP protocol. e.g. IPPROTO_TCP, IPPROTO_UDP */__u32bind_inany;/* Is sock bound to an INANY address? */__u32hash;/* A hash of the packet 4 tuples */+__u8migration;/* Migration type */};#define BPF_TAG_SIZE 8
@@ -4462,6 +4462,7 @@ struct sk_reuseport_md {__u32ip_protocol;/* IP protocol. e.g. IPPROTO_TCP, IPPROTO_UDP */__u32bind_inany;/* Is sock bound to an INANY address? */__u32hash;/* A hash of the packet 4 tuples */+__u8migration;/* Migration type */};#define BPF_TAG_SIZE 8
We will call sock_reuseport.prog for socket migration in the next commit,
so the eBPF program has to know which listener is closing in order to
select the new listener.
Currently, we can get a unique ID for each listener in the userspace by
calling bpf_map_lookup_elem() for BPF_MAP_TYPE_REUSEPORT_SOCKARRAY map.
This patch makes the sk pointer available in sk_reuseport_md so that we can
get the ID by BPF_FUNC_get_socket_cookie() in the eBPF program.
Link: https://lore.kernel.org/netdev/20201119001154.kapwihc2plp4f7zc@kafai-mbp.dhcp.thefacebook.com/
Suggested-by: Martin KaFai Lau <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
include/uapi/linux/bpf.h | 8 ++++++++
net/core/filter.c | 22 ++++++++++++++++++++++
tools/include/uapi/linux/bpf.h | 8 ++++++++
3 files changed, 38 insertions(+)
@@ -1655,6 +1655,13 @@ union bpf_attr {*A8-bytelongnon-decreasingnumberonsuccess,or0ifthe*socketfieldismissinginside*skb*.*+*u64bpf_get_socket_cookie(structbpf_sock*sk)+*Description+*Equivalenttobpf_get_socket_cookie()helperthataccepts+**skb*,butgetssocketfrom**structbpf_sock**context.+*Return+*A8-bytelongnon-decreasingnumber.+**u64bpf_get_socket_cookie(structbpf_sock_addr*ctx)*Description*Equivalenttobpf_get_socket_cookie()helperthataccepts
@@ -4463,6 +4470,7 @@ struct sk_reuseport_md {__u32bind_inany;/* Is sock bound to an INANY address? */__u32hash;/* A hash of the packet 4 tuples */__u8migration;/* Migration type */+__bpf_md_ptr(structbpf_sock*,sk);/* Current listening socket */};#define BPF_TAG_SIZE 8
@@ -1655,6 +1655,13 @@ union bpf_attr {*A8-bytelongnon-decreasingnumberonsuccess,or0ifthe*socketfieldismissinginside*skb*.*+*u64bpf_get_socket_cookie(structbpf_sock*sk)+*Description+*Equivalenttobpf_get_socket_cookie()helperthataccepts+**skb*,butgetssocketfrom**structbpf_sock**context.+*Return+*A8-bytelongnon-decreasingnumber.+**u64bpf_get_socket_cookie(structbpf_sock_addr*ctx)*Description*Equivalenttobpf_get_socket_cookie()helperthataccepts
@@ -4463,6 +4470,7 @@ struct sk_reuseport_md {__u32bind_inany;/* Is sock bound to an INANY address? */__u32hash;/* A hash of the packet 4 tuples */__u8migration;/* Migration type */+__bpf_md_ptr(structbpf_sock*,sk);/* Current listening socket */};#define BPF_TAG_SIZE 8
This patch supports socket migration by eBPF. If the attached type is
BPF_SK_REUSEPORT_SELECT_OR_MIGRATE, we can select a new listener by
BPF_FUNC_sk_select_reuseport(). Also, we can cancel migration by returning
SK_DROP. This feature is useful when listeners have different settings at
the socket API level or when we want to free resources as soon as possible.
There are two noteworthy points. The first is that we select a listening
socket in reuseport_detach_sock() and __reuseport_select_sock(), but we do
not have struct skb at closing a listener or retransmitting a SYN+ACK.
However, some helper functions do not expect skb is NULL (e.g.
skb_header_pointer() in BPF_FUNC_skb_load_bytes(), skb_tail_pointer() in
BPF_FUNC_skb_load_bytes_relative()). So we allocate an empty skb
temporarily before running the eBPF program. The second is that we do not
have struct request_sock in unhash path, and the sk_hash of the listener is
always zero. So we pass zero as hash to bpf_run_sk_reuseport().
Reviewed-by: Benjamin Herrenschmidt <redacted>
Signed-off-by: Kuniyuki Iwashima <redacted>
---
net/core/filter.c | 19 +++++++++++++++++++
net/core/sock_reuseport.c | 21 +++++++++++----------
net/ipv4/inet_hashtables.c | 2 +-
3 files changed, 31 insertions(+), 11 deletions(-)
@@ -9890,10 +9890,29 @@ struct sock *bpf_run_sk_reuseport(struct sock_reuseport *reuse, struct sock *sk,{structsk_reuseport_kernreuse_kern;enumsk_actionaction;+boolallocated=false;++if(migration){+/* cancel migration for possibly incapable eBPF program */+if(prog->expected_attach_type!=BPF_SK_REUSEPORT_SELECT_OR_MIGRATE)+returnERR_PTR(-ENOTSUPP);++if(!skb){+allocated=true;+skb=alloc_skb(0,GFP_ATOMIC);+if(!skb)+returnERR_PTR(-ENOMEM);+}+}elseif(!skb){+returnNULL;/* fall back to select by hash */+}bpf_init_reuseport_kern(&reuse_kern,reuse,sk,skb,hash,migration);action=BPF_PROG_RUN(prog,&reuse_kern);+if(allocated)+kfree_skb(skb);+if(action==SK_PASS)returnreuse_kern.selected_sk;else
@@ -22,6 +22,8 @@ int in_lock_functions(unsigned long addr);void__lockfunc_raw_spin_lock(raw_spinlock_t*lock)__acquires(lock);void__lockfunc_raw_spin_lock_nested(raw_spinlock_t*lock,intsubclass)__acquires(lock);+void__lockfunc_raw_spin_lock_bh_nested(raw_spinlock_t*lock,intsubclass)+__acquires(lock);void__lockfunc_raw_spin_lock_nest_lock(raw_spinlock_t*lock,structlockdep_map*map)__acquires(lock);