From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:35:57
Update on io_uring zerocopy tx, still RFC. For v1 and design notes see
https://lore.kernel.org/io-uring/cover.1638282789.git.asml.silence@gmail.com/
Absolute numbers (against dummy) got higher since v1, + ~10-12% requests/s for
the peak performance case. 5/19 brought a couple of percents, but most of it
came with 8/19 and 9/19 (+8-11% in numbers, 5-7% in profiles). It will also
be needed in the future for p2p. Any reason not to do alike for paged non-zc?
Small (under 100-150B) packets?
Most of checks are removed from non-zc paths. Implemented a bit trickier in
__ip_append_data(), but considering already existing assumptions around "from"
argument it should be fine.
Benchmarks for dummy netdev, UDP/IPv4, payload size=4096:
-n<N> is how many requests we submit per syscall. From io_uring perspective -n1
is wasteful and far from optimal, but included for comparison.
-z0 disables zerocopy, just normal io_uring send requests
-f makes to flush "buffer free" notifications for every request
| K reqs/s | speedup
msg_zerocopy (non-zc) | 1120 | 1.12
msg_zerocopy (zc) | 997 | 1
io_uring -n1 -z0 | 1469 | 1.47
io_uring -n8 -z0 | 1780 | 1.78
io_uring -n1 -f | 1688 | 1.69
io_uring -n1 | 1774 | 1.77
io_uring -n8 -f | 2075 | 2.08
io_uring -n8 | 2265 | 2.27
note: it might be not too interesting to compare zc vs non-zc, the performance
relative difference can be shifted in favour of zerocopy by cutting constant
per-request overhead, and there are easy ways of doing that, e.g. by compiling
out unused features. Even more true for the table below as there was additional
noise taking a good quarter of CPU cycles.
Some data for UDP/IPv6 between a pair of NICs. 9/19 wasn't there at the time of
testing. All tests are CPU bound and so as expected reqs/s for zerocopy doesn't
vary much between different payload sizes. io_uring to msg_zerocopy ratio is not
too representative for reasons similar to described above.
payload | test | K reqs/s
___________________________________________
8192 | io_uring -n8 (dummy) | 599
| io_uring -n1 -z0 | 264
| io_uring -n8 -z0 | 302
| msg_zerocopy | 248
| msg_zerocopy -z | 183
| io_uring -n1 -f | 306
| io_uring -n1 | 318
| io_uring -n8 -f | 373
| io_uring -n8 | 401
4096 | io_uring -n8 (dummy) | 601
| io_uring -n1 -z0 | 303
| io_uring -n8 -z0 | 366
| msg_zerocopy | 278
| msg_zerocopy -z | 187
| io_uring -n1 -f | 317
| io_uring -n1 | 325
| io_uring -n8 -f | 387
| io_uring -n8 | 405
1024 | io_uring -n8 (dummy) | 601
| io_uring -n1 -z0 | 329
| io_uring -n8 -z0 | 407
| msg_zerocopy | 301
| msg_zerocopy -z | 186
| io_uring -n1 -f | 317
| io_uring -n1 | 327
| io_uring -n8 -f | 390
| io_uring -n8 | 403
512 | io_uring -n8 (dummy) | 601
| io_uring -n1 -z0 | 340
| io_uring -n8 -z0 | 417
| msg_zerocopy | 310
| msg_zerocopy -z | 186
| io_uring -n1 -f | 317
| io_uring -n1 | 328
| io_uring -n8 -f | 392
| io_uring -n8 | 406
128 | io_uring -n8 (dummy) | 602
| io_uring -n1 -z0 | 341
| io_uring -n8 -z0 | 428
| msg_zerocopy | 317
| msg_zerocopy -z | 188
| io_uring -n1 -f | 318
| io_uring -n1 | 331
| io_uring -n8 -f | 391
| io_uring -n8 | 408
https://github.com/isilence/linux/tree/zc_v2https://github.com/isilence/liburing/tree/zc_v2
The Benchmark is <liburing>/test/send-zc,
send-zc [-f] [-n<N>] [-z0] -s<payload size> -D<dst ip> (-6|-4) [-t<sec>] udp
As a server you can use msg_zerocopy from in kernel's selftests, or a copy of
it at <liburing>/test/msg_zerocopy. No server is needed for dummy testing.
dummy setup:
sudo ip li add dummy0 type dummy && sudo ip li set dummy0 up mtu 65536
# make traffic for the specified IP to go through dummy0
sudo ip route add <ip_address> dev dummy0
v2: remove additional overhead for non-zc from skb_release_data() (Jonathan)
avoid msg propagation, hide extra bits of non-zc overhead
task_work based "buffer free" notifications
improve io_uring's notification refcounting
added 5/19, (no pfmemalloc tracking)
added 8/19 and 9/19 preventing small copies with zc
misc small changes
Pavel Begunkov (19):
skbuff: add SKBFL_DONT_ORPHAN flag
skbuff: pass a struct ubuf_info in msghdr
net: add zerocopy_sg_from_iter for bvec
net: optimise page get/free for bvec zc
net: don't track pfmemalloc for zc registered mem
ipv4/udp: add support msgdr::msg_ubuf
ipv6/udp: add support msgdr::msg_ubuf
ipv4: avoid partial copy for zc
ipv6: avoid partial copy for zc
io_uring: add send notifiers registration
io_uring: infrastructure for send zc notifications
io_uring: wire send zc request type
io_uring: add an option to flush zc notifications
io_uring: opcode independent fixed buf import
io_uring: sendzc with fixed buffers
io_uring: cache struct ubuf_info
io_uring: unclog ctx refs waiting with zc notifiers
io_uring: task_work for notification delivery
io_uring: optimise task referencing by notifiers
fs/io_uring.c | 440 +++++++++++++++++++++++++++++++++-
include/linux/skbuff.h | 46 ++--
include/linux/socket.h | 1 +
include/uapi/linux/io_uring.h | 14 ++
net/compat.c | 1 +
net/core/datagram.c | 58 +++++
net/core/skbuff.c | 16 +-
net/ipv4/ip_output.c | 55 +++--
net/ipv6/ip6_output.c | 54 ++++-
net/socket.c | 3 +
10 files changed, 633 insertions(+), 55 deletions(-)
--
2.34.1
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:35:59
We don't want to list every single ubuf_info callback in
skb_orphan_frags(), add a flag controlling the behaviour.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
include/linux/skbuff.h | 8 +++++---
net/core/skbuff.c | 2 +-
2 files changed, 6 insertions(+), 4 deletions(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:01
get_page() in __zerocopy_sg_from_bvec() and matching put_page()s are
expensive. However, we can avoid it if the caller can guarantee that
pages stay alive until the corresponding ubuf_info is not released.
In particular, it targets io_uring with fixed buffers following the
described contract.
Assuming that nobody yet uses bvec together with zerocopy, make all
calls with bvec iterators follow this model.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
include/linux/skbuff.h | 12 ++++++++++--
net/core/datagram.c | 9 +++++++--
net/core/skbuff.c | 14 +++++++++++++-
3 files changed, 30 insertions(+), 5 deletions(-)
@@ -1725,6 +1734,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,refcount_inc(&skb_uarg(skb)->refcnt);for(i=0;i<skb_shinfo(skb)->nr_frags;i++)skb_frag_ref(skb,i);+skb_shinfo(skb)->flags&=~SKBFL_MANAGED_FRAGS;if(skb_has_frag_list(skb))skb_clone_fraglist(skb);
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:04
Add a separate path for bvec iterators in __zerocopy_sg_from_iter, first
it's quite faster but also will be needed to optimise out
get/put_page()
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
net/core/datagram.c | 50 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 50 insertions(+)
@@ -616,11 +616,61 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset,}EXPORT_SYMBOL(skb_copy_datagram_from_iter);+staticint__zerocopy_sg_from_bvec(structsock*sk,structsk_buff*skb,+structiov_iter*from,size_tlength)+{+intret,frag=skb_shinfo(skb)->nr_frags;+structbvec_iterbi;+structbio_vecv;+ssize_tcopied=0;+unsignedlongtruesize=0;++bi.bi_size=min(from->count,length);+bi.bi_bvec_done=from->iov_offset;+bi.bi_idx=0;++while(bi.bi_size){+if(frag==MAX_SKB_FRAGS){+ret=-EMSGSIZE;+gotoout;+}++v=mp_bvec_iter_bvec(from->bvec,bi);+copied+=v.bv_len;+truesize+=PAGE_ALIGN(v.bv_len+v.bv_offset);+get_page(v.bv_page);+skb_fill_page_desc(skb,frag++,v.bv_page,v.bv_offset,v.bv_len);+bvec_iter_advance_single(from->bvec,&bi,v.bv_len);+}+ret=0;+out:+skb->data_len+=copied;+skb->len+=copied;+skb->truesize+=truesize;++if(sk&&sk->sk_type==SOCK_STREAM){+sk_wmem_queued_add(sk,truesize);+if(!skb_zcopy_pure(skb))+sk_mem_charge(sk,truesize);+}else{+refcount_add(truesize,&skb->sk->sk_wmem_alloc);+}++from->bvec+=bi.bi_idx;+from->nr_segs-=bi.bi_idx;+from->count=bi.bi_size;+from->iov_offset=bi.bi_bvec_done;+returnret;+}+int__zerocopy_sg_from_iter(structsock*sk,structsk_buff*skb,structiov_iter*from,size_tlength){intfrag=skb_shinfo(skb)->nr_frags;+if(iov_iter_is_bvec(from))+return__zerocopy_sg_from_bvec(sk,skb,from,length);+while(length&&iov_iter_count(from)){structpage*pages[MAX_SKB_FRAGS];structpage*last_head=NULL;
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:07
Instead of the net stack managing ubuf_info, allow to pass it in from
outside in a struct msghdr (in-kernel structure), so io_uring can make
use of it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 2 ++
include/linux/socket.h | 1 +
net/compat.c | 1 +
net/socket.c | 3 +++
4 files changed, 7 insertions(+)
@@ -65,6 +65,7 @@ struct msghdr {__kernel_size_tmsg_controllen;/* ancillary data buffer length */unsignedintmsg_flags;/* flags on received message */structkiocb*msg_iocb;/* ptr to iocb for async requests */+structubuf_info*msg_ubuf;};structuser_msghdr{
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:08
In case of zerocopy frags are filled with userspace allocated memory, we
shouldn't care about setting skb->pfmemalloc for them, and especially
when the buffers were somehow pre-registered (i.e. getting bvec from
io_uring). Remove the tracking from __zerocopy_sg_from_bvec().
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
include/linux/skbuff.h | 28 +++++++++++++++++-----------
net/core/datagram.c | 7 +++++--
2 files changed, 22 insertions(+), 13 deletions(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:13
Make ipv4/udp to use ubuf_info passed in struct msghdr if it was
specified.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
net/ipv4/ip_output.c | 50 ++++++++++++++++++++++++++++++++------------
1 file changed, 37 insertions(+), 13 deletions(-)
@@ -953,7 +953,6 @@ static int __ip_append_data(struct sock *sk,structinet_sock*inet=inet_sk(sk);structubuf_info*uarg=NULL;structsk_buff*skb;-structip_options*opt=cork->opt;inthh_len;intexthdrlen;
@@ -967,6 +966,7 @@ static int __ip_append_data(struct sock *sk,unsignedintwmem_alloc_delta=0;boolpaged,extra_uref=false;u32tskey=0;+boolzc=false;skb=skb_peek_tail(queue);
@@ -1001,17 +1001,37 @@ static int __ip_append_data(struct sock *sk,(!exthdrlen||(rt->dst.dev->features&NETIF_F_HW_ESP_TX_CSUM)))csummode=CHECKSUM_PARTIAL;-if(flags&MSG_ZEROCOPY&&length&&sock_flag(sk,SOCK_ZEROCOPY)){-uarg=msg_zerocopy_realloc(sk,length,skb_zcopy(skb));-if(!uarg)-return-ENOBUFS;-extra_uref=!skb_zcopy(skb);/* only ref on new uarg */-if(rt->dst.dev->features&NETIF_F_SG&&-csummode==CHECKSUM_PARTIAL){-paged=true;-}else{-uarg->zerocopy=0;-skb_zcopy_set(skb,uarg,&extra_uref);+if((flags&MSG_ZEROCOPY)&&length){+structmsghdr*msg=from;++if(getfrag==ip_generic_getfrag&&msg->msg_ubuf){+uarg=msg->msg_ubuf;+if(skb_zcopy(skb)&&uarg!=skb_zcopy(skb))+return-EINVAL;++if(rt->dst.dev->features&NETIF_F_SG&&+csummode==CHECKSUM_PARTIAL){+paged=true;+zc=true;+}else{+/* Drop uarg if can't zerocopy, callers should+*beabletohandleit.+*/+uarg=NULL;+}+}elseif(sock_flag(sk,SOCK_ZEROCOPY)){+uarg=msg_zerocopy_realloc(sk,length,skb_zcopy(skb));+if(!uarg)+return-ENOBUFS;+extra_uref=!skb_zcopy(skb);/* only ref on new uarg */+if(rt->dst.dev->features&NETIF_F_SG&&+csummode==CHECKSUM_PARTIAL){+paged=true;+zc=true;+}else{+uarg->zerocopy=0;+skb_zcopy_set(skb,uarg,&extra_uref);+}}}
@@ -1172,9 +1192,13 @@ static int __ip_append_data(struct sock *sk,err=-EFAULT;gotoerror;}-}elseif(!uarg||!uarg->zerocopy){+}elseif(!zc){inti=skb_shinfo(skb)->nr_frags;+if(skb_shinfo(skb)->flags&SKBFL_MANAGED_FRAGS){+err=-EFAULT;+gotoerror;+}err=-ENOMEM;if(!sk_page_frag_refill(sk,pfrag))gotoerror;
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:15
Make ipv6/udp to use ubuf_info passed in struct msghdr if it was
specified.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
net/ipv6/ip6_output.c | 49 ++++++++++++++++++++++++++++++++-----------
1 file changed, 37 insertions(+), 12 deletions(-)
@@ -1452,6 +1452,7 @@ static int __ip6_append_data(struct sock *sk,unsignedintmaxnonfragsize,headersize;unsignedintwmem_alloc_delta=0;boolpaged,extra_uref=false;+boolzc=false;skb=skb_peek_tail(queue);if(!skb){
@@ -1516,17 +1517,37 @@ static int __ip6_append_data(struct sock *sk,rt->dst.dev->features&(NETIF_F_IPV6_CSUM|NETIF_F_HW_CSUM))csummode=CHECKSUM_PARTIAL;-if(flags&MSG_ZEROCOPY&&length&&sock_flag(sk,SOCK_ZEROCOPY)){-uarg=msg_zerocopy_realloc(sk,length,skb_zcopy(skb));-if(!uarg)-return-ENOBUFS;-extra_uref=!skb_zcopy(skb);/* only ref on new uarg */-if(rt->dst.dev->features&NETIF_F_SG&&-csummode==CHECKSUM_PARTIAL){-paged=true;-}else{-uarg->zerocopy=0;-skb_zcopy_set(skb,uarg,&extra_uref);+if((flags&MSG_ZEROCOPY)&&length){+structmsghdr*msg=from;++if(getfrag==ip_generic_getfrag&&msg->msg_ubuf){+uarg=msg->msg_ubuf;+if(skb_zcopy(skb)&&uarg!=skb_zcopy(skb))+return-EINVAL;++if(rt->dst.dev->features&NETIF_F_SG&&+csummode==CHECKSUM_PARTIAL){+paged=true;+zc=true;+}else{+/* Drop uarg if can't zerocopy, callers should+*beabletohandleit.+*/+uarg=NULL;+}+}elseif(sock_flag(sk,SOCK_ZEROCOPY)){+uarg=msg_zerocopy_realloc(sk,length,skb_zcopy(skb));+if(!uarg)+return-ENOBUFS;+extra_uref=!skb_zcopy(skb);/* only ref on new uarg */+if(rt->dst.dev->features&NETIF_F_SG&&+csummode==CHECKSUM_PARTIAL){+paged=true;+zc=true;+}else{+uarg->zerocopy=0;+skb_zcopy_set(skb,uarg,&extra_uref);+}}}
@@ -1717,9 +1738,13 @@ static int __ip6_append_data(struct sock *sk,err=-EFAULT;gotoerror;}-}elseif(!uarg||!uarg->zerocopy){+}elseif(!zc){inti=skb_shinfo(skb)->nr_frags;+if(skb_shinfo(skb)->flags&SKBFL_MANAGED_FRAGS){+err=-EFAULT;+gotoerror;+}err=-ENOMEM;if(!sk_page_frag_refill(sk,pfrag))gotoerror;
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:21
Even when zerocopy transmission is requested and possible,
__ip_append_data() will still copy a small chunk of data just because it
allocated some extra linear space (e.g. 148 bytes). It wastes CPU cycles
on copy and iter manipulations and also misalignes potentially aligned
data. Avoid such coies. And as a bonus we can allocate smaller skb.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
net/ipv4/ip_output.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:24
Even when zerocopy transmission is requested and possible,
__ip_append_data() will still copy a small chunk of data just because it
allocated some extra linear space (e.g. 128 bytes). It wastes CPU cycles
on copy and iter manipulations and also misalignes potentially aligned
data. Avoid such coies. And as a bonus we can allocate smaller skb.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
net/ipv6/ip6_output.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:28
Add IORING_REGISTER_TX_CTX and IORING_UNREGISTER_TX_CTX. Transmission
(i.e. send) context will serve be used to notify the userspace when
fixed buffers used for zerocopy sends are released by the kernel.
Notification of a single tx context lives in generations, where each
generation posts one CQE with ->user_data equal to the specified tag and
->res is a generation number starting from 0. All requests issued
against a ctx will get attached to the current generation of
notifications. Then, the userspace will be able to request to flush the
notification allowing it to post a CQE when all buffers of all requests
attached to it are released by the kernel. It'll also switch the
generation to a new one with a sequence number incremented by one.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 72 +++++++++++++++++++++++++++++++++++
include/uapi/linux/io_uring.h | 7 ++++
2 files changed, 79 insertions(+)
@@ -326,6 +328,15 @@ struct io_submit_state {structblk_plugplug;};+structio_tx_notifier{+};++structio_tx_ctx{+structio_tx_notifier*notifier;+u64tag;+u32seq;+};+structio_ring_ctx{/* const or read-mostly hot data */struct{
@@ -325,6 +325,9 @@ enum {/* set/get max number of io-wq workers */IORING_REGISTER_IOWQ_MAX_WORKERS=19,+IORING_REGISTER_TX_CTX=20,+IORING_UNREGISTER_TX_CTX=21,+/* this goes last */IORING_REGISTER_LAST};
@@ -365,6 +368,10 @@ struct io_uring_rsrc_update2 {__u32resv2;};+structio_uring_tx_ctx_register{+__u64tag;+};+/* Skip updating fd indexes set to this value in the fd table */#define IORING_REGISTER_FILES_SKIP (-2)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:29
Add a new ubuf_info callback io_uring_tx_zerocopy_callback(), which
should post an CQE when it completes. Also, implement some
infrastructuire for allocating and managing struct ubuf_info.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 108 insertions(+), 6 deletions(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:34
Add a new io_uring opcode IORING_OP_SENDZC. The main distinction from
other send requests is that the user should specify a tx context index,
which will notifiy the userspace when the kernel doesn't need the
buffers anymore and it's safe to reuse them. So, overwriting data
buffers is racy before getting a separate notification even when the
request is already completed.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 120 +++++++++++++++++++++++++++++++++-
include/uapi/linux/io_uring.h | 2 +
2 files changed, 121 insertions(+), 1 deletion(-)
@@ -1123,6 +1134,12 @@ static const struct io_op_def io_op_defs[] = {[IORING_OP_MKDIRAT]={},[IORING_OP_SYMLINKAT]={},[IORING_OP_LINKAT]={},+[IORING_OP_SENDZC]={+.needs_file=1,+.unbound_nonreg_file=1,+.pollout=1,+.audit_skip=1,+},};/* requests with any of those set should undergo io_disarm_next() */
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:40
Extract an opcode independent helper from io_import_fixed for
initialising an iov_iter with a fixed buffer with
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:48
Allocation/deallocation of ubuf_info takes some time, add an
optimisation caching them. The implementation is alike to how we cache
requests in io_req_complete_post().
->ubuf_list is protected by ->uring_lock and requests try grab directly
from it, and there is also ->ubuf_list_locked list protected by
->completion_lock, which is eventually batch spliced to ->ubuf_list.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 74 ++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 64 insertions(+), 10 deletions(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:51
Currently every instance of struct io_tx_notifier holds a ctx reference,
including ones sitting in caches. So, when we try to quiesce the ring
(e.g. for register) we'd be waiting for refs that nobody can release.
That's worked around in for cancellation.
Don't do ctx references but wait for all notifiers to return into
caches when needed. Even better solution would be to wait for all rsrc
refs. It's also nice to remove an extra pair of percpu_ref_get/put().
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 19 ++++++++++++++++---
1 file changed, 16 insertions(+), 3 deletions(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:54
workqueues are way too heavy for tx notification delivery. We still
need some non-irq context because ->completion_lock is not irq-safe, so
use task_work instead. Expectedly, performance for test cases with real
hardware and juggling lots of notifications the perfomance is
drastically better, e.g. profiles percetage of relevant parts drops
from 30% to less than 3%
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 57 ++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 43 insertions(+), 14 deletions(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:36:55
Use io_put_task()/etc. infra holding task references for notifiers, so
we can get rid of atomics there.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-12-21 15:43:39
On 12/21/21 15:35, Pavel Begunkov wrote:
Update on io_uring zerocopy tx, still RFC. For v1 and design notes see
[...]
Benchmarks for dummy netdev, UDP/IPv4, payload size=4096:
-n<N> is how many requests we submit per syscall. From io_uring perspective -n1
is wasteful and far from optimal, but included for comparison.
-z0 disables zerocopy, just normal io_uring send requests
-f makes to flush "buffer free" notifications for every request
Instead of the net stack managing ubuf_info, allow to pass it in from
outside in a struct msghdr (in-kernel structure), so io_uring can make
use of it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
Hi Pavel,
I've some confusions here since I have a lack of
network knowledge.
The first one is why do we make ubuf_info visible
for io_uring. Why not just follow the old MSG_ZEROCOPY
logic?
The second one, my understanding about the buffer
lifecycle is that the kernel side informs
the userspace by a cqe generated by the ubuf_info
callback that all the buffers attaching to the
same notifier is now free to use when all the data
is sent, then why is the flush in 13/19 needed as
it is at the submission period?
Regards,
Hao
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2022-01-11 15:54:06
On 1/11/22 13:51, Hao Xu wrote:
在 2021/12/21 下午11:35, Pavel Begunkov 写道:
quoted
Instead of the net stack managing ubuf_info, allow to pass it in from
outside in a struct msghdr (in-kernel structure), so io_uring can make
use of it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
Hi Pavel,
I've some confusions here since I have a lack of
network knowledge.
The first one is why do we make ubuf_info visible
for io_uring. Why not just follow the old MSG_ZEROCOPY
logic?
I assume you mean leaving allocation up and so in socket awhile the
patchset let's io_uring to manage and control ubufs. In short,
performance and out convenience
TL;DR;
First, we want a nice and uniform API with io_uring, i.e. posting
an CQE instead of polling an err queue/etc., and for that the network
will need to know about io_uring ctx in some way. As an alternative it
may theoretically be registered in socket, but it'll quickly turn into
a huge mess, consider that it's a many to many relation b/w io_uring and
sockets. The fact that io_uring holds refs to files will only complicate
it.
It will also limit API. For instance, we won't be able to use a single
ubuf with several different sockets.
Another problem is performance, registration or some other tricks
would some additional sync. It'd also need sync on use, say it's
just one rcu_read, but the problem that it only adds up to complexity
and prevents some other optimisations. E.g. we amortise to ~0 atomics
getting refs on skb setups based on guarantees io_uring provides, and
not only. SKBFL_MANAGED_FRAGS can only work with pages being controlled
by the issuer, and so it needs some context as currently provided by
ubuf. io_uring also caches ubufs, which relies on io_uring locking, so
it removes kmalloc/free for almost zero overhead.
The second one, my understanding about the buffer
lifecycle is that the kernel side informs
the userspace by a cqe generated by the ubuf_info
callback that all the buffers attaching to the
same notifier is now free to use when all the data
is sent, then why is the flush in 13/19 needed as
it is at the submission period?
Probably I wasn't clear enough. A user has to flush a notifier, only
then it's expected to post an CQE after all buffers attached to it
are freed. io_uring holds one ubuf ref, which will be release on flush.
I also need to add a way to flush without send.
Will spend some time documenting for next iteration.
--
Pavel Begunkov
Instead of the net stack managing ubuf_info, allow to pass it in from
outside in a struct msghdr (in-kernel structure), so io_uring can make
use of it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
Hi Pavel,
I've some confusions here since I have a lack of
network knowledge.
The first one is why do we make ubuf_info visible
for io_uring. Why not just follow the old MSG_ZEROCOPY
logic?
I assume you mean leaving allocation up and so in socket awhile the
patchset let's io_uring to manage and control ubufs. In short,
performance and out convenience
TL;DR;
First, we want a nice and uniform API with io_uring, i.e. posting
an CQE instead of polling an err queue/etc., and for that the network
will need to know about io_uring ctx in some way. As an alternative it
may theoretically be registered in socket, but it'll quickly turn into
a huge mess, consider that it's a many to many relation b/w io_uring and
sockets. The fact that io_uring holds refs to files will only complicate
it.
Make sense to me, thanks.
It will also limit API. For instance, we won't be able to use a single
ubuf with several different sockets.
Is there any use cases for this multiple sockets with single
notification?
Another problem is performance, registration or some other tricks
would some additional sync. It'd also need sync on use, say it's
just one rcu_read, but the problem that it only adds up to complexity
and prevents some other optimisations. E.g. we amortise to ~0 atomics
getting refs on skb setups based on guarantees io_uring provides, and
not only. SKBFL_MANAGED_FRAGS can only work with pages being controlled
by the issuer, and so it needs some context as currently provided by
ubuf. io_uring also caches ubufs, which relies on io_uring locking, so
it removes kmalloc/free for almost zero overhead.
quoted
The second one, my understanding about the buffer
lifecycle is that the kernel side informs
the userspace by a cqe generated by the ubuf_info
callback that all the buffers attaching to the
same notifier is now free to use when all the data
is sent, then why is the flush in 13/19 needed as
it is at the submission period?
Probably I wasn't clear enough. A user has to flush a notifier, only
then it's expected to post an CQE after all buffers attached to it
are freed. io_uring holds one ubuf ref, which will be release on flush.
I see, I saw another ref inc in skb_zcopy_set() which I previously
misunderstood and thus thought there was only one refcount. Thanks!
I also need to add a way to flush without send.
Will spend some time documenting for next iteration.
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2022-01-12 16:54:01
On 1/12/22 03:39, Hao Xu wrote:
在 2022/1/11 下午11:50, Pavel Begunkov 写道:
quoted
On 1/11/22 13:51, Hao Xu wrote:
quoted
在 2021/12/21 下午11:35, Pavel Begunkov 写道:
quoted
Instead of the net stack managing ubuf_info, allow to pass it in from
outside in a struct msghdr (in-kernel structure), so io_uring can make
use of it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
Hi Pavel,
I've some confusions here since I have a lack of
network knowledge.
The first one is why do we make ubuf_info visible
for io_uring. Why not just follow the old MSG_ZEROCOPY
logic?
I assume you mean leaving allocation up and so in socket awhile the
patchset let's io_uring to manage and control ubufs. In short,
performance and out convenience
TL;DR;
First, we want a nice and uniform API with io_uring, i.e. posting
an CQE instead of polling an err queue/etc., and for that the network
will need to know about io_uring ctx in some way. As an alternative it
may theoretically be registered in socket, but it'll quickly turn into
a huge mess, consider that it's a many to many relation b/w io_uring and
sockets. The fact that io_uring holds refs to files will only complicate
it.
Make sense to me, thanks.
quoted
It will also limit API. For instance, we won't be able to use a single
ubuf with several different sockets.
Is there any use cases for this multiple sockets with single
notification?
Don't know, scatter send maybe? It's just one of those moments when
a design that feels right (to me) yields more flexibility than was
initially planned, which is definitely a good thing
quoted
Another problem is performance, registration or some other tricks
would some additional sync. It'd also need sync on use, say it's
just one rcu_read, but the problem that it only adds up to complexity
and prevents some other optimisations. E.g. we amortise to ~0 atomics
getting refs on skb setups based on guarantees io_uring provides, and
not only. SKBFL_MANAGED_FRAGS can only work with pages being controlled
by the issuer, and so it needs some context as currently provided by
ubuf. io_uring also caches ubufs, which relies on io_uring locking, so
it removes kmalloc/free for almost zero overhead.
quoted
The second one, my understanding about the buffer
lifecycle is that the kernel side informs
the userspace by a cqe generated by the ubuf_info
callback that all the buffers attaching to the
same notifier is now free to use when all the data
is sent, then why is the flush in 13/19 needed as
it is at the submission period?
Probably I wasn't clear enough. A user has to flush a notifier, only
then it's expected to post an CQE after all buffers attached to it
are freed. io_uring holds one ubuf ref, which will be release on flush.
I see, I saw another ref inc in skb_zcopy_set() which I previously
misunderstood and thus thought there was only one refcount. Thanks!
quoted
I also need to add a way to flush without send.
Will spend some time documenting for next iteration.