From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:25
This series introduce XDP multi-buffer support. The mvneta driver is
the first to support these new "non-linear" xdp_{buff,frame}. Reviewers
please focus on how these new types of xdp_{buff,frame} packets
traverse the different layers and the layout design. It is on purpose
that BPF-helpers are kept simple, as we don't want to expose the
internal layout to allow later changes.
The main idea for the new multi-buffer layout is to reuse the same
structure used for non-linear SKB. This rely on the "skb_shared_info"
struct at the end of the first buffer to link together subsequent
buffers. Keeping the layout compatible with SKBs is also done to ease
and speedup creating a SKB from an xdp_{buff,frame}.
Converting xdp_frame to SKB and deliver it to the network stack is shown
in patch 05/18 (e.g. cpumaps).
A multi-buffer bit (mb) has been introduced in the flags field of xdp_{buff,frame}
structure to notify the bpf/network layer if this is a xdp multi-buffer frame
(mb = 1) or not (mb = 0).
The mb bit will be set by a xdp multi-buffer capable driver only for
non-linear frames maintaining the capability to receive linear frames
without any extra cost since the skb_shared_info structure at the end
of the first buffer will be initialized only if mb is set.
Moreover the flags field in xdp_{buff,frame} will be reused even for
xdp rx csum offloading in future series.
Typical use cases for this series are:
- Jumbo-frames
- Packet header split (please see Googleâs use-case @ NetDevConf 0x14, [0])
- TSO/GRO for XDP_REDIRECT
The three following ebpf helpers (and related selftests) has been introduced:
- bpf_xdp_load_bytes:
This helper is provided as an easy way to load data from a xdp buffer. It
can be used to load len bytes from offset from the frame associated to
xdp_md, into the buffer pointed by buf.
- bpf_xdp_store_bytes:
Store len bytes from buffer buf into the frame associated to xdp_md, at
offset.
- bpf_xdp_get_buff_len:
Return the total frame size (linear + paged parts)
bpf_xdp_adjust_tail and bpf_xdp_copy helpers have been modified to take into
account xdp multi-buff frames.
Moreover, similar to skb_header_pointer, we introduced bpf_xdp_pointer utility
routine to return a pointer to a given position in the xdp_buff if the
requested area (offset + len) is contained in a contiguous memory area
otherwise it will be copied in a bounce buffer provided by the caller.
BPF_F_XDP_MB flag for bpf_attr has been introduced to notify the kernel the
eBPF program fully support xdp multi-buffer.
SEC("xdp_mb/"), SEC_DEF("xdp_devmap_mb/") and SEC_DEF("xdp_cpumap_mb/" have been
introduced to declare xdp multi-buffer support.
The NIC driver is expected to reject an eBPF program if it is running in XDP
multi-buffer mode and the program does not support XDP multi-buffer.
In the same way it is not possible to mix xdp multi-buffer and xdp legacy
programs in a CPUMAP/DEVMAP or tailcall a xdp multi-buffer/legacy program from
a legacy/multi-buff one.
More info about the main idea behind this approach can be found here [1][2].
Changes since v17:
- rework bpf_xdp_copy to squash base and frag management
- remove unused variable in bpf_xdp_mb_shrink_tail()
- move bpf_xdp_copy_buf() out of bpf_xdp_pointer()
- add sanity check for len in bpf_xdp_pointer()
- remove EXPORT_SYMBOL for __xdp_return()
- introduce frag_size field in xdp_rxq_info to let the driver specify max value
for xdp fragments. frag_size set to 0 means the tail increase of last the
fragment is not supported.
Changes since v16:
- do not allow tailcalling a xdp multi-buffer/legacy program from a
legacy/multi-buff one.
- do not allow mixing xdp multi-buffer and xdp legacy programs in a
CPUMAP/DEVMAP
- add selftests for CPUMAP/DEVMAP xdp mb compatibility
- disable XDP_REDIRECT for xdp multi-buff for the moment
- set max offset value to 0xffff in bpf_xdp_pointer
- use ARG_PTR_TO_UNINIT_MEM and ARG_CONST_SIZE for arg3_type and arg4_type
of bpf_xdp_store_bytes/bpf_xdp_load_bytes
Changes since v15:
- let the verifier check buf is not NULL in
bpf_xdp_load_bytes/bpf_xdp_store_bytes helpers
- return an error if offset + length is over frame boundaries in
bpf_xdp_pointer routine
- introduce BPF_F_XDP_MB flag for bpf_attr to notify the kernel the eBPF
program fully supports xdp multi-buffer.
- reject a non XDP multi-buffer program if the driver is running in
XDP multi-buffer mode.
Changes since v14:
- intrudce bpf_xdp_pointer utility routine and
bpf_xdp_load_bytes/bpf_xdp_store_bytes helpers
- drop bpf_xdp_adjust_data helper
- drop xdp_frags_truesize in skb_shared_info
- explode bpf_xdp_mb_adjust_tail in bpf_xdp_mb_increase_tail and
bpf_xdp_mb_shrink_tail
Changes since v13:
- use u32 for xdp_buff/xdp_frame flags field
- rename xdp_frags_tsize in xdp_frags_truesize
- fixed comments
Changes since v12:
- fix bpf_xdp_adjust_data helper for single-buffer use case
- return -EFAULT in bpf_xdp_adjust_{head,tail} in case the data pointers are not
properly reset
- collect ACKs from John
Changes since v11:
- add missing static to bpf_xdp_get_buff_len_proto structure
- fix bpf_xdp_adjust_data helper when offset is smaller than linear area length.
Changes since v10:
- move xdp->data to the requested payload offset instead of to the beginning of
the fragment in bpf_xdp_adjust_data()
Changes since v9:
- introduce bpf_xdp_adjust_data helper and related selftest
- add xdp_frags_size and xdp_frags_tsize fields in skb_shared_info
- introduce xdp_update_skb_shared_info utility routine in ordere to not reset
frags array in skb_shared_info converting from a xdp_buff/xdp_frame to a skb
- simplify bpf_xdp_copy routine
Changes since v8:
- add proper dma unmapping if XDP_TX fails on mvneta for a xdp multi-buff
- switch back to skb_shared_info implementation from previous xdp_shared_info
one
- avoid using a bietfield in xdp_buff/xdp_frame since it introduces performance
regressions. Tested now on 10G NIC (ixgbe) to verify there are no performance
penalties for regular codebase
- add bpf_xdp_get_buff_len helper and remove frame_length field in xdp ctx
- add data_len field in skb_shared_info struct
- introduce XDP_FLAGS_FRAGS_PF_MEMALLOC flag
Changes since v7:
- rebase on top of bpf-next
- fix sparse warnings
- improve comments for frame_length in include/net/xdp.h
Changes since v6:
- the main difference respect to previous versions is the new approach proposed
by Eelco to pass full length of the packet to eBPF layer in XDP context
- reintroduce multi-buff support to eBPF kself-tests
- reintroduce multi-buff support to bpf_xdp_adjust_tail helper
- introduce multi-buffer support to bpf_xdp_copy helper
- rebase on top of bpf-next
Changes since v5:
- rebase on top of bpf-next
- initialize mb bit in xdp_init_buff() and drop per-driver initialization
- drop xdp->mb initialization in xdp_convert_zc_to_xdp_frame()
- postpone introduction of frame_length field in XDP ctx to another series
- minor changes
Changes since v4:
- rebase ontop of bpf-next
- introduce xdp_shared_info to build xdp multi-buff instead of using the
skb_shared_info struct
- introduce frame_length in xdp ctx
- drop previous bpf helpers
- fix bpf_xdp_adjust_tail for xdp multi-buff
- introduce xdp multi-buff self-tests for bpf_xdp_adjust_tail
- fix xdp_return_frame_bulk for xdp multi-buff
Changes since v3:
- rebase ontop of bpf-next
- add patch 10/13 to copy back paged data from a xdp multi-buff frame to
userspace buffer for xdp multi-buff selftests
Changes since v2:
- add throughput measurements
- drop bpf_xdp_adjust_mb_header bpf helper
- introduce selftest for xdp multibuffer
- addressed comments on bpf_xdp_get_frags_count
- introduce xdp multi-buff support to cpumaps
Changes since v1:
- Fix use-after-free in xdp_return_{buff/frame}
- Introduce bpf helpers
- Introduce xdp_mb sample program
- access skb_shared_info->nr_frags only on the last fragment
Changes since RFC:
- squash multi-buffer bit initialization in a single patch
- add mvneta non-linear XDP buff support for tx side
[0] https://netdevconf.info/0x14/session.html?talk-the-path-to-tcp-4k-mtu-and-rx-zerocopy
[1] https://github.com/xdp-project/xdp-project/blob/master/areas/core/xdp-multi-buffer01-design.org
[2] https://netdevconf.info/0x14/session.html?tutorial-add-XDP-support-to-a-NIC-driver (XDPmulti-buffers section)
Eelco Chaudron (3):
bpf: add multi-buff support to the bpf_xdp_adjust_tail() API
bpf: add multi-buffer support to xdp copy helpers
bpf: selftests: update xdp_adjust_tail selftest to include
multi-buffer
Lorenzo Bianconi (19):
net: skbuff: add size metadata to skb_shared_info for xdp
xdp: introduce flags field in xdp_buff/xdp_frame
net: mvneta: update mb bit before passing the xdp buffer to eBPF layer
net: mvneta: simplify mvneta_swbm_add_rx_fragment management
net: xdp: add xdp_update_skb_shared_info utility routine
net: marvell: rely on xdp_update_skb_shared_info utility routine
xdp: add multi-buff support to xdp_return_{buff/frame}
net: mvneta: add multi buffer support to XDP_TX
bpf: introduce BPF_F_XDP_MB flag in prog_flags loading the ebpf
program
net: mvneta: enable jumbo frames if the loaded XDP program support mb
bpf: introduce bpf_xdp_get_buff_len helper
bpf: move user_size out of bpf_test_init
bpf: introduce multibuff support to bpf_prog_test_run_xdp()
bpf: test_run: add xdp_shared_info pointer in bpf_test_finish
signature
libbpf: Add SEC name for xdp_mb programs
net: xdp: introduce bpf_xdp_pointer utility routine
bpf: selftests: introduce bpf_xdp_{load,store}_bytes selftest
bpf: selftests: add CPUMAP/DEVMAP selftests for xdp multi-buff
xdp: disable XDP_REDIRECT for xdp multi-buff
Toke Hoiland-Jorgensen (1):
bpf: generalise tail call map compatibility check
drivers/net/ethernet/marvell/mvneta.c | 204 ++++++++-----
include/linux/bpf.h | 32 +-
include/linux/skbuff.h | 1 +
include/net/xdp.h | 108 ++++++-
include/uapi/linux/bpf.h | 30 ++
kernel/bpf/arraymap.c | 4 +-
kernel/bpf/core.c | 28 +-
kernel/bpf/cpumap.c | 8 +-
kernel/bpf/devmap.c | 3 +-
kernel/bpf/syscall.c | 25 +-
kernel/trace/bpf_trace.c | 3 +
net/bpf/test_run.c | 115 ++++++--
net/core/filter.c | 275 +++++++++++++++++-
net/core/xdp.c | 78 ++++-
tools/include/uapi/linux/bpf.h | 30 ++
tools/lib/bpf/libbpf.c | 8 +
.../bpf/prog_tests/xdp_adjust_frags.c | 81 ++++++
.../bpf/prog_tests/xdp_adjust_tail.c | 118 ++++++++
.../selftests/bpf/prog_tests/xdp_bpf2bpf.c | 151 +++++++---
.../bpf/prog_tests/xdp_cpumap_attach.c | 65 ++++-
.../bpf/prog_tests/xdp_devmap_attach.c | 56 ++++
.../bpf/progs/test_xdp_adjust_tail_grow.c | 10 +-
.../bpf/progs/test_xdp_adjust_tail_shrink.c | 32 +-
.../selftests/bpf/progs/test_xdp_bpf2bpf.c | 2 +-
.../bpf/progs/test_xdp_update_frags.c | 42 +++
.../bpf/progs/test_xdp_with_cpumap_helpers.c | 6 +
.../progs/test_xdp_with_cpumap_mb_helpers.c | 27 ++
.../bpf/progs/test_xdp_with_devmap_helpers.c | 7 +
.../progs/test_xdp_with_devmap_mb_helpers.c | 27 ++
29 files changed, 1364 insertions(+), 212 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_adjust_frags.c
create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_update_frags.c
create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_with_cpumap_mb_helpers.c
create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_with_devmap_mb_helpers.c
--
2.31.1
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:32
Introduce xdp_frags_size field in skb_shared_info data structure
to store xdp_buff/xdp_frame frame paged size (xdp_frags_size will
be used in xdp multi-buff support). In order to not increase
skb_shared_info size we will use a hole due to skb_shared_info
alignment.
Acked-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Jesper Dangaard Brouer <redacted>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
include/linux/skbuff.h | 1 +
1 file changed, 1 insertion(+)
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:40
Update multi-buffer bit (mb) in xdp_buff to notify XDP/eBPF layer and
XDP remote drivers if this is a "non-linear" XDP buffer. Access
skb_shared_info only if xdp_buff mb is set in order to avoid possible
cache-misses.
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 23 ++++++++++++++++++-----
1 file changed, 18 insertions(+), 5 deletions(-)
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:40
Introduce flags field in xdp_frame and xdp_buffer data structures
to define additional buffer features. At the moment the only
supported buffer feature is multi-buffer bit (mb). Multi-buffer bit
is used to specify if this is a linear buffer (mb = 0) or a multi-buffer
frame (mb = 1). In the latter case the driver is expected to initialize
the skb_shared_info structure at the end of the first buffer to link
together subsequent buffers belonging to the same frame.
Acked-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Jesper Dangaard Brouer <redacted>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
include/net/xdp.h | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
@@ -74,13 +78,30 @@ struct xdp_buff {structxdp_rxq_info*rxq;structxdp_txq_info*txq;u32frame_sz;/* frame size to deduce data_hard_end/reserved tailroom*/+u32flags;/* supported values defined in xdp_buff_flags */};+static__always_inlineboolxdp_buff_is_mb(structxdp_buff*xdp)+{+return!!(xdp->flags&XDP_FLAGS_MULTI_BUFF);+}++static__always_inlinevoidxdp_buff_set_mb(structxdp_buff*xdp)+{+xdp->flags|=XDP_FLAGS_MULTI_BUFF;+}++static__always_inlinevoidxdp_buff_clear_mb(structxdp_buff*xdp)+{+xdp->flags&=~XDP_FLAGS_MULTI_BUFF;+}+static__always_inlinevoidxdp_init_buff(structxdp_buff*xdp,u32frame_sz,structxdp_rxq_info*rxq){xdp->frame_sz=frame_sz;xdp->rxq=rxq;+xdp->flags=0;}static__always_inlinevoid
@@ -122,8 +143,14 @@ struct xdp_frame {*/structxdp_mem_infomem;structnet_device*dev_rx;/* used by cpumap */+u32flags;/* supported values defined in xdp_buff_flags */};+static__always_inlineboolxdp_frame_is_mb(structxdp_frame*frame)+{+return!!(frame->flags&XDP_FLAGS_MULTI_BUFF);+}+#define XDP_BULK_QUEUE_SIZE 16structxdp_frame_bulk{intcount;
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:41
Relying on xdp mb bit, remove skb_shared_info structure allocated on the
stack in mvneta_rx_swbm routine and simplify mvneta_swbm_add_rx_fragment
accessing skb_shared_info in the xdp_buff structure directly. There is no
performance penalty in this approach since mvneta_swbm_add_rx_fragment
is run just for multi-buff use-case.
Acked-by: John Fastabend <john.fastabend@gmail.com>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 42 ++++++++++-----------------
1 file changed, 15 insertions(+), 27 deletions(-)
@@ -2307,16 +2309,6 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp,}else{page_pool_put_full_page(rxq->page_pool,page,true);}--/* last fragment */-if(len==*size){-structskb_shared_info*sinfo;--sinfo=xdp_get_shared_info_from_buff(xdp);-sinfo->nr_frags=xdp_sinfo->nr_frags;-memcpy(sinfo->frags,xdp_sinfo->frags,-sinfo->nr_frags*sizeof(skb_frag_t));-}*size-=len;}
@@ -2364,7 +2356,6 @@ static int mvneta_rx_swbm(struct napi_struct *napi,{intrx_proc=0,rx_todo,refill,size=0;structnet_device*dev=pp->dev;-structskb_shared_infosinfo;structmvneta_statsps={};structbpf_prog*xdp_prog;u32desc_status,frame_sz;
@@ -2373,8 +2364,6 @@ static int mvneta_rx_swbm(struct napi_struct *napi,xdp_init_buff(&xdp_buf,PAGE_SIZE,&rxq->xdp_rxq);xdp_buf.data_hard_start=NULL;-sinfo.nr_frags=0;-/* Get number of received packets */rx_todo=mvneta_rxq_busy_desc_num_get(pp,rxq);
@@ -2416,7 +2405,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,}mvneta_swbm_add_rx_fragment(pp,rx_desc,rxq,&xdp_buf,-&size,&sinfo,page);+&size,page);}/* Middle or Last descriptor */if(!(rx_status&MVNETA_RXD_LAST_DESC))
@@ -2424,7 +2413,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,continue;if(size){-mvneta_xdp_put_buff(pp,rxq,&xdp_buf,&sinfo,-1);+mvneta_xdp_put_buff(pp,rxq,&xdp_buf,-1);gotonext;}
@@ -2436,7 +2425,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi,if(IS_ERR(skb)){structmvneta_pcpu_stats*stats=this_cpu_ptr(pp->stats);-mvneta_xdp_put_buff(pp,rxq,&xdp_buf,&sinfo,-1);+mvneta_xdp_put_buff(pp,rxq,&xdp_buf,-1);u64_stats_update_begin(&stats->syncp);stats->es.skb_alloc_error++;
@@ -2453,11 +2442,10 @@ static int mvneta_rx_swbm(struct napi_struct *napi,napi_gro_receive(napi,skb);next:xdp_buf.data_hard_start=NULL;-sinfo.nr_frags=0;}if(xdp_buf.data_hard_start)-mvneta_xdp_put_buff(pp,rxq,&xdp_buf,&sinfo,-1);+mvneta_xdp_put_buff(pp,rxq,&xdp_buf,-1);if(ps.xdp_redirect)xdp_do_flush_map();
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:43
Introduce xdp_update_skb_shared_info routine to update frags array
metadata in skb_shared_info data structure converting to a skb from
a xdp_buff or xdp_frame.
According to the current skb_shared_info architecture in
xdp_frame/xdp_buff and to the xdp multi-buff support, there is
no need to run skb_add_rx_frag() and reset frags array converting the buffer
to a skb since the frag array will be in the same position for xdp_buff/xdp_frame
and for the skb, we just need to update memory metadata.
Introduce XDP_FLAGS_PF_MEMALLOC flag in xdp_buff_flags in order to mark
the xdp_buff or xdp_frame as under memory-pressure if pages of the frags array
are under memory pressure. Doing so we can avoid looping over all fragments in
xdp_update_skb_shared_info routine. The driver is expected to set the
flag constructing the xdp_buffer using xdp_buff_set_frag_pfmemalloc
utility routine.
Rely on xdp_update_skb_shared_info in __xdp_build_skb_from_frame routine
converting the multi-buff xdp_frame to a skb after performing a XDP_REDIRECT.
Acked-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Jesper Dangaard Brouer <redacted>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
include/net/xdp.h | 33 ++++++++++++++++++++++++++++++++-
net/core/xdp.c | 12 ++++++++++++
2 files changed, 44 insertions(+), 1 deletion(-)
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:49
Rely on xdp_update_skb_shared_info routine in order to avoid
resetting frags array in skb_shared_info structure building
the skb in mvneta_swbm_build_skb(). Frags array is expected to
be initialized by the receiving driver building the xdp_buff
and here we just need to update memory metadata.
Acked-by: John Fastabend <john.fastabend@gmail.com>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 23 ++++++++++-------------
1 file changed, 10 insertions(+), 13 deletions(-)
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:56
Take into account if the received xdp_buff/xdp_frame is non-linear
recycling/returning the frame memory to the allocator or into
xdp_frame_bulk.
Acked-by: John Fastabend <john.fastabend@gmail.com>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
include/net/xdp.h | 18 ++++++++++++++--
net/core/xdp.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 69 insertions(+), 3 deletions(-)
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:57
Enable the capability to receive jumbo frames even if the interface is
running in XDP mode if the loaded program declare to properly support
xdp multi-buff. At same time reject a xdp program not supporting xdp
multi-buffer if the driver is running in xdp multi-buffer mode.
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
drivers/net/ethernet/marvell/mvneta.c | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
@@ -3758,8 +3759,11 @@ static int mvneta_change_mtu(struct net_device *dev, int mtu)mtu=ALIGN(MVNETA_RX_PKT_SIZE(mtu),8);}-if(pp->xdp_prog&&mtu>MVNETA_MAX_RX_BUF_SIZE){-netdev_info(dev,"Illegal MTU value %d for XDP mode\n",mtu);+if(prog&&!prog->aux->xdp_mb&&mtu>MVNETA_MAX_RX_BUF_SIZE){+netdev_info(dev,+"Illegal MTU %d for XDP prog without multi-buf\n",+mtu);+return-EINVAL;}
@@ -4457,8 +4461,9 @@ static int mvneta_xdp_setup(struct net_device *dev, struct bpf_prog *prog,structmvneta_port*pp=netdev_priv(dev);structbpf_prog*old_prog;-if(prog&&dev->mtu>MVNETA_MAX_RX_BUF_SIZE){-NL_SET_ERR_MSG_MOD(extack,"MTU too large for XDP");+if(prog&&!prog->aux->xdp_mb&&dev->mtu>MVNETA_MAX_RX_BUF_SIZE){+NL_SET_ERR_MSG_MOD(extack,+"prog does not support XDP multi-buff");return-EOPNOTSUPP;}
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:58
From: Eelco Chaudron <echaudro@redhat.com>
This change adds support for tail growing and shrinking for XDP multi-buff.
When called on a multi-buffer packet with a grow request, it will work
on the last fragment of the packet. So the maximum grow size is the
last fragments tailroom, i.e. no new buffer will be allocated.
A XDP mb capable driver is expected to set frag_size in xdp_rxq_info data
structure to notify the XDP core the fragment size. frag_size set to 0 is
interpreted by the XDP core as tail growing is not allowed.
Introduce __xdp_rxq_info_reg utility routine to initialize frag_size field.
When shrinking, it will work from the last fragment, all the way down to
the base buffer depending on the shrinking size. It's important to mention
that once you shrink down the fragment(s) are freed, so you can not grow
again to the original size.
Co-developed-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Eelco Chaudron <echaudro@redhat.com>
---
drivers/net/ethernet/marvell/mvneta.c | 3 +-
include/net/xdp.h | 16 ++++++-
net/core/filter.c | 67 +++++++++++++++++++++++++++
net/core/xdp.c | 12 +++--
4 files changed, 90 insertions(+), 8 deletions(-)
@@ -4962,6 +4962,12 @@ union bpf_attr {***-ENOENT**if*task->mm*isNULL,ornovmacontains*addr*.***-EBUSY**iffailedtotrylockmmap_lock.***-EINVAL**forinvalid**flags**.+*+*u64bpf_xdp_get_buff_len(structxdp_buff*xdp_md)+*Description+*Getthetotalsizeofagivenxdpbuff(linearandpagedarea)+*Return+*Thetotalsizeofagivenxdpbuffer.*/#define __BPF_FUNC_MAPPER(FN) \FN(unspec),\
@@ -5145,6 +5151,7 @@ union bpf_attr {FN(skc_to_unix_sock),\FN(kallsyms_lookup_name),\FN(find_vma),\+FN(xdp_get_buff_len),\/* *//* integer value in 'imm' field of BPF_CALL instruction selects which helper
@@ -4962,6 +4962,12 @@ union bpf_attr {***-ENOENT**if*task->mm*isNULL,ornovmacontains*addr*.***-EBUSY**iffailedtotrylockmmap_lock.***-EINVAL**forinvalid**flags**.+*+*u64bpf_xdp_get_buff_len(structxdp_buff*xdp_md)+*Description+*Getthetotalsizeofagivenxdpbuff(linearandpagedarea)+*Return+*Thetotalsizeofagivenxdpbuffer.*/#define __BPF_FUNC_MAPPER(FN) \FN(unspec),\
@@ -5145,6 +5151,7 @@ union bpf_attr {FN(skc_to_unix_sock),\FN(kallsyms_lookup_name),\FN(find_vma),\+FN(xdp_get_buff_len),\/* *//* integer value in 'imm' field of BPF_CALL instruction selects which helper
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:51:59
Introduce BPF_F_XDP_MB and the related field in bpf_prog_aux in order to
notify the driver the loaded program support xdp multi-buffer.
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
include/linux/bpf.h | 1 +
include/uapi/linux/bpf.h | 5 +++++
kernel/bpf/syscall.c | 4 +++-
tools/include/uapi/linux/bpf.h | 5 +++++
4 files changed, 14 insertions(+), 1 deletion(-)
@@ -1111,6 +1111,11 @@ enum bpf_link_type {*/#define BPF_F_SLEEPABLE (1U << 4)+/* If BPF_F_XDP_MB is used in BPF_PROG_LOAD command, the loaded program+*fullysupportxdpmulti-buffer+*/+#define BPF_F_XDP_MB (1U << 5)+/* When BPF ldimm64's insn[0].src_reg != 0 then this can have*thefollowingextensions:*
@@ -1111,6 +1111,11 @@ enum bpf_link_type {*/#define BPF_F_SLEEPABLE (1U << 4)+/* If BPF_F_XDP_MB is used in BPF_PROG_LOAD command, the loaded program+*fullysupportxdpmulti-buffer+*/+#define BPF_F_XDP_MB (1U << 5)+/* When BPF ldimm64's insn[0].src_reg != 0 then this can have*thefollowingextensions:*
@@ -10,11 +10,20 @@ struct meta {intpkt_len;};+structtest_ctx_s{+boolpassed;+intpkt_size;+};++structtest_ctx_stest_ctx;+staticvoidon_sample(void*ctx,intcpu,void*data,__u32size){-intduration=0;structmeta*meta=(structmeta*)data;structipv4_packet*trace_pkt_v4=data+sizeof(*meta);+unsignedchar*raw_pkt=data+sizeof(*meta);+structtest_ctx_s*tst_ctx=ctx;+intduration=0;if(CHECK(size<sizeof(pkt_v4)+sizeof(*meta),"check_size","size %u < %zu\n",
@@ -25,25 +34,114 @@ static void on_sample(void *ctx, int cpu, void *data, __u32 size)"meta->ifindex = %d\n",meta->ifindex))return;-if(CHECK(meta->pkt_len!=sizeof(pkt_v4),"check_meta_pkt_len",-"meta->pkt_len = %zd\n",sizeof(pkt_v4)))+if(CHECK(meta->pkt_len!=tst_ctx->pkt_size,"check_meta_pkt_len",+"meta->pkt_len = %d\n",tst_ctx->pkt_size))return;if(CHECK(memcmp(trace_pkt_v4,&pkt_v4,sizeof(pkt_v4)),"check_packet_content","content not the same\n"))return;-*(bool*)ctx=true;+if(meta->pkt_len>sizeof(pkt_v4)){+for(inti=0;i<(meta->pkt_len-sizeof(pkt_v4));i++){+if(raw_pkt[i+sizeof(pkt_v4)]!=(unsignedchar)i){+CHECK(true,"check_packet_content",+"byte %zu does not match %u != %u\n",+i+sizeof(pkt_v4),+raw_pkt[i+sizeof(pkt_v4)],+(unsignedchar)i);+break;+}+}+}++tst_ctx->passed=true;}-voidtest_xdp_bpf2bpf(void)+#define BUF_SZ 9000++staticintrun_xdp_bpf2bpf_pkt_size(intpkt_fd,structperf_buffer*pb,+structtest_xdp_bpf2bpf*ftrace_skel,+intpkt_size){__u32duration=0,retval,size;-charbuf[128];+__u8*buf,*buf_in;+interr,ret=0;++if(pkt_size>BUF_SZ||pkt_size<sizeof(pkt_v4))+return-EINVAL;++buf_in=malloc(BUF_SZ);+if(CHECK(!buf_in,"buf_in malloc()","error:%s\n",strerror(errno)))+return-ENOMEM;++buf=malloc(BUF_SZ);+if(CHECK(!buf,"buf malloc()","error:%s\n",strerror(errno))){+ret=-ENOMEM;+gotofree_buf_in;+}++test_ctx.passed=false;+test_ctx.pkt_size=pkt_size;++memcpy(buf_in,&pkt_v4,sizeof(pkt_v4));+if(pkt_size>sizeof(pkt_v4)){+for(inti=0;i<(pkt_size-sizeof(pkt_v4));i++)+buf_in[i+sizeof(pkt_v4)]=i;+}++/* Run test program */+err=bpf_prog_test_run(pkt_fd,1,buf_in,pkt_size,+buf,&size,&retval,&duration);++if(CHECK(err||retval!=XDP_PASS||size!=pkt_size,+"ipv4","err %d errno %d retval %d size %d\n",+err,errno,retval,size)){+ret=err?err:-EINVAL;+gotofree_buf;+}++/* Make sure bpf_xdp_output() was triggered and it sent the expected+*datatotheperfringbuffer.+*/+err=perf_buffer__poll(pb,100);+if(CHECK(err<=0,"perf_buffer__poll","err %d\n",err)){+ret=-EINVAL;+gotofree_buf;+}++if(CHECK_FAIL(!test_ctx.passed)){+ret=-EINVAL;+gotofree_buf;+}++/* Verify test results */+if(CHECK(ftrace_skel->bss->test_result_fentry!=if_nametoindex("lo"),+"result","fentry failed err %llu\n",+ftrace_skel->bss->test_result_fentry)){+ret=-EINVAL;+gotofree_buf;+}++if(CHECK(ftrace_skel->bss->test_result_fexit!=XDP_PASS,"result",+"fexit failed err %llu\n",+ftrace_skel->bss->test_result_fexit))+ret=-EINVAL;++free_buf:+free(buf);+free_buf_in:+free(buf_in);++returnret;+}++voidtest_xdp_bpf2bpf(void)+{interr,pkt_fd,map_fd;-boolpassed=false;-structiphdr*iph=(void*)buf+sizeof(structethhdr);-structiptnl_infovalue4={.family=AF_INET};+__u32duration=0;+intpkt_sizes[]={sizeof(pkt_v4),1024,4100,8200};+structiptnl_infovalue4={.family=AF_INET6};structtest_xdp*pkt_skel=NULL;structtest_xdp_bpf2bpf*ftrace_skel=NULL;structvipkey4={.protocol=6,.family=AF_INET};
@@ -85,39 +183,14 @@ void test_xdp_bpf2bpf(void)gotoout;/* Set up perf buffer */-pb=perf_buffer__new(bpf_map__fd(ftrace_skel->maps.perf_buf_map),1,-on_sample,NULL,&passed,NULL);+pb=perf_buffer__new(bpf_map__fd(ftrace_skel->maps.perf_buf_map),8,+on_sample,NULL,&test_ctx,NULL);if(!ASSERT_OK_PTR(pb,"perf_buf__new"))gotoout;-/* Run test program */-err=bpf_prog_test_run(pkt_fd,1,&pkt_v4,sizeof(pkt_v4),-buf,&size,&retval,&duration);--if(CHECK(err||retval!=XDP_TX||size!=74||-iph->protocol!=IPPROTO_IPIP,"ipv4",-"err %d errno %d retval %d size %d\n",-err,errno,retval,size))-gotoout;--/* Make sure bpf_xdp_output() was triggered and it sent the expected-*datatotheperfringbuffer.-*/-err=perf_buffer__poll(pb,100);-if(CHECK(err<0,"perf_buffer__poll","err %d\n",err))-gotoout;--CHECK_FAIL(!passed);--/* Verify test results */-if(CHECK(ftrace_skel->bss->test_result_fentry!=if_nametoindex("lo"),-"result","fentry failed err %llu\n",-ftrace_skel->bss->test_result_fentry))-gotoout;--CHECK(ftrace_skel->bss->test_result_fexit!=XDP_TX,"result",-"fexit failed err %llu\n",ftrace_skel->bss->test_result_fexit);-+for(inti=0;i<ARRAY_SIZE(pkt_sizes);i++)+run_xdp_bpf2bpf_pkt_size(pkt_fd,pb,ftrace_skel,+pkt_sizes[i]);out:if(pb)perf_buffer__free(pb);
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:52:02
Introduce the capability to allocate a xdp multi-buff in
bpf_prog_test_run_xdp routine. This is a preliminary patch to introduce
the selftests for new xdp multi-buff ebpf helpers
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
net/bpf/test_run.c | 58 +++++++++++++++++++++++++++++++++++-----------
1 file changed, 45 insertions(+), 13 deletions(-)
@@ -758,16 +758,16 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,unionbpf_attr__user*uattr){u32tailroom=SKB_DATA_ALIGN(sizeof(structskb_shared_info));-u32headroom=XDP_PACKET_HEADROOM;u32size=kattr->test.data_size_in;+u32headroom=XDP_PACKET_HEADROOM;+u32retval,duration,max_data_sz;u32repeat=kattr->test.repeat;structnetdev_rx_queue*rxqueue;+structskb_shared_info*sinfo;structxdp_buffxdp={};-u32retval,duration;+inti,ret=-EINVAL;structxdp_md*ctx;-u32max_data_sz;void*data;-intret=-EINVAL;if(prog->expected_attach_type==BPF_XDP_DEVMAP||prog->expected_attach_type==BPF_XDP_CPUMAP)
@@ -787,27 +787,60 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,headroom-=ctx->data;}-/* XDP have extra tailroom as (most) drivers use full page */max_data_sz=4096-headroom-tailroom;+size=min_t(u32,size,max_data_sz);-data=bpf_test_init(kattr,kattr->test.data_size_in,-max_data_sz,headroom,tailroom);+data=bpf_test_init(kattr,size,max_data_sz,headroom,tailroom);if(IS_ERR(data)){ret=PTR_ERR(data);gotofree_ctx;}rxqueue=__netif_get_rx_queue(current->nsproxy->net_ns->loopback_dev,0);-xdp_init_buff(&xdp,headroom+max_data_sz+tailroom,-&rxqueue->xdp_rxq);+rxqueue->xdp_rxq.frag_size=headroom+max_data_sz+tailroom;+xdp_init_buff(&xdp,rxqueue->xdp_rxq.frag_size,&rxqueue->xdp_rxq);xdp_prepare_buff(&xdp,data,headroom,size,true);+sinfo=xdp_get_shared_info_from_buff(&xdp);ret=xdp_convert_md_to_buff(ctx,&xdp);if(ret)gotofree_data;+if(unlikely(kattr->test.data_size_in>size)){+void__user*data_in=u64_to_user_ptr(kattr->test.data_in);++while(size<kattr->test.data_size_in){+structpage*page;+skb_frag_t*frag;+intdata_len;++page=alloc_page(GFP_KERNEL);+if(!page){+ret=-ENOMEM;+gotoout;+}++frag=&sinfo->frags[sinfo->nr_frags++];+__skb_frag_set_page(frag,page);++data_len=min_t(int,kattr->test.data_size_in-size,+PAGE_SIZE);+skb_frag_size_set(frag,data_len);++if(copy_from_user(page_address(page),data_in+size,+data_len)){+ret=-EFAULT;+gotoout;+}+sinfo->xdp_frags_size+=data_len;+size+=data_len;+}+xdp_buff_set_mb(&xdp);+}+if(repeat>1)bpf_prog_change_xdp(NULL,prog);+ret=bpf_test_run(prog,&xdp,repeat,&retval,&duration,true);/* We convert the xdp_buff back to an xdp_md before checking the return*codesothereferencecountofanyheldnetdevicewillbedecremented
@@ -817,10 +850,7 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,if(ret)gotoout;-if(xdp.data_meta!=data+headroom||-xdp.data_end!=xdp.data_meta+size)-size=xdp.data_end-xdp.data_meta;-+size=xdp.data_end-xdp.data_meta+sinfo->xdp_frags_size;ret=bpf_test_finish(kattr,uattr,xdp.data_meta,size,retval,duration);if(!ret)
@@ -831,6 +861,8 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,if(repeat>1)bpf_prog_change_xdp(prog,NULL);free_data:+for(i=0;i<sinfo->nr_frags;i++)+__free_page(skb_frag_page(&sinfo->frags[i]));kfree(data);free_ctx:kfree(ctx);
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:52:06
XDP_REDIRECT is not fully supported yet for xdp multi-buff since not
all XDP capable drivers can map non-linear xdp_frame in ndo_xdp_xmit
so disable it for the moment.
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
net/core/filter.c | 7 +++++++
1 file changed, 7 insertions(+)
@@ -130,6 +130,120 @@ static void test_xdp_adjust_tail_grow2(void)bpf_object__close(obj);}+voidtest_xdp_adjust_mb_tail_shrink(void)+{+constchar*file="./test_xdp_adjust_tail_shrink.o";+__u32duration,retval,size,exp_size;+structbpf_object*obj;+interr,prog_fd;+__u8*buf;++/* For the individual test cases, the first byte in the packet+*indicateswhichtestwillberun.+*/++err=bpf_prog_load(file,BPF_PROG_TYPE_XDP,&obj,&prog_fd);+if(CHECK_FAIL(err))+return;++buf=malloc(9000);+if(CHECK(!buf,"malloc()","error:%s\n",strerror(errno)))+return;++memset(buf,0,9000);++/* Test case removing 10 bytes from last frag, NOT freeing it */+exp_size=8990;/* 9000 - 10 */+err=bpf_prog_test_run(prog_fd,1,buf,9000,+buf,&size,&retval,&duration);++CHECK(err||retval!=XDP_TX||size!=exp_size,+"9k-10b","err %d errno %d retval %d[%d] size %d[%u]\n",+err,errno,retval,XDP_TX,size,exp_size);++/* Test case removing one of two pages, assuming 4K pages */+buf[0]=1;+exp_size=4900;/* 9000 - 4100 */+err=bpf_prog_test_run(prog_fd,1,buf,9000,+buf,&size,&retval,&duration);++CHECK(err||retval!=XDP_TX||size!=exp_size,+"9k-1p","err %d errno %d retval %d[%d] size %d[%u]\n",+err,errno,retval,XDP_TX,size,exp_size);++/* Test case removing two pages resulting in a non mb xdp_buff */+buf[0]=2;+exp_size=800;/* 9000 - 8200 */+err=bpf_prog_test_run(prog_fd,1,buf,9000,+buf,&size,&retval,&duration);++CHECK(err||retval!=XDP_TX||size!=exp_size,+"9k-2p","err %d errno %d retval %d[%d] size %d[%u]\n",+err,errno,retval,XDP_TX,size,exp_size);++free(buf);++bpf_object__close(obj);+}++voidtest_xdp_adjust_mb_tail_grow(void)+{+constchar*file="./test_xdp_adjust_tail_grow.o";+__u32duration,retval,size,exp_size;+structbpf_object*obj;+interr,i,prog_fd;+__u8*buf;++err=bpf_prog_load(file,BPF_PROG_TYPE_XDP,&obj,&prog_fd);+if(CHECK_FAIL(err))+return;++buf=malloc(16384);+if(CHECK(!buf,"malloc()","error:%s\n",strerror(errno)))+return;++/* Test case add 10 bytes to last frag */+memset(buf,1,16384);+size=9000;+exp_size=size+10;+err=bpf_prog_test_run(prog_fd,1,buf,size,+buf,&size,&retval,&duration);++CHECK(err||retval!=XDP_TX||size!=exp_size,+"9k+10b","err %d retval %d[%d] size %d[%u]\n",+err,retval,XDP_TX,size,exp_size);++for(i=0;i<9000;i++)+CHECK(buf[i]!=1,"9k+10b-old",+"Old data not all ok, offset %i is failing [%u]!\n",+i,buf[i]);++for(i=9000;i<9010;i++)+CHECK(buf[i]!=0,"9k+10b-new",+"New data not all ok, offset %i is failing [%u]!\n",+i,buf[i]);++for(i=9010;i<16384;i++)+CHECK(buf[i]!=1,"9k+10b-untouched",+"Unused data not all ok, offset %i is failing [%u]!\n",+i,buf[i]);++/* Test a too large grow */+memset(buf,1,16384);+size=9001;+exp_size=size;+err=bpf_prog_test_run(prog_fd,1,buf,size,+buf,&size,&retval,&duration);++CHECK(err||retval!=XDP_DROP||size!=exp_size,+"9k+10b","err %d retval %d[%d] size %d[%u]\n",+err,retval,XDP_TX,size,exp_size);++free(buf);++bpf_object__close(obj);+}+voidtest_xdp_adjust_tail(void){if(test__start_subtest("xdp_adjust_tail_shrink"))
@@ -7,11 +7,10 @@ int _xdp_adjust_tail_grow(struct xdp_md *xdp){void*data_end=(void*)(long)xdp->data_end;void*data=(void*)(long)xdp->data;-unsignedintdata_len;+intdata_len=bpf_xdp_get_buff_len(xdp);intoffset=0;/* Data length determine test case */-data_len=data_end-data;if(data_len==54){/* sizeof(pkt_v4) */offset=4096;/* test too large offset */
@@ -20,7 +19,12 @@ int _xdp_adjust_tail_grow(struct xdp_md *xdp)}elseif(data_len==64){offset=128;}elseif(data_len==128){-offset=4096-256-320-data_len;/* Max tail grow 3520 */+/* Max tail grow 3520 */+offset=4096-256-320-data_len;+}elseif(data_len==9000){+offset=10;+}elseif(data_len==9001){+offset=4096;}else{returnXDP_ABORTED;/* No matching test */}
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:52:22
From: Toke Hoiland-Jorgensen <redacted>
The check for tail call map compatibility ensures that tail calls only
happen between maps of the same type. To ensure backwards compatibility for
XDP multi-buffer we need a similar type of check for cpumap and devmap
programs, so move the state from bpf_array_aux into bpf_map, add xdp_mb to
the check, and apply the same check to cpumap and devmap.
Co-developed-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: Toke Hoiland-Jorgensen <redacted>
---
include/linux/bpf.h | 31 ++++++++++++++++++++-----------
kernel/bpf/arraymap.c | 4 +---
kernel/bpf/core.c | 28 ++++++++++++++--------------
kernel/bpf/cpumap.c | 8 +++++---
kernel/bpf/devmap.c | 3 ++-
kernel/bpf/syscall.c | 21 +++++++++++----------
6 files changed, 53 insertions(+), 42 deletions(-)
@@ -194,6 +194,18 @@ struct bpf_map {structwork_structwork;structmutexfreeze_mutex;u64writecnt;/* writable mmap cnt; protected by freeze_mutex */++/* 'Ownership' of program-containing map is claimed by the first program+*thatisgoingtousethismaporbythefirstprogramwhichFDis+*storedinthemaptomakesurethatallcallersandcalleeshavethe+*sameprogtype,JITedflagandxdp_mbflag.+*/+struct{+spinlock_tlock;+enumbpf_prog_typetype;+booljited;+boolxdp_mb;+}owner;};staticinlineboolmap_value_has_spin_lock(conststructbpf_map*map)
@@ -930,16 +942,6 @@ struct bpf_prog_aux {};structbpf_array_aux{-/* 'Ownership' of prog array is claimed by the first program that-*isgoingtousethismaporbythefirstprogramwhichFDis-*storedinthemaptomakesurethatallcallersandcalleeshave-*thesameprogtypeandJITedflag.-*/-struct{-spinlock_tlock;-enumbpf_prog_typetype;-booljited;-}owner;/* Programs with direct jumps into programs part of this array. */structlist_headpoke_progs;structbpf_map*map;
@@ -1821,28 +1821,30 @@ static unsigned int __bpf_prog_ret0_warn(const void *ctx,}#endif-boolbpf_prog_array_compatible(structbpf_array*array,-conststructbpf_prog*fp)+boolbpf_prog_map_compatible(structbpf_map*map,+conststructbpf_prog*fp){boolret;if(fp->kprobe_override)returnfalse;-spin_lock(&array->aux->owner.lock);--if(!array->aux->owner.type){+spin_lock(&map->owner.lock);+if(!map->owner.type){/* There's no owner yet where we could check for*compatibility.*/-array->aux->owner.type=fp->type;-array->aux->owner.jited=fp->jited;+map->owner.type=fp->type;+map->owner.jited=fp->jited;+map->owner.xdp_mb=fp->aux->xdp_mb;ret=true;}else{-ret=array->aux->owner.type==fp->type&&-array->aux->owner.jited==fp->jited;+ret=map->owner.type==fp->type&&+map->owner.jited==fp->jited&&+map->owner.xdp_mb==fp->aux->xdp_mb;}-spin_unlock(&array->aux->owner.lock);+spin_unlock(&map->owner.lock);+returnret;}
@@ -1854,13 +1856,11 @@ static int bpf_check_tail_call(const struct bpf_prog *fp)mutex_lock(&aux->used_maps_mutex);for(i=0;i<aux->used_map_cnt;i++){structbpf_map*map=aux->used_maps[i];-structbpf_array*array;-if(map->map_type!=BPF_MAP_TYPE_PROG_ARRAY)+if(!map_type_contains_progs(map))continue;-array=container_of(map,structbpf_array,map);-if(!bpf_prog_array_compatible(array,fp)){+if(!bpf_prog_map_compatible(map,fp)){ret=-EINVAL;gotoout;}
@@ -54,6 +55,68 @@ void serial_test_xdp_cpumap_attach(void)err=bpf_map_update_elem(map_fd,&idx,&val,0);ASSERT_NEQ(err,0,"Add non-BPF_XDP_CPUMAP program to cpumap entry");+/* try to attach BPF_XDP_CPUMAP multi-buff program when we have already+*loadedalegacyXDPprogramonthemap+*/+idx=1;+val.qsize=192;+val.bpf_prog.fd=bpf_program__fd(skel->progs.xdp_dummy_cm_mb);+err=bpf_map_update_elem(map_fd,&idx,&val,0);+ASSERT_NEQ(err,0,+"Add BPF_XDP_CPUMAP multi-buff program to cpumap entry");+out_close:test_xdp_with_cpumap_helpers__destroy(skel);}++voidtest_xdp_with_cpumap_mb_helpers(void)+{+structtest_xdp_with_cpumap_mb_helpers*skel;+structbpf_prog_infoinfo={};+__u32len=sizeof(info);+structbpf_cpumap_valval={+.qsize=192,+};+interr,mb_prog_fd,map_fd;+__u32idx=0;++skel=test_xdp_with_cpumap_mb_helpers__open_and_load();+if(!ASSERT_OK_PTR(skel,"test_xdp_with_cpumap_helpers__open_and_load"))+return;++mb_prog_fd=bpf_program__fd(skel->progs.xdp_dummy_cm_mb);+map_fd=bpf_map__fd(skel->maps.cpu_map);+err=bpf_obj_get_info_by_fd(mb_prog_fd,&info,&len);+if(!ASSERT_OK(err,"bpf_obj_get_info_by_fd"))+gotoout_close;++val.bpf_prog.fd=mb_prog_fd;+err=bpf_map_update_elem(map_fd,&idx,&val,0);+ASSERT_OK(err,"Add program to cpumap entry");++err=bpf_map_lookup_elem(map_fd,&idx,&val);+ASSERT_OK(err,"Read cpumap entry");+ASSERT_EQ(info.id,val.bpf_prog.id,+"Match program id to cpumap entry prog_id");++/* try to attach BPF_XDP_CPUMAP program when we have already+*loadedamulti-buffXDPprogramonthemap+*/+idx=1;+val.qsize=192;+val.bpf_prog.fd=bpf_program__fd(skel->progs.xdp_dummy_cm);+err=bpf_map_update_elem(map_fd,&idx,&val,0);+ASSERT_NEQ(err,0,"Add BPF_XDP_CPUMAP program to cpumap entry");++out_close:+test_xdp_with_cpumap_mb_helpers__destroy(skel);+}++voidserial_test_xdp_cpumap_attach(void)+{+if(test__start_subtest("CPUMAP with programs in entries"))+test_xdp_with_cpumap_helpers();++if(test__start_subtest("CPUMAP with multi-buff programs in entries"))+test_xdp_with_cpumap_mb_helpers();+}
@@ -56,6 +57,16 @@ static void test_xdp_with_devmap_helpers(void)err=bpf_map_update_elem(map_fd,&idx,&val,0);ASSERT_NEQ(err,0,"Add non-BPF_XDP_DEVMAP program to devmap entry");+/* try to attach BPF_XDP_DEVMAP multi-buff program when we have already+*loadedalegacyXDPprogramonthemap+*/+idx=1;+val.ifindex=1;+val.bpf_prog.fd=bpf_program__fd(skel->progs.xdp_dummy_dm_mb);+err=bpf_map_update_elem(map_fd,&idx,&val,0);+ASSERT_NEQ(err,0,+"Add BPF_XDP_DEVMAP multi-buff program to devmap entry");+out_close:test_xdp_with_devmap_helpers__destroy(skel);}
@@ -71,12 +82,57 @@ static void test_neg_xdp_devmap_helpers(void)}}+voidtest_xdp_with_devmap_mb_helpers(void)+{+structtest_xdp_with_devmap_mb_helpers*skel;+structbpf_prog_infoinfo={};+structbpf_devmap_valval={+.ifindex=IFINDEX_LO,+};+__u32len=sizeof(info);+interr,dm_fd_mb,map_fd;+__u32idx=0;++skel=test_xdp_with_devmap_mb_helpers__open_and_load();+if(!ASSERT_OK_PTR(skel,"test_xdp_with_devmap_helpers__open_and_load"))+return;++dm_fd_mb=bpf_program__fd(skel->progs.xdp_dummy_dm_mb);+map_fd=bpf_map__fd(skel->maps.dm_ports);+err=bpf_obj_get_info_by_fd(dm_fd_mb,&info,&len);+if(!ASSERT_OK(err,"bpf_obj_get_info_by_fd"))+gotoout_close;++val.bpf_prog.fd=dm_fd_mb;+err=bpf_map_update_elem(map_fd,&idx,&val,0);+ASSERT_OK(err,"Add multi-buff program to devmap entry");++err=bpf_map_lookup_elem(map_fd,&idx,&val);+ASSERT_OK(err,"Read devmap entry");+ASSERT_EQ(info.id,val.bpf_prog.id,+"Match program id to devmap entry prog_id");++/* try to attach BPF_XDP_DEVMAP program when we have already loaded a+*multi-buffXDPprogramonthemap+*/+idx=1;+val.ifindex=1;+val.bpf_prog.fd=bpf_program__fd(skel->progs.xdp_dummy_dm);+err=bpf_map_update_elem(map_fd,&idx,&val,0);+ASSERT_NEQ(err,0,"Add BPF_XDP_DEVMAP program to devmap entry");++out_close:+test_xdp_with_devmap_mb_helpers__destroy(skel);+}voidserial_test_xdp_devmap_attach(void){if(test__start_subtest("DEVMAP with programs in entries"))test_xdp_with_devmap_helpers();+if(test__start_subtest("DEVMAP with multi-buff programs in entries"))+test_xdp_with_devmap_mb_helpers();+if(test__start_subtest("Verifier check of DEVMAP programs"))test_neg_xdp_devmap_helpers();}
@@ -0,0 +1,27 @@+// SPDX-License-Identifier: GPL-2.0+#include<linux/bpf.h>+#include<bpf/bpf_helpers.h>++struct{+__uint(type,BPF_MAP_TYPE_DEVMAP);+__uint(key_size,sizeof(__u32));+__uint(value_size,sizeof(structbpf_devmap_val));+__uint(max_entries,4);+}dm_portsSEC(".maps");++/* valid program on DEVMAP entry via SEC name;+*hasaccesstoegressandingressifindex+*/+SEC("xdp_devmap/map_prog")+intxdp_dummy_dm(structxdp_md*ctx)+{+returnXDP_PASS;+}++SEC("xdp_devmap_mb/mp_map_prog")+intxdp_dummy_dm_mb(structxdp_md*ctx)+{+returnXDP_PASS;+}++char_license[]SEC("license")="GPL";
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:52:25
Rely on data_size_in in bpf_test_init routine signature. This is a
preliminary patch to introduce xdp multi-buff selftest
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
net/bpf/test_run.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
@@ -581,7 +580,8 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,if(kattr->test.flags||kattr->test.cpu)return-EINVAL;-data=bpf_test_init(kattr,size,NET_SKB_PAD+NET_IP_ALIGN,+data=bpf_test_init(kattr,kattr->test.data_size_in,+size,NET_SKB_PAD+NET_IP_ALIGN,SKB_DATA_ALIGN(sizeof(structskb_shared_info)));if(IS_ERR(data))returnPTR_ERR(data);
@@ -790,7 +790,8 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,/* XDP have extra tailroom as (most) drivers use full page */max_data_sz=4096-headroom-tailroom;-data=bpf_test_init(kattr,max_data_sz,headroom,tailroom);+data=bpf_test_init(kattr,kattr->test.data_size_in,+max_data_sz,headroom,tailroom);if(IS_ERR(data)){ret=PTR_ERR(data);gotofree_ctx;
@@ -876,7 +877,7 @@ int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog,if(size<ETH_HLEN)return-EINVAL;-data=bpf_test_init(kattr,size,0,0);+data=bpf_test_init(kattr,kattr->test.data_size_in,size,0,0);if(IS_ERR(data))returnPTR_ERR(data);
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:52:28
Similar to skb_header_pointer, introduce bpf_xdp_pointer utility routine
to return a pointer to a given position in the xdp_buff if the requested
area (offset + len) is contained in a contiguous memory area otherwise it
will be copied in a bounce buffer provided by the caller.
Similar to the tc counterpart, introduce the two following xdp helpers:
- bpf_xdp_load_bytes
- bpf_xdp_store_bytes
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
include/uapi/linux/bpf.h | 18 +++++
net/core/filter.c | 129 +++++++++++++++++++++++++++++++++
tools/include/uapi/linux/bpf.h | 18 +++++
3 files changed, 165 insertions(+)
@@ -4968,6 +4968,22 @@ union bpf_attr {*Getthetotalsizeofagivenxdpbuff(linearandpagedarea)*Return*Thetotalsizeofagivenxdpbuffer.+*+*longbpf_xdp_load_bytes(structxdp_buff*xdp_md,u32offset,void*buf,u32len)+*Description+*Thishelperisprovidedasaneasywaytoloaddatafroma+*xdpbuffer.Itcanbeusedtoload*len*bytesfrom*offset*from+*theframeassociatedto*xdp_md*,intothebufferpointedby+**buf*.+*Return+*0onsuccess,oranegativeerrorincaseoffailure.+*+*longbpf_xdp_store_bytes(structxdp_buff*xdp_md,u32offset,void*buf,u32len)+*Description+*Store*len*bytesfrombuffer*buf*intotheframe+*associatedto*xdp_md*,at*offset*.+*Return+*0onsuccess,oranegativeerrorincaseoffailure.*/#define __BPF_FUNC_MAPPER(FN) \FN(unspec),\
@@ -5152,6 +5168,8 @@ union bpf_attr {FN(kallsyms_lookup_name),\FN(find_vma),\FN(xdp_get_buff_len),\+FN(xdp_load_bytes),\+FN(xdp_store_bytes),\/* *//* integer value in 'imm' field of BPF_CALL instruction selects which helper
@@ -4968,6 +4968,22 @@ union bpf_attr {*Getthetotalsizeofagivenxdpbuff(linearandpagedarea)*Return*Thetotalsizeofagivenxdpbuffer.+*+*longbpf_xdp_load_bytes(structxdp_buff*xdp_md,u32offset,void*buf,u32len)+*Description+*Thishelperisprovidedasaneasywaytoloaddatafroma+*xdpbuffer.Itcanbeusedtoload*len*bytesfrom*offset*from+*theframeassociatedto*xdp_md*,intothebufferpointedby+**buf*.+*Return+*0onsuccess,oranegativeerrorincaseoffailure.+*+*longbpf_xdp_store_bytes(structxdp_buff*xdp_md,u32offset,void*buf,u32len)+*Description+*Store*len*bytesfrombuffer*buf*intotheframe+*associatedto*xdp_md*,at*offset*.+*Return+*0onsuccess,oranegativeerrorincaseoffailure.*/#define __BPF_FUNC_MAPPER(FN) \FN(unspec),\
@@ -5152,6 +5168,8 @@ union bpf_attr {FN(kallsyms_lookup_name),\FN(find_vma),\FN(xdp_get_buff_len),\+FN(xdp_load_bytes),\+FN(xdp_store_bytes),\/* *//* integer value in 'imm' field of BPF_CALL instruction selects which helper
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-15 22:52:30
introduce xdp_shared_info pointer in bpf_test_finish signature in order
to copy back paged data from a xdp multi-buff frame to userspace buffer
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
net/bpf/test_run.c | 48 +++++++++++++++++++++++++++++++++++++---------
1 file changed, 39 insertions(+), 9 deletions(-)
@@ -683,7 +712,8 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,/* bpf program can never convert linear skb to non-linear */if(WARN_ON_ONCE(skb_is_nonlinear(skb)))size=skb_headlen(skb);-ret=bpf_test_finish(kattr,uattr,skb->data,size,retval,duration);+ret=bpf_test_finish(kattr,uattr,skb->data,NULL,size,retval,+duration);if(!ret)ret=bpf_ctx_finish(kattr,uattr,ctx,sizeof(struct__sk_buff));
@@ -851,8 +881,8 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,gotoout;size=xdp.data_end-xdp.data_meta+sinfo->xdp_frags_size;-ret=bpf_test_finish(kattr,uattr,xdp.data_meta,size,retval,-duration);+ret=bpf_test_finish(kattr,uattr,xdp.data_meta,sinfo,size,+retval,duration);if(!ret)ret=bpf_ctx_finish(kattr,uattr,ctx,sizeof(structxdp_md));
@@ -944,8 +974,8 @@ int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog,if(ret<0)gotoout;-ret=bpf_test_finish(kattr,uattr,&flow_keys,sizeof(flow_keys),-retval,duration);+ret=bpf_test_finish(kattr,uattr,&flow_keys,NULL,+sizeof(flow_keys),retval,duration);if(!ret)ret=bpf_ctx_finish(kattr,uattr,user_ctx,sizeof(structbpf_flow_keys));
@@ -1049,7 +1079,7 @@ int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog, const union bpf_attr *katuser_ctx->cookie=sock_gen_cookie(ctx.selected_sk);}-ret=bpf_test_finish(kattr,uattr,NULL,0,retval,duration);+ret=bpf_test_finish(kattr,uattr,NULL,NULL,0,retval,duration);if(!ret)ret=bpf_ctx_finish(kattr,uattr,user_ctx,sizeof(*user_ctx));
Looks good. We can adjust/extend the exact semantics as drivers add
support. Important part is we should be safe from negligence.
Acked-by: Jakub Kicinski <kuba@kernel.org>
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-11-16 15:02:16
On Mon, 15 Nov 2021 23:33:07 +0100 Lorenzo Bianconi wrote:
From: Eelco Chaudron <echaudro@redhat.com>
This patch adds support for multi-buffer for the following helpers:
- bpf_xdp_output()
- bpf_perf_event_output()
Signed-off-by: Eelco Chaudron <echaudro@redhat.com>
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
is this missing
else
offset -= headsize;
?
I'm struggling to understand this. Say
headsize = 400
frag[0].size = 200
offset = 500
len = 50
we enter the loop having missed the previous if...
+ for (i = 0; i < sinfo->nr_frags; i++) {
+ skb_frag_t *frag = &sinfo->frags[i];
+ u32 frag_size = skb_frag_size(frag);
+
+ if (count >= len)
+ break;
+
+ if (offset < frame_offset + frag_size) {
500 < 400 + 200 => true
+ int size = min_t(int, frag_size - offset, len - count);
is this missing
else
offset -= headsize;
?
I'm struggling to understand this. Say
headsize = 400
frag[0].size = 200
offset = 500
len = 50
we enter the loop having missed the previous if...
quoted
+ for (i = 0; i < sinfo->nr_frags; i++) {
+ skb_frag_t *frag = &sinfo->frags[i];
+ u32 frag_size = skb_frag_size(frag);
+
+ if (count >= len)
+ break;
+
+ if (offset < frame_offset + frag_size) {
500 < 400 + 200 => true
quoted
+ int size = min_t(int, frag_size - offset, len - count);
size = min(200 - 500, 50 - 0)
size = -300 ??
ack, you are right. Sorry for the issue.
I did not trigger the problem with xdp-mb self-tests since we will not run
bpf_xdp_copy_buf() in this specific case, but just the memcpy()
(but what you reported is a bug and must be fixed). I will add more
self-tests.
Moreover, reviewing the code I guess we can just update bpf_xdp_copy() for our case.
Something like:
static void bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off,
void *buf, unsigned long len, bool flush)
{
unsigned long ptr_len, ptr_off = 0;
skb_frag_t *next_frag, *end_frag;
struct skb_shared_info *sinfo;
void *src, *dst;
u8 *ptr_buf;
if (likely(xdp->data_end - xdp->data >= off + len)) {
src = flush ? buf : xdp->data + off;
dst = flush ? xdp->data + off : buf;
memcpy(dst, src, len);
return;
}
sinfo = xdp_get_shared_info_from_buff(xdp);
end_frag = &sinfo->frags[sinfo->nr_frags];
next_frag = &sinfo->frags[0];
ptr_len = xdp->data_end - xdp->data;
ptr_buf = xdp->data;
while (true) {
if (off < ptr_off + ptr_len) {
unsigned long copy_off = off - ptr_off;
unsigned long copy_len = min(len, ptr_len - copy_off);
src = flush ? buf : ptr_buf + copy_off;
dst = flush ? ptr_buf + copy_off : buf;
memcpy(dst, src, copy_len);
off += copy_len;
len -= copy_len;
buf += copy_len;
}
if (!len || next_frag == end_frag)
break;
ptr_off += ptr_len;
ptr_buf = skb_frag_address(next_frag);
ptr_len = skb_frag_size(next_frag);
next_frag++;
}
}
...
static unsigned long bpf_xdp_copy(void *dst, const void *ctx,
unsigned long off, unsigned long len)
{
struct xdp_buff *xdp = (struct xdp_buff *)ctx;
bpf_xdp_copy_buf(xdp, off, dst, len, false);
return 0;
}
What do you think?
Regards,
Lorenzo
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-11-17 00:32:03
On Wed, 17 Nov 2021 01:12:41 +0100 Lorenzo Bianconi wrote:
ack, you are right. Sorry for the issue.
I did not trigger the problem with xdp-mb self-tests since we will not run
bpf_xdp_copy_buf() in this specific case, but just the memcpy()
(but what you reported is a bug and must be fixed). I will add more
self-tests.
Moreover, reviewing the code I guess we can just update bpf_xdp_copy() for our case.
Something like:
Seems reasonable. We could probably play some tricks with double
pointers to avoid the ternary operator being re-evaluated for each
chunk. But even if it's faster it is probably not worth the ugliness
of the code.
From: Lorenzo Bianconi <lorenzo@kernel.org> Date: 2021-11-17 11:13:03
On Wed, 17 Nov 2021 01:12:41 +0100 Lorenzo Bianconi wrote:
quoted
ack, you are right. Sorry for the issue.
I did not trigger the problem with xdp-mb self-tests since we will not run
bpf_xdp_copy_buf() in this specific case, but just the memcpy()
(but what you reported is a bug and must be fixed). I will add more
self-tests.
Moreover, reviewing the code I guess we can just update bpf_xdp_copy() for our case.
Something like:
Seems reasonable. We could probably play some tricks with double
pointers to avoid the ternary operator being re-evaluated for each
chunk. But even if it's faster it is probably not worth the ugliness
of the code.
ack, moreover I guess the slowest operation here is the mempcy().
I added a new self-test to cover the case where buf is across frag0
and frag1.
I will wait for some more feedbacks and then I will post a new version.
Thanks.
Regards,
Lorenzo