Hi!
AF_XDP zero copy needs a second RX allocator in every driver. This
series makes UMEM a page-pool memory provider instead: a page_pool
driver opts in, and keeps its refill, recycling and DMA sync code.
fbnic is the first user. Patches 1-7 are core, 8-11 fbnic.
Since the RFC [RFC], net_iov is a first class netmem, the core owns
AF_XDP setup, and a refill contract replaces the NAPI kicks. The fixes
are out.
Changes since the RFC:
- A readable net_iov has an address. netmem_address() and
skb_frag_address() find it through struct net_iov_area, which now
holds niov_shift. xdp_frag_address() is gone. (Mina, Stanislav)
- PP_FLAG_ALLOW_UNREADABLE_NETMEM keeps its meaning. A pool takes a
readable provider only with the new PP_FLAG_MEM_PROVIDER. (Mina)
- The core installs the provider at bind, through the queue API, as
for devmem and io_uring. A driver sets
netdev_queue_mgmt_ops.xsk_page_pool, and checks chunk size,
headroom and multi-buffer mode in XDP_VALIDATE_XSK_POOL. MTU and
XDP program changes of a bound device get the same check.
(Stanislav)
- No NAPI kicks. Drivers report the end of each refill with
page_pool_refill_done(), and the core publishes the queue's NAPI
for provider and socket wakeups, so no ndo_xsk_wakeup. (Mina)
- Page pools declare the RX buffers they supply, header or payload.
A per-queue generation tells new pools from old ones in the restart
check.
- MP_CAP_RETAIN says whether the stack may keep provider buffers.
The stack copies the others on XDP_PASS, frame conversion and
non-XSKMAP redirects. Frame copies are linear, so multi-buffer
frames to devmap and cpumap are dropped.
- xdp_buff does not grow. The head netmem and shared info live in
driver memory, reached through a union with txq.
- The buffer pool allocates xdp_buff_xsk only for copy mode and
driver zero copy: 128 bytes less per chunk. (Stanislav)
- With XDP_USE_NEED_WAKEUP the FILL flag stays set. Setting it only
when starved races with userspace.
- fbnic is split in four: XDP redirect, netmem heads and XDP_TX,
unsplittable buffers, zero copy.
- The RFC's fixes are out. The fbnic one is upstream, and the skb
head sizing is Clement's fix [1]. Other fixes found on the way go
to net separately. (Stanislav, Maciej)
Unchanged: aligned chunks only (4 KiB on fbnic), and generic XDP and
CPUMAP cannot feed a provider socket.
fbnic, one queue, 66-byte packets: XSK RX 21.2 Mpps (17.0 with epoll),
XSK TX 28.9 Mpps, XDP_DROP costs 1.9%, 30 instructions per packet,
against net-next. Tested on fbnic hardware and the fbnic QEMU model.
Still open from the RFC: how should userspace learn the chunk
constraints before bind?
netkit queue leases and bnxt come later.
Björn
[RFC] https://lore.kernel.org/netdev/20261002190018.696925-1-bjorn@kernel.org/ (local)
[1] https://lore.kernel.org/bpf/20260629072300.1664622-1-clecigne@google.com/ (local)
Björn Töpel (11):
net: Get skb fragment addresses through netmem
net: Add memory provider capabilities
xdp: Copy buffers with restricted lifetime
xsk: Allocate UMEM chunks through page_pool
xsk: Install UMEM page pool providers in the core
page_pool: Coordinate refill with memory providers
xsk: Deliver provider chunks without copying
eth: fbnic: Support XDP redirect
eth: fbnic: Use netmem for RX headers and XDP_TX
eth: fbnic: Support RX buffers it cannot split
eth: fbnic: Support AF_XDP page pool zero copy
Documentation/networking/af_xdp.rst | 64 +-
Documentation/networking/netmem.rst | 15 +
Documentation/networking/xsk-tx-metadata.rst | 3 +-
.../net/ethernet/meta/fbnic/fbnic_netdev.c | 22 +-
drivers/net/ethernet/meta/fbnic/fbnic_txrx.c | 601 ++++++++++++---
drivers/net/ethernet/meta/fbnic/fbnic_txrx.h | 28 +
drivers/net/netkit.c | 3 +-
include/linux/netdevice.h | 8 +
include/linux/skbuff.h | 22 +-
include/net/netdev_queues.h | 12 +
include/net/netdev_rx_queue.h | 8 +
include/net/netmem.h | 41 +-
include/net/page_pool/helpers.h | 38 +-
include/net/page_pool/memory_provider.h | 21 +
include/net/page_pool/types.h | 33 +-
include/net/xdp.h | 94 ++-
include/net/xdp_sock.h | 4 +
include/net/xdp_sock_drv.h | 45 ++
include/net/xsk_buff_pool.h | 3 +
io_uring/zcrx.c | 2 +
net/core/dev.c | 30 +-
net/core/dev.h | 8 +
net/core/devmem.c | 9 +-
net/core/devmem.h | 4 +-
net/core/filter.c | 20 +-
net/core/netdev_config.c | 2 +
net/core/netdev_rx_queue.c | 143 +++-
net/core/page_pool.c | 100 ++-
net/core/page_pool_user.c | 25 +-
net/core/xdp.c | 78 +-
net/ethtool/rings.c | 7 +-
net/xdp/Kconfig | 1 +
net/xdp/Makefile | 2 +-
net/xdp/xsk.c | 86 ++-
net/xdp/xsk.h | 15 +
net/xdp/xsk_buff_pool.c | 72 +-
net/xdp/xsk_pp.c | 697 ++++++++++++++++++
net/xdp/xsk_queue.h | 18 +-
38 files changed, 2159 insertions(+), 225 deletions(-)
create mode 100644 net/xdp/xsk_pp.c
base-commit: 2d916a8c307c31b433318509bf0311d52d509f09
--
2.56.0