Remove the per-SKB skb->pp_recycle bitfield and skb_mark_for_recycle()
helper, and reference-count netmems directly by their underlying type
via netmem_is_pp() inside netmem_inc_ref() and netmem_dec_ref().
Establish the invariant that pp_ref_count is only incremented by callers
that already hold an active pp_ref_count (>= 1) on that netmem, and a
raw page_ref_count is never converted into a pp_ref_count:
- Detach page_pool pages in tcp_zerocopy_receive() before mapping them
into userspace via vm_insert_pages(), so userspace VMAs and subsequent
GUP callers (such as MSG_ZEROCOPY or vmsplice()) never observe an
active page_pool page.
- Fall back to copying in tls_push_data() when MSG_SPLICE_PAGES
encounters a page_pool page, since kTLS device offload record->frags[]
tracks pages via raw struct page references.
- Keep page_pool_unref_netmem(), napi_pp_get_page(), napi_pp_put_page(),
and __page_pool_put_page() free of 0 -> 1 pp_ref_count conversion
workarounds or extra barriers, preserving the non-atomic
page_pool_unref_netmem() fast path unchanged.
bench_page_pool_simple results (5-iteration interleaved UML runs):
- no-softirq-page_pool01 (fast-path cache recycle): 40 cycles (20.3 ns)
baseline vs 30-31 cycles (15.3 ns) patched (~9-10 cycles / -24.7%
improvement over baseline: using __netmem_to_page() in
page_pool_netmem_is_exclusively_owned() and
__page_pool_page_can_be_recycled() after checking netmem_is_net_iov()
eliminates two redundant WARN_ON_ONCE() expansions and out-of-line
netmem_to_page() calls per recycle).
- no-softirq-page_pool02 (ptr_ring recycle): 81 cycles (40.7 ns)
baseline vs 74 cycles (37.2 ns) patched (~7 cycles / -8.6%
improvement over baseline).
- no-softirq-page_pool03 (slow page-allocator path with elevated
page_ref_count): 458-461 cycles (230.0 ns) baseline vs 454-455 cycles
(227.6 ns) patched (~5 cycles / -1.0% improvement over baseline).
This series simplifies SKB and netmem reference counting:
- Reclaims 1 bit from struct sk_buff (skb->pp_recycle).
- Unifies reference counting of SKB fragment netmems and regular netmems
under netmem_inc_ref() and netmem_dec_ref(), eliminating the separate
skb_netmem_ref(), skb_netmem_unref(), and skb_mark_for_recycle() APIs.
- Removes 95 net lines across 65 files (339 insertions, 434 deletions).
1/5: net: skbuff: skip managed zerocopy frags in skb_frag_ref()
2/5: net: skbuff: refcount spliced page_pool frags via skb_pipe_buf_ops
3/5: tcp: detach page_pool pages before mapping in tcp_zerocopy_receive()
4/5: net: remove skb->pp_recycle and consolidate netmem refcounting
5/5: net: remove skb_mark_for_recycle()
v1 -> v2:
- Patch 2/5: Check page_ref_count(compound_head(buf->page)) <= 0 with
WARN_ON_ONCE() in skb_pipe_buf_get() to guard against refcount
overflow, and fall back to copying in tls_push_data() when
MSG_SPLICE_PAGES encounters a page_pool page so kTLS device offload
never converts a raw page_ref_count into a pp_ref_count.
- Patch 3/5: Replace zerocopy_fill_skb_from_iter() refcount conversion
with detaching page_pool pages up front in tcp_zerocopy_receive()
before vm_insert_pages().
- Patch 4/5: Keep net/tls/ record->frags[] and fallback scatterlists on
get_page()/put_page() to match mlx5 tx_sync_info_get(), and restore
page_pool_unref_netmem(), napi_pp_get_page(), napi_pp_put_page(), and
__page_pool_put_page() to their clean baseline non-RMW fast paths.
- v1:
https://lore.kernel.org/netdev/20261103211424.127627-1-almasrymina@google.com/ (local)
Notes on Medium/Low Sashiko findings intentionally skipped:
- Patch 1/5 (Low): iptfs_output_queued() skb_zcopy_managed() check does
not carry a separate Fixes: tag as it is part of this net-next
refactoring series.
- Patch 3/5 (Medium): can_map_frag() requires page_pool_ref_count == 1,
so drivers holding an undrained fragment bias (e.g. mlx5 mpwqe, fbnic)
fall back to copying in tcp_zerocopy_receive() (documented in commit
message; detaching requires holding the sole pp_ref_count).
- Patch 3/5 (Low): page_pool_ref_count() naming kept to mirror
page_ref_count().
- Patch 4/5 (Medium): skb_irq_freeable() in netpoll_zap_completion_queue()
does not check netmem_is_pp() (pre-existing for forwarded pp_recycle=1
RX SKBs; documented in commit message Caveats).
- Patch 4/5 (Low): napi_pp_get_page() and napi_pp_put_page() remain
out-of-line exported functions in net/core/skbuff.c.
Mina Almasry (5):
net: skbuff: skip managed zerocopy frags in skb_frag_ref()
net: skbuff: refcount spliced page_pool frags via skb_pipe_buf_ops
tcp: detach page_pool pages before mapping in tcp_zerocopy_receive()
net: remove skb->pp_recycle and consolidate netmem refcounting
net: remove skb_mark_for_recycle()
Documentation/networking/page_pool.rst | 1 -
drivers/net/ethernet/airoha/airoha_eth.c | 1 -
drivers/net/ethernet/alibaba/eea/eea_rx.c | 3 -
.../net/ethernet/broadcom/asp2/bcmasp_intf.c | 1 -
.../net/ethernet/broadcom/bnge/bnge_txrx.c | 5 -
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 -
.../net/ethernet/broadcom/genet/bcmgenet.c | 2 -
drivers/net/ethernet/cisco/enic/enic_rq.c | 1 -
drivers/net/ethernet/engleder/tsnep_main.c | 2 -
drivers/net/ethernet/freescale/fec_main.c | 1 -
drivers/net/ethernet/google/gve/gve_rx_dqo.c | 9 -
.../net/ethernet/hisilicon/hibmcge/hbg_txrx.c | 1 -
.../net/ethernet/hisilicon/hns3/hns3_enet.c | 6 -
.../net/ethernet/huawei/hinic3/hinic3_rx.c | 1 -
drivers/net/ethernet/intel/iavf/iavf_txrx.c | 2 -
drivers/net/ethernet/marvell/mvneta.c | 2 -
.../net/ethernet/marvell/mvpp2/mvpp2_main.c | 2 -
.../marvell/octeontx2/nic/otx2_txrx.c | 1 -
drivers/net/ethernet/mediatek/mtk_eth_soc.c | 1 -
drivers/net/ethernet/mellanox/mlx4/en_rx.c | 1 -
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 8 -
drivers/net/ethernet/mellanox/mlxsw/pci.c | 2 -
.../ethernet/microchip/lan966x/lan966x_fdma.c | 2 -
.../microchip/sparx5/lan969x/lan969x_fdma.c | 1 -
drivers/net/ethernet/microsoft/mana/mana_en.c | 3 -
.../net/ethernet/mucse/rnpgbe/rnpgbe_lib.c | 1 -
.../net/ethernet/pensando/ionic/ionic_txrx.c | 2 -
.../net/ethernet/realtek/rtase/rtase_main.c | 1 -
drivers/net/ethernet/renesas/ravb_main.c | 2 -
drivers/net/ethernet/socionext/netsec.c | 1 -
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 1 -
drivers/net/ethernet/tehuti/tn40.c | 1 -
drivers/net/ethernet/ti/am65-cpsw-nuss.c | 1 -
drivers/net/ethernet/ti/cpsw.c | 2 -
drivers/net/ethernet/ti/cpsw_new.c | 2 -
drivers/net/ethernet/ti/icssg/icssg_common.c | 1 -
drivers/net/ethernet/wangxun/libwx/wx_lib.c | 2 -
drivers/net/veth.c | 3 +-
drivers/net/virtio_net.c | 16 +-
drivers/net/vmxnet3/vmxnet3_xdp.c | 1 -
drivers/net/wireless/mediatek/mt76/dma.c | 1 -
.../net/wireless/mediatek/mt76/mt7996/mac.c | 1 -
drivers/net/wireless/mediatek/mt76/npu.c | 1 -
drivers/net/wireless/mediatek/mt76/usb.c | 1 -
drivers/net/xen-netfront.c | 1 -
include/linux/mm.h | 2 +-
include/linux/skbuff.h | 12 +-
include/linux/skbuff_ref.h | 60 +---
include/net/netmem.h | 37 ++-
include/net/page_pool/helpers.h | 25 +-
net/core/datagram.c | 2 +
net/core/gro.c | 9 -
net/core/net_test.c | 274 ++++++++----------
net/core/netmem_priv.h | 21 +-
net/core/page_pool.c | 6 +-
net/core/skbuff.c | 74 ++---
net/core/xdp.c | 13 +-
net/ipv4/esp4.c | 11 +-
net/ipv4/tcp.c | 44 ++-
net/ipv6/esp6.c | 11 +-
net/tls/tls_device.c | 8 +-
net/tls/tls_device_fallback.c | 2 +-
net/wireless/util.c | 31 +-
net/xfrm/trace_iptfs.h | 6 +-
net/xfrm/xfrm_iptfs.c | 16 +-
65 files changed, 339 insertions(+), 434 deletions(-)
base-commit: 7b6ab4dbc156e2f96419715130b3d5738a306c93
--
2.56.0.455.g6c5a503b38-goog