Re: [PATCH v2 net-next] net: add sysctl to toggle napi_consume_skb() alien skb defer
From: Paolo Abeni <pabeni@redhat.com>
Date: 2026-03-31 11:39:44
On 3/27/26 4:33 PM, Jason Xing wrote:
Commit e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs")
defers freeing of alien SKBs (alloc_cpu != current cpu) via
skb_attempt_defer_free() on the TX completion path to reduce cross-NUMA
SLUB spinlock contention to improve multi-queue UDP workloads.
However, this unconditionally impacts the napi_skb_cache fast recycle
path for single-flow / few-flow workloads (e.g. AF_XDP benchmarks[1]):
when the TX completion NAPI CPU differs from the SKB allocation CPU,
SKBs are deferred instead of being returned to the local napi_skb_cache,
forcing RX allocations back to the slow slab path.
The existing net.core.skb_defer_max=0 could disable this, but it is a
global switch that also disables the defer mechanism in TCP/UDP/MPTCP
recvmsg paths, losing its positive SLUB locality benefits there. AF_XDP
can co-exist with other protocols. That's the reason why I gave up
reusing skb_defer_disable_key. Besides, if the defer path is disabled,
that means TCP/UDP/MPTCP in process path will trigger directly freeing
skb with enabling/disabling bottom half(in kfree_skb_napi_cache())
which could affect others. So my thinking is not to touch this path.
Add a dedicated sysctl net.core.napi_consume_skb_defer backed by a
static key to selectively control the alien skb defer feature. Let
users decide which is the best fit for their own requirements.
This patch also avoids touching local_bh* pair(in kfree_skb_napi_cache())
to minimize the overhead.
[1]: taskset -c 0 ./xdpsock -i enp2s0f1 -q 1 -t -S -s 64
1) sysctl -w net.core.napi_consume_skb_defer=1 (as default)
sock0@enp2s0f1:1 txonly xdp-skb
pps pkts 1.00
rx 0 0
tx 1,851,950 20,397,952
2)sysctl -w net.core.napi_consume_skb_defer=0
sock0@enp2s0f1:1 txonly xdp-skb
pps pkts 1.00
rx 0 0
tx 1,985,067 25,530,432
For AF_XDP scenario, it turns out to be around 6.6% improvement.I'm not a big fan of multiple tunables around the same feature, but possibly here the use-case extends beyond AF_XDP right? Do you observe some measurable positive delta even with UDP? Possibly even with TCP, when the bottleneck is the sender? More data would be helpful.
quoted hunk ↗ jump to hunk
Signed-off-by: Jason Xing <redacted> --- V2 Link: https://lore.kernel.org/all/20260326144249.97213-1-kerneljasonxing@gmail.com/ (local) 1. reuse proc_do_static_key() (Eric) 2. add doc (Stan) --- Documentation/admin-guide/sysctl/net.rst | 13 +++++++++++++ net/core/net-sysfs.h | 1 + net/core/skbuff.c | 5 ++++- net/core/sysctl_net_core.c | 7 +++++++ 4 files changed, 25 insertions(+), 1 deletion(-)diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst index 0724a793798f..42e06f93306f 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst@@ -368,6 +368,19 @@ by the cpu which allocated them. Default: 128 +napi_consume_skb_defer +---------------------- +When set to 1 (default), napi_consume_skb() defers freeing SKBs whose +allocation CPU differs from the current CPU via skb_attempt_defer_free(). +This reduces cross-NUMA SLUB spinlock contention for multi-queue workloads. + +Setting this to 0 disables the defer path in napi_consume_skb() only, +allowing SKBs to be returned to the local napi_skb_cache immediately. +This can benefit single-flow or few-flow workloads (e.g. AF_XDP TX) +where the defer detour hurts the fast recycle path.
I think it should be clarified that skb_defer_max takes priority, i.e. no defer with skb_defer_max == 0. Also it would be great if you could additional extend skb_defer_max documentation noting that 0 disable such feature. Thanks, Paolo