On Sun, Nov 08, 2026 at 10:13:03AM -0500, Willem de Bruijn wrote:
Pushing back on another design choice: the UAPI change. That both cannot be
undone, and is fragile and laborious, requiring manual admin arithmetic.
Thanks for the valuable feedback.
On fragile: a wrong value never changes the packet. If it is too
small, the push reallocates, as it does today. If it is too large,
locally generated skbs on that device use more memory. So if the value
drifts from what the programs push, only performance changes. The
software that loads the encapsulating program knows the size and can
set the value at the same time.
On manual arithmetic: user space already does this for netkit. Commit
b9ed315d3c4c ("netkit: Allow for configuring needed_{head,tail}room")
lets the Cilium CNI plugin compute the headroom and set it on the
device.
On "cannot be undone": the IFLA_TX_EXTRA_HEADROOM UAPI does not block
in-kernel mechanisms. They can write the same field later, and the UAPI
stays a bounded, explicit value.
The RFC patch mentioned two other approaches: explicit headroom requirement
labeling for tc and BPF, and dynamic adjustment of this field whenever a
realloc in the stack is encountered.
They are left for future work. But should those instead be implemented
first? As said, this manual hackery cannot be undone. While in-kernel
mechanisms can be refined and built upon.
It seems to be that this series is trying to address a few related
situations and argues that no single solution fixes all except for manual
adjustment. But likely a combination of the above (and more) will.
This does leave the veth peer to solve, but that does not seem
fundamentally intractable.
Regarding labeling for tc and BPF: it is also a new UAPI, and a label
increases the value of the device that the program is attached to.
That is often not the device that the skb was allocated on, as in the
cases below. If we have to add a new UAPI anyway, I would prefer one
that covers all of them.
I looked at what dynamic adjustment would take. The kernel can find
the device to increase in these cases:
- XDP on a veth peer: veth_xdp_set() and generic_xdp_install() can
increase the headroom of the peer to XDP_PACKET_HEADROOM on attach
and reset it on detach.
- A TC egress program on the device that the skb was allocated on: a
check around the tc egress hook catches a head reallocation, and the
kernel increases the headroom of skb->dev.
- Stacked devices: a vlan leaves its tag in skb->vlan_tci, so the lower
device can find the vlan with __vlan_find_dev_deep_rcu(). macvlan,
ipvlan and team keep the dst.
It cannot in this case, which we have at Cloudflare:
- A service in a network namespace sends through a veth pair. The host
forwards the traffic to the uplink, and a TC egress program there
encapsulates it. ip_rcv_core() has already orphaned the skb, so the
kernel can only guess that the veth peer of skb_iif allocated it.
Even where the kernel finds the right device, auto-tuning has problems:
- It grows too easily. Background traffic counts the same as the main
traffic, so a few encapsulated health-check probes increase the
headroom of every skb on the device. A threshold to filter them out
would be one more policy value to choose.
- That growth can move skbs into a larger kmalloc size without anyone
deciding it. On x86_64, for 1500-byte MTU UDP over IPv4 on an
Ethernet device, 194 bytes or more moves the skb head from
kmalloc-2048 to kmalloc-4096. That adds 2048 bytes to the memory and
the truesize of each such skb. The IFLA_TX_EXTRA_HEADROOM UAPI can
do the same, but then the operator chose to pay that cost.
- It only grows, unless it is reset on detach. A reset on detach must
find every device that the auto-tuning increased, which can be a veth
peer in another namespace.
So I prefer to keep the IFLA_TX_EXTRA_HEADROOM UAPI. If we need
auto-tuning on top of it, we can discuss it when someone actually needs
it.
Thanks again for pushing on this. If you see a simpler way to do
this, I am happy to change direction after discussing it with David
Keogh, who posted the initial RFC.