Hi Stephen,
Thanks for the invaluable and detailed review.
On Sun, 8 Nov 2026 19:08:31 -0800, Stephen Hemminger wrote:
Since the motivating case is XDP, the mechanism should let the XDP program
ask for the padding it needs, rather than making the operator translate
the program's requirement into a device setting by hand.
The series started from David Keogh's TC egress case, as described in
the RFC [1] and in the first paragraph of the v1 cover letter. The veth
XDP case came later, when I found that the same field also avoids the
copy for generic XDP on veth.
Your own reply already works this out:
quoted
- XDP on a veth peer: veth_xdp_set() and generic_xdp_install() can
increase the headroom of the peer to XDP_PACKET_HEADROOM on attach
and reset it on detach.
That is the mechanism, and it needs no uapi, no operator arithmetic, and
no stale state. I would rather see it implemented for the cases it covers
than add a permanent uapi that removes the incentive to do so.
Agreed. For this case, I plan to move to this in v2. When an XDP program
is attached to a veth device, tx_extra_headroom of the peer is set to
XDP_PACKET_HEADROOM.
On the argument that program-side declaration is itself new uapi, and that
the labeled device is often not the device the skb was allocated on:
granted for the general TC egress case.
For TC egress, the kernel does not know at attach time how many bytes
the program pushes. Only the program, or the software that loads it,
knows. Attach-time propagation works for XDP because XDP_PACKET_HEADROOM
is a fixed value.
Since we both agree that auto-tuning has real problems, a narrow UAPI,
such as a BPF-side declaration, seems unavoidable for TC egress. The
program declares the headroom that it needs through this UAPI. The
initial plan is a BTF decl_tag, as Dragos Tatulea suggested on the RFC
[2]. On attach, the kernel increases tx_extra_headroom of the device
that the program is attached to.
Which leaves the genuinely hard case, the one you describe at Cloudflare: a
service in a netns sends through a veth pair, the host forwards to the
uplink, and a TC egress program there encapsulates, after ip_rcv_core()
has orphaned the skb so the allocating device cannot be identified. If
that is the real residue, make it the problem being solved and argue for
the narrowest mechanism that solves it.
This case only came up after I started to look at in-kernel approaches,
because the general IFLA_TX_EXTRA_HEADROOM UAPI already solves it. It is
not a new case, and it is not uncommon. Cilium, Flannel and Calico use a
tunnel by default [3][4][5], so the device that encapsulates is not the
device that allocated the skb.
netkit already has a similar UAPI. Commit b9ed315d3c4c ("netkit: Allow
for configuring needed_{head,tail}room") [6] by Daniel Borkmann added
IFLA_NETKIT_HEADROOM for Cilium in VXLAN or Geneve tunneling mode. As
you point out, the value can go stale. But the CNI plugin sets it when
it creates the device, as Cilium does for netkit.
As far as I can see, a narrow UAPI, such as a veth attribute
VETH_INFO_TX_EXTRA_HEADROOM, is unavoidable for this case.
As posted, the series leads with
cases that in-kernel propagation handles and uses them to justify an
interface sized for the case it does not.
Based on what I have learned from both of you, this is my plan for v2:
- Drop IFLA_TX_EXTRA_HEADROOM.
- Keep 1/8 and 2/8, i.e. tx_extra_headroom, as an in-kernel field. It is
needed to avoid a perf regression if we just use needed_headroom. It
is needed in all three cases below.
- veth XDP: set on attach as described above.
- TC egress on the device that allocates the skb: the BTF decl_tag UAPI
described above.
- Cross-device encapsulation: the veth attribute
VETH_INFO_TX_EXTRA_HEADROOM UAPI described above. I will rework the
existing iproute2 series [7] to set this attribute, and post it
together with v2.
This would introduce two narrow UAPIs instead of one generic UAPI.
Please let me know if everyone is happy with this new plan. If there are
no objections, I will proceed with v2 following this plan. I am happy to
change it if there is a better solution for these cases.
To David Keogh: this plan drops the generic device attribute UAPI that
your RFC proposed. Your TC egress use case stays in v2, with a BPF-side
declaration instead of the device attribute. Before I proceed, please
let me know if you are unhappy with this plan.
Thanks,
Tianyi
[1] https://lore.kernel.org/all/20261015235335.4162993-1-dkeogh@amazon.com/ (local)
[2] https://lore.kernel.org/all/bba453eb-769e-458b-88f7-8cc9bcafc1db@nvidia.com/ (local)
[3] Cilium, veth with VXLAN by default:
https://github.com/cilium/cilium/blob/main/pkg/defaults/defaults.go
[4] Flannel, VXLAN by default:
https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml
[5] Calico, IPIP by default:
https://github.com/projectcalico/calico/blob/master/manifests/calico.yaml
[6] https://git.kernel.org/torvalds/c/b9ed315d3c4c
[7] https://lore.kernel.org/all/cover.1794022936.git.tianyi@cloudflare.com/ (local)