On 16/07/2025 23.20, Jakub Kicinski wrote:

On Wed, 16 Jul 2025 13:17:53 +0200 Lorenzo Bianconi wrote:

quoted

quoted

quoted

I can't see what the non-redirected use-case could be. Can you please provide
more details?
Moreover, can it be solved without storing the rx_hash (or the other
hw-metadata) in a non-driver specific format?

Having setters feels more generic than narrowly solving only the redirect,
but I don't have a good use-case in mind.
   

quoted

Storing the hw-metadata in some of hw-specific format in xdp_frame will not
allow to consume them directly building the skb and we will require to decode
them again. What is the upside/use-case of this approach? (not considering the
orthogonality with the get method).

If we add the store kfuncs to regular drivers, the metadata  won't be stored
in the xdp_frame; it will go into the rx descriptors so regular path that
builds skbs will use it.

IIUC, the described use-case would be to modify the hw metadata via a
'setter' kfunc executed by an eBPF program bounded to the NIC and to store
the new metadata in the DMA descriptor in order to be consumed by the driver
codebase building the skb, right?
If so:
- we can get the same result just storing (running a kfunc) the modified hw
   metadata in the xdp_buff struct using a well-known/generic layout and
   consume it in the driver codebase (e.g. if the bounded eBPF program
   returns XDP_PASS) using a generic xdp utility routine. This part is not in
   the current series.
- Using this approach we are still not preserving the hw metadata if we pass
   the xdp_frame to a remote CPU returning XDP_REDIRCT (we need to add more
   code)
- I am not completely sure if can always modify the DMA descriptor directly
   since it is DMA mapped.

Let me explain why it is a bad idea of writing into the RX descriptors.
The DMA descriptors are allocated as coherent DMA (dma_alloc_coherent).
This is memory that is shared with the NIC hardware device, which
implies cache-line coherence.  NIC performance is tightly coupled to
limiting cache misses for descriptors.  One common trick is to pack more
descriptors into a single cache-line.  Thus, if we start to write into
the current RX-descriptor, then we invalidate that cache-line seen from
the device, and next RX-descriptor (from this cache-line) will be in an
unfortunate coherent state.  Behind the scene this might lead to some
extra PCIe transactions.

By writing to the xdp_frame, we don't have to modify the DMA descriptors
directly and risk invalidating cache lines for the NIC.

quoted

What do you think?

FWIW I commented on an earlier revision to similar effect as Stanislav.
To me the main concern is that we're adding another adhoc scheme, and
are making xdp_frame grow into a para-skb. We added XDP to make raw
packet access fast, now we're making drivers convert metadata twice :/

Thanks for the feedback. I can see why you'd be concerned about adding
another adhoc scheme or making xdp_frame grow into a "para-skb".

However, I'd like to frame this as part of a long-term plan we've been
calling the "mini-SKB" concept. This isn't a new idea, but a
continuation of architectural discussions from as far back as [2016].

The long-term goal, described in these presentations from [2018] and
[2019], has always been to evolve the xdp_frame to handle more hardware
offloads, with the ultimate vision of moving SKB allocation out of NIC
drivers entirely. In the future, the netstack could perform L3
forwarding (and L2 bridging) directly on these enhanced xdp_frames
[2019-slide20]. The main blocker for this vision has been the lack of
hardware metadata in the xdp_frame.

This patchset is a small but necessary first step towards that goal. It
focuses on the concrete XDP_REDIRECT use-case where we can immediately
benefit for our production use-case. Storing this metadata in the
xdp_frame is fundamental to the plan. It's no coincidence the fields are
compatible with the SKB; they need to be.

I'm certainly open to debating the bigger picture, but I hope we can
agree that it shouldn't hold up this first step, which solves an
immediate need. Perhaps we can evaluate the merits of this specific
change first, and discuss the overall architecture in parallel?

--Jesper


Links:
------
[2019] XDP closer integration with network stack
  - 
https://people.netfilter.org/hawk/presentations/KernelRecipes2019/xdp-netstack-concert.pdf
  - 
https://github.com/xdp-project/xdp-project/blob/main/conference/KernelRecipes2019/xdp-netstack-concert.org#slide-move-skb-allocations-out-of-nic-drivers
  - [2019-slide20] 
https://github.com/xdp-project/xdp-project/blob/main/conference/KernelRecipes2019/xdp-netstack-concert.org#slide-fun-with-xdp_frame-before-skb-alloc

[2018] LPC Networking Track: XDP - challenges and future work
  - https://people.netfilter.org/hawk/presentations/LinuxPlumbers2018/
  - 
https://github.com/xdp-project/xdp-project/blob/main/conference/LinuxPlumbers2018/presentation-lpc2018-xdp-future.org#topic-moving-skb-allocation-out-of-driver

[2016] Network Performance Workshop
  - 
https://people.netfilter.org/hawk/presentations/NetDev1.2_2016/net_performance_workshop_netdev1.2.pdf