Re: [PATCH net-next 00/10] net/mlx5e: Add netdev support for data direct

On Sat, Oct 10, 2026 at 7:16 AM Dragos Tatulea [off-list ref] wrote:

On 10.10.26 01:54, Mina Almasry wrote:

quoted

On Fri, Oct 9, 2026 at 9:42 AM Mina Almasry [off-list ref] wrote:

quoted

On Thu, Oct 8, 2026 at 6:28 AM Tariq Toukan [off-list ref] wrote:

quoted

Hi,

This patch series by Dragos adds data direct support for netdev devices,
allowing a mlx5 netdev to issue DMA commands through multiple paths.
This feature is critical for improving performance and reaching line
rate in certain environments where issuing PCI transactions over one
path may be significantly faster than over another. These differences
can arise from various PCI generations in the system or the specific
system topology.

Data direct for netdev will work with devmem by returning the data
direct DMA device instead of the DMA device of the NIC. Each PF netdev
registers its own data direct device and support for RX and TX datapath
is added. The feature is selected through a new data direct priv flag.

The feature is enabled though a ethtool private flag. When this flag
is enabled, the driver will return the data direct device in the
ndo_queue_get_dma_dev op.

During data_direct device unbind, a re-creation of the channels is
triggered.

A note about the unbind: it does not does not tear down dmabuf bindings
attached to the data direct device; the recreated channels will still
pick up the binding via rxq->mp_params and post DMA addresses from the
data direct IOMMU domain to the PF, causing IOMMU faults. A devmem
revoke hook is needed to close those bindings before the switch. This is
out of scope for this series.

Elaborate on this please. Are you referring to the fact that on
page_pool_scrub we unmap the netmems dma-mapped by the page-pool, but
we don't unmap the netmems dma-mapped by the memory provider?

What's the impact of leaving this unfixed? Is the machine going to
crash if the device goes away but we don't unmap the dmabuf?

IOMMU faults because the dma device is invalid. Either because the
DD dev is still in use and was unbound. Or because the non-DD
one is used and is invalid (due to channel reopen).

quoted

Oh, you're referring to an even different bug than the one I had in
mind. (I'll see if I can submit a fix for the bug I had in mind).

But in general I think not addressing the issue you're referring to is
way too messy. I think we do indeed need to unbind the dma-buf if the
dma-dev is going away. The invariant should be that the
binding->attachment->dev should not change at all during the entire
lifetime of the binding. We already 'revoke' the dma-buf binding if
the netdev is going away, sorta (there is a bug there I need to fix.
The uninstall function in the provider doesn't actually unmap the
dma-buf :sad face:).

I've worked with my LLM to suggest some changes that (we think) make
this work, but you may disagree. I'll post the suggestions. But please
I think this should be handled one way or another.

My idea was to handle the unbind part in a subsequent series because
it probably needs some more back and forth to get this cleanup path
right.

If not acceptable I'll work on adding it to this series.

Since these are mostly driver-local changes I don't want to be so
forceful to block it by saying it's unacceptable, but if at all
possible lets try not to exacerbate the current mess. It looks like we
already a bunch of lifetime bugs with regards to the device going
away. I've posted this series, which - I think - gives you the devmem
'revoke' mechanism that you want:

https://lore.kernel.org/netdev/20261010025532.839559-1-almasrymina@google.com/ (local)

Also working on that, I realized we don't use binding->vdev and
binding->dev correctly on binding rx. binding->vdev is meant to be the
device we do the bind to, and binding->dev is meant to be the
underlying dma_device (if they're different). That fits into your
model where you have the netdev you want to bind to, but have an
underlying device that you want to dma-map to.

I've gotten some feedback on my series and I'm working on v2. If it
makes sense for you to wait on my devmem core changes so you can
integrate into them better and avoid these lifetime bugs, lets do
that.

-- 
Thanks,
Mina