Re: Kernel oops with mlx5 and dual XDP redirect programs

On Tue, 2018-10-23 at 12:10 +0200, Toke Høiland-Jørgensen wrote:

Saeed Mahameed [off-list ref] writes:

quoted

On Thu, 2018-10-18 at 23:53 +0200, Toke Høiland-Jørgensen wrote:

quoted

Saeed Mahameed [off-list ref] writes:

quoted

I think that the mlx5 driver doesn't know how to tell the other
device
to stop transmitting to it while it is resetting.. Maybe tariq
or
Jesper know more about this ?
I will look at this tomorrow after noon and will try to
repro...

Hi Saeed

Did you have a chance to poke at this? :)

HI Toke, yes i have been planing to respond but also i wanted to
dig
more,

so the root cause is very clear.

1. core 1 is doing tx_dev->ndo_xdp_xmit()
2. core 2 is doing tx_dev->xdp_set() //remove xdp program.

Right, it was also my guess that it was related to this interaction.
Thanks for looking into it!

quoted

and the problem is beyond mlx5, since we don't have a way to tell a
different core/different netdev to stop xmitting, or at least
synchronize with it.

Hmm, ideally there should be some way for the higher level XDP API to
notice this and abort the call before it even reaches the driver on
the
TX side, shouldn't there? At LPC, Jesper and I will be talking about
a
proposal for decoupling the ndo_xdp_xmit() resource allocation from
loading and unloading XDP programs, which I guess could be a way to
deal
with this as well.

In the meantime...

Yes totally agree, this is why my fix is temporary. 
Good Idea about LPC, let's discuss this there.

quoted

I will be waiting for your confirmation that the fix did work.

I tested your patch, and it does indeed fix the crash. However, it
also
seems to have the effect that the XDP redirect continues to function
even after removing the XDP program on the target device.

I.e., after the call to ./xdp_fwd -d $TX_IF, I still see packets
being
redirected out $TX_IF. Is this intentional?

Interesting, shouldn't happen, unless there is something weird going on
when running xpd_fwd -d together with xdp_redirect_map, i just checked
the code and if ndo_xdp_set was called with null program we will remove
xdp tx resources, nothing suspicious in the driver.

I will look at this later this week.

-Toke