Hi Brenden,
I've discovered a bug with XDP_TX recycling of pages in the mlx4 driver.
If I increase the number of RX and TX queues/channels via ethtool cmd:
ethtool -L mlx4p1 rx 10 tx 10
Then when running the xdp2 program, which does XDP_TX, the kernel will
crash with page errors, because the page refcnt goes to zero or even
minus. I've noticed pages delivered to mlx4_en_rx_recycle() can have
a page refcnt of zero, which is wrong, they should always have 1 (for
XDP).
Debugging it further, I find that this can happen when mlx4_en_rx_recycle()
is called from mlx4_en_recycle_tx_desc(). This is the TX cleanup function,
associated with TX ring queues used for XDP_TX only. No others than the
XDP_TX action should be able to place packets into these TX rings
which call mlx4_en_recycle_tx_desc().
Do you have any idea of what could be going wrong in this case?
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer
On Fri, Sep 16, 2016 at 09:03:40PM +0200, Jesper Dangaard Brouer wrote:
Hi Brenden,
I've discovered a bug with XDP_TX recycling of pages in the mlx4 driver.
If I increase the number of RX and TX queues/channels via ethtool cmd:
ethtool -L mlx4p1 rx 10 tx 10
Then when running the xdp2 program, which does XDP_TX, the kernel will
crash with page errors, because the page refcnt goes to zero or even
minus. I've noticed pages delivered to mlx4_en_rx_recycle() can have
a page refcnt of zero, which is wrong, they should always have 1 (for
XDP).
Debugging it further, I find that this can happen when mlx4_en_rx_recycle()
is called from mlx4_en_recycle_tx_desc(). This is the TX cleanup function,
associated with TX ring queues used for XDP_TX only. No others than the
XDP_TX action should be able to place packets into these TX rings
which call mlx4_en_recycle_tx_desc().
Sounds pretty straightforward, let me look into it.
On Fri, Sep 16, 2016 at 09:03:40PM +0200, Jesper Dangaard Brouer wrote:
quoted
Hi Brenden,
I've discovered a bug with XDP_TX recycling of pages in the mlx4 driver.
If I increase the number of RX and TX queues/channels via ethtool cmd:
ethtool -L mlx4p1 rx 10 tx 10
Then when running the xdp2 program, which does XDP_TX, the kernel will
crash with page errors, because the page refcnt goes to zero or even
minus. I've noticed pages delivered to mlx4_en_rx_recycle() can have
a page refcnt of zero, which is wrong, they should always have 1 (for
XDP).
Debugging it further, I find that this can happen when mlx4_en_rx_recycle()
is called from mlx4_en_recycle_tx_desc(). This is the TX cleanup function,
associated with TX ring queues used for XDP_TX only. No others than the
XDP_TX action should be able to place packets into these TX rings
which call mlx4_en_recycle_tx_desc().
Sounds pretty straightforward, let me look into it.
Here is some debug info I instrumented my kernel with, and I've
attached my minicom output with a warning and a panic.
Enable some driver debug printks via::
ethtool -s mlx4p1 msglvl drv on
Debug normal situation::
$ grep recycle_ring minicom_capturefile.log08
[ 520.746610] mlx4_en: mlx4p1: Set tx_ring[56]->recycle_ring = rx_ring[0]
[ 520.747042] mlx4_en: mlx4p1: Set tx_ring[57]->recycle_ring = rx_ring[1]
[ 520.747470] mlx4_en: mlx4p1: Set tx_ring[58]->recycle_ring = rx_ring[2]
[ 520.747918] mlx4_en: mlx4p1: Set tx_ring[59]->recycle_ring = rx_ring[3]
[ 520.748330] mlx4_en: mlx4p1: Set tx_ring[60]->recycle_ring = rx_ring[4]
[ 520.748749] mlx4_en: mlx4p1: Set tx_ring[61]->recycle_ring = rx_ring[5]
[ 520.749181] mlx4_en: mlx4p1: Set tx_ring[62]->recycle_ring = rx_ring[6]
[ 520.749620] mlx4_en: mlx4p1: Set tx_ring[63]->recycle_ring = rx_ring[7]
Change $ ethtool -L mlx4p1 rx 9 tx 9 ::
[ 911.594692] mlx4_en: mlx4p1: Set tx_ring[56]->recycle_ring = rx_ring[0]
[ 911.608345] mlx4_en: mlx4p1: Set tx_ring[57]->recycle_ring = rx_ring[1]
[ 911.622008] mlx4_en: mlx4p1: Set tx_ring[58]->recycle_ring = rx_ring[2]
[ 911.636364] mlx4_en: mlx4p1: Set tx_ring[59]->recycle_ring = rx_ring[3]
[ 911.650015] mlx4_en: mlx4p1: Set tx_ring[60]->recycle_ring = rx_ring[4]
[ 911.663690] mlx4_en: mlx4p1: Set tx_ring[61]->recycle_ring = rx_ring[5]
[ 911.677356] mlx4_en: mlx4p1: Set tx_ring[62]->recycle_ring = rx_ring[6]
[ 911.690924] mlx4_en: mlx4p1: Set tx_ring[63]->recycle_ring = rx_ring[7]
[ 911.704544] mlx4_en: mlx4p1: Set tx_ring[64]->recycle_ring = rx_ring[8]
[ 911.718171] mlx4_en: mlx4p1: Set tx_ring[65]->recycle_ring = rx_ring[9]
[ 911.731772] mlx4_en: mlx4p1: Set tx_ring[66]->recycle_ring = rx_ring[10]
[ 911.745438] mlx4_en: mlx4p1: Set tx_ring[67]->recycle_ring = rx_ring[11]
[ 911.759063] mlx4_en: mlx4p1: Set tx_ring[68]->recycle_ring = rx_ring[12]
[ 911.772741] mlx4_en: mlx4p1: Set tx_ring[69]->recycle_ring = rx_ring[13]
[ 911.786415] mlx4_en: mlx4p1: Set tx_ring[70]->recycle_ring = rx_ring[14]
[ 911.800070] mlx4_en: mlx4p1: Set tx_ring[71]->recycle_ring = rx_ring[15]
Change $ ethtool -L mlx4p1 rx 10 tx 10::
netif_set_real_num_tx_queues() setting dev->real_num_tx_queues(now:80) = 64
mlx4_en: mlx4p1: frag:0 - size:1522 prefix:0 stride:4096
mlx4_en_init_recycle_ring() Set tx_ring[64]->recycle_ring = rx_ring[0]
mlx4_en_init_recycle_ring() Set tx_ring[65]->recycle_ring = rx_ring[1]
mlx4_en_init_recycle_ring() Set tx_ring[66]->recycle_ring = rx_ring[2]
mlx4_en_init_recycle_ring() Set tx_ring[67]->recycle_ring = rx_ring[3]
mlx4_en_init_recycle_ring() Set tx_ring[68]->recycle_ring = rx_ring[4]
mlx4_en_init_recycle_ring() Set tx_ring[69]->recycle_ring = rx_ring[5]
mlx4_en_init_recycle_ring() Set tx_ring[70]->recycle_ring = rx_ring[6]
mlx4_en_init_recycle_ring() Set tx_ring[71]->recycle_ring = rx_ring[7]
mlx4_en_init_recycle_ring() Set tx_ring[72]->recycle_ring = rx_ring[8]
mlx4_en_init_recycle_ring() Set tx_ring[73]->recycle_ring = rx_ring[9]
mlx4_en_init_recycle_ring() Set tx_ring[74]->recycle_ring = rx_ring[10]
mlx4_en_init_recycle_ring() Set tx_ring[75]->recycle_ring = rx_ring[11]
mlx4_en_init_recycle_ring() Set tx_ring[76]->recycle_ring = rx_ring[12]
mlx4_en_init_recycle_ring() Set tx_ring[77]->recycle_ring = rx_ring[13]
mlx4_en_init_recycle_ring() Set tx_ring[78]->recycle_ring = rx_ring[14]
mlx4_en_init_recycle_ring() Set tx_ring[79]->recycle_ring = rx_ring[15]
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer
On Fri, Sep 16, 2016 at 09:03:40PM +0200, Jesper Dangaard Brouer wrote:
quoted
Hi Brenden,
I've discovered a bug with XDP_TX recycling of pages in the mlx4 driver.
If I increase the number of RX and TX queues/channels via ethtool cmd:
ethtool -L mlx4p1 rx 10 tx 10
Then when running the xdp2 program, which does XDP_TX, the kernel will
crash with page errors, because the page refcnt goes to zero or even
minus. I've noticed pages delivered to mlx4_en_rx_recycle() can have
a page refcnt of zero, which is wrong, they should always have 1 (for
XDP).
Debugging it further, I find that this can happen when mlx4_en_rx_recycle()
is called from mlx4_en_recycle_tx_desc(). This is the TX cleanup function,
associated with TX ring queues used for XDP_TX only. No others than the
XDP_TX action should be able to place packets into these TX rings
which call mlx4_en_recycle_tx_desc().
Sounds pretty straightforward, let me look into it.
Here is some debug info I instrumented my kernel with, and I've
attached my minicom output with a warning and a panic.
Thanks for the info.
I've also spent this weekend trying to debug (pretty easy to reproduce),
but with no conclusive answer. I was investigating the sequence in
mlx4_en_stop_port to see if rx might still be running through the
function, on the theory that the xdp tx might race with
mlx4_en_free_tx_buf. For instance, I tried moving the napi_synchronize
loop to be just below the msleep(10). No improvement.
Unfortunately, I'm out of options, since my one test box has decided not
to reboot itself automatically, and I'll be out of email communication
(for 3 weeks) before anybody can physicially resuscitate it (tomorrow).
Enable some driver debug printks via::
ethtool -s mlx4p1 msglvl drv on
Debug normal situation::
On Fri, Sep 16, 2016 at 09:03:40PM +0200, Jesper Dangaard Brouer wrote:
quoted
Hi Brenden,
I've discovered a bug with XDP_TX recycling of pages in the mlx4 driver.
If I increase the number of RX and TX queues/channels via ethtool cmd:
ethtool -L mlx4p1 rx 10 tx 10
Then when running the xdp2 program, which does XDP_TX, the kernel will
crash with page errors, because the page refcnt goes to zero or even
minus. I've noticed pages delivered to mlx4_en_rx_recycle() can have
a page refcnt of zero, which is wrong, they should always have 1 (for
XDP).
Debugging it further, I find that this can happen when mlx4_en_rx_recycle()
is called from mlx4_en_recycle_tx_desc(). This is the TX cleanup function,
associated with TX ring queues used for XDP_TX only. No others than the
XDP_TX action should be able to place packets into these TX rings
which call mlx4_en_recycle_tx_desc().
Sounds pretty straightforward, let me look into it.
Here is some debug info I instrumented my kernel with, and I've
attached my minicom output with a warning and a panic.
Thanks for the info.
I've also spent this weekend trying to debug (pretty easy to reproduce),
but with no conclusive answer. I was investigating the sequence in
mlx4_en_stop_port to see if rx might still be running through the
function, on the theory that the xdp tx might race with
mlx4_en_free_tx_buf. For instance, I tried moving the napi_synchronize
loop to be just below the msleep(10). No improvement.
Unfortunately, I'm out of options, since my one test box has decided not
to reboot itself automatically, and I'll be out of email communication
(for 3 weeks) before anybody can physicially resuscitate it (tomorrow).
Sorry to leave this hanging for so long. I've just recently come back
from vacation and I think I now have a fix for this. I'll send a patch
to the list shortly.
quoted
Enable some driver debug printks via::
ethtool -s mlx4p1 msglvl drv on
Debug normal situation::