Fix several bugs in the mlxbf_gige driver:
1) panic at shutdown
2) no ip assigned although link is up
3) clogged port due to RX pause frames
Asmaa Mnebhi (3):
mlxbf_gige: Fix kernel panic at shutdown
mlxbf_gige: Fix intermittent no ip issue
mlxbf_gige: Enable the GigE port in mlxbf_gige_open
.../mellanox/mlxbf_gige/mlxbf_gige_main.c | 47 +++++++++----------
.../mellanox/mlxbf_gige/mlxbf_gige_rx.c | 9 ++--
2 files changed, 28 insertions(+), 28 deletions(-)
--
2.30.1
Although the link is up, there is no ip assigned on a setup with high background
traffic. Nothing is transmitted nor received.
The RX error count keeps on increasing. After several minutes, the RX error count
stagnates and the GigE interface finally gets an ip.
The issue is in the mlxbf_gige_rx_init function. As soon as the RX DMA is enabled,
the RX CI reaches the max of 128, and it becomes equal to RX PI. RX CI doesn't decrease
since the code hasn't ran phy_start yet.
The solution is to move the rx init after phy_start.
Fixes: f92e1869d74e ("Add Mellanox BlueField Gigabit Ethernet driver")
Signed-off-by: Asmaa Mnebhi <asmaa@nvidia.com>
Reviewed-by: David Thompson <davthompson@nvidia.com>
---
v2->v3:
- No changes
v1->v2:
- No changes
.../ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c | 14 +++++++-------
.../ethernet/mellanox/mlxbf_gige/mlxbf_gige_rx.c | 6 +++---
2 files changed, 10 insertions(+), 10 deletions(-)
There is a race condition happening during shutdown due to pending napi transactions.
Since mlxbf_gige_poll is still running, it tries to access a NULL pointer and as a
result causes a kernel panic:
[ 284.074822] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000070
...
[ 284.322326] Call trace:
[ 284.324757] mlxbf_gige_handle_tx_complete+0xc8/0x170 [mlxbf_gige]
[ 284.330924] mlxbf_gige_poll+0x54/0x160 [mlxbf_gige]
[ 284.335876] __napi_poll+0x40/0x1c8
[ 284.339353] net_rx_action+0x314/0x3a0
[ 284.343086] __do_softirq+0x128/0x334
[ 284.346734] run_ksoftirqd+0x54/0x6c
[ 284.350294] smpboot_thread_fn+0x14c/0x190
[ 284.354375] kthread+0x10c/0x110
[ 284.357588] ret_from_fork+0x10/0x20
[ 284.361150] Code: 8b070000 f9000ea0 f95056c0 f86178a1 (b9407002)
[ 284.367227] ---[ end trace a18340bbb9ea2fa7 ]---
To fix this, invoke mlxbf_gige_remove to disable and dequeue napi during shutdown,
and also return in the case where "priv" is NULL in the poll function.
Fixes: f92e1869d74e ("Add Mellanox BlueField Gigabit Ethernet driver")
Signed-off-by: Asmaa Mnebhi <asmaa@nvidia.com>
Reviewed-by: David Thompson <davthompson@nvidia.com>
---
v2->v3:
- Add the logic to clean the port to the remove() function
v1-v2:
- make mlxbf_gige_shutdown() the same as the mlxbf_gige_remove()
.../mellanox/mlxbf_gige/mlxbf_gige_main.c | 21 ++++++++-----------
.../mellanox/mlxbf_gige/mlxbf_gige_rx.c | 3 +++
2 files changed, 12 insertions(+), 12 deletions(-)
@@ -298,6 +298,9 @@ int mlxbf_gige_poll(struct napi_struct *napi, int budget)priv=container_of(napi,structmlxbf_gige,napi);+if(!priv)+return0;+mlxbf_gige_handle_tx_complete(priv);do{
At the moment, the GigE port is enabled in the mlxbf_gige_probe
function. If the mlxbf_gige_open is not executed, this could cause
pause frames to increase in the case where there is high backgroud
traffic. This results in clogging the port.
So move enabling the OOB port to mlxbf_gige_open.
Fixes: f92e1869d74e ("Add Mellanox BlueField Gigabit Ethernet driver")
Signed-off-by: Asmaa Mnebhi <asmaa@nvidia.com>
Reviewed-by: David Thompson <davthompson@nvidia.com>
---
v2->v3:
- No changes
v1->v2:
- Fix typo: "base" to "priv->base"
.../ethernet/mellanox/mlxbf_gige/mlxbf_gige_main.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
There is a race condition happening during shutdown due to pending napi transactions.
Since mlxbf_gige_poll is still running, it tries to access a NULL pointer and as a
result causes a kernel panic:
[ 284.074822] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000070
...
[ 284.322326] Call trace:
[ 284.324757] mlxbf_gige_handle_tx_complete+0xc8/0x170 [mlxbf_gige]
[ 284.330924] mlxbf_gige_poll+0x54/0x160 [mlxbf_gige]
[ 284.335876] __napi_poll+0x40/0x1c8
[ 284.339353] net_rx_action+0x314/0x3a0
[ 284.343086] __do_softirq+0x128/0x334
[ 284.346734] run_ksoftirqd+0x54/0x6c
[ 284.350294] smpboot_thread_fn+0x14c/0x190
[ 284.354375] kthread+0x10c/0x110
[ 284.357588] ret_from_fork+0x10/0x20
[ 284.361150] Code: 8b070000 f9000ea0 f95056c0 f86178a1 (b9407002)
[ 284.367227] ---[ end trace a18340bbb9ea2fa7 ]---
To fix this, invoke mlxbf_gige_remove to disable and dequeue napi during shutdown,
and also return in the case where "priv" is NULL in the poll function.
Fixes: f92e1869d74e ("Add Mellanox BlueField Gigabit Ethernet driver")
Signed-off-by: Asmaa Mnebhi <asmaa@nvidia.com>
Reviewed-by: David Thompson <davthompson@nvidia.com>
---
v2->v3:
- Add the logic to clean the port to the remove() function
v1-v2:
- make mlxbf_gige_shutdown() the same as the mlxbf_gige_remove()
.../mellanox/mlxbf_gige/mlxbf_gige_main.c | 21 ++++++++-----------
.../mellanox/mlxbf_gige/mlxbf_gige_rx.c | 3 +++
2 files changed, 12 insertions(+), 12 deletions(-)
Although the link is up, there is no ip assigned on a setup with high background
traffic. Nothing is transmitted nor received.
The RX error count keeps on increasing. After several minutes, the RX error count
stagnates and the GigE interface finally gets an ip.
The issue is in the mlxbf_gige_rx_init function. As soon as the RX DMA is enabled,
the RX CI reaches the max of 128, and it becomes equal to RX PI. RX CI doesn't decrease
since the code hasn't ran phy_start yet.
The solution is to move the rx init after phy_start.
Fixes: f92e1869d74e ("Add Mellanox BlueField Gigabit Ethernet driver")
Signed-off-by: Asmaa Mnebhi <asmaa@nvidia.com>
Reviewed-by: David Thompson <davthompson@nvidia.com>
This seems fine, but your description of the problem still looks like there may be a more fundamental ordering issue when you enable your RX pipe here.
It seems to me like you should enable it from "inner" as in closest to the CPU/DMA subsystem towards "outer" which is the MAC and finally the PHY.
It should be fine to enable your RX DMA as long as you keep the MAC's RX disabled, and then you can enable your MAC's RX enable and later start the PHY.
--
Florian
At the moment, the GigE port is enabled in the mlxbf_gige_probe
function. If the mlxbf_gige_open is not executed, this could cause
pause frames to increase in the case where there is high backgroud
traffic. This results in clogging the port.
So move enabling the OOB port to mlxbf_gige_open.
Fixes: f92e1869d74e ("Add Mellanox BlueField Gigabit Ethernet driver")
Signed-off-by: Asmaa Mnebhi <asmaa@nvidia.com>
Reviewed-by: David Thompson <davthompson@nvidia.com>
Although the link is up, there is no ip assigned on a setup with high
background traffic. Nothing is transmitted nor received.
The RX error count keeps on increasing. After several minutes, the RX
error count stagnates and the GigE interface finally gets an ip.
The issue is in the mlxbf_gige_rx_init function. As soon as the RX DMA
is enabled, the RX CI reaches the max of 128, and it becomes equal to
RX PI. RX CI doesn't decrease since the code hasn't ran phy_start yet.
The solution is to move the rx init after phy_start.
Fixes: f92e1869d74e ("Add Mellanox BlueField Gigabit Ethernet driver")
Signed-off-by: Asmaa Mnebhi <asmaa@nvidia.com>
Reviewed-by: David Thompson <davthompson@nvidia.com>
This seems fine, but your description of the problem still looks like there may
be a more fundamental ordering issue when you enable your RX pipe here.
It seems to me like you should enable it from "inner" as in closest to the
CPU/DMA subsystem towards "outer" which is the MAC and finally the PHY.
It should be fine to enable your RX DMA as long as you keep the MAC's RX
disabled, and then you can enable your MAC's RX enable and later start the
PHY.
Thanks for your feedback Florian. I will take a look and address your comments shortly. Sorry for the delayed response, I was OOO.
Although the link is up, there is no ip assigned on a setup with
high background traffic. Nothing is transmitted nor received.
The RX error count keeps on increasing. After several minutes, the
RX error count stagnates and the GigE interface finally gets an ip.
The issue is in the mlxbf_gige_rx_init function. As soon as the RX
DMA is enabled, the RX CI reaches the max of 128, and it becomes
equal to RX PI. RX CI doesn't decrease since the code hasn't ran phy_start
yet.
quoted
quoted
The solution is to move the rx init after phy_start.
Fixes: f92e1869d74e ("Add Mellanox BlueField Gigabit Ethernet
driver")
Signed-off-by: Asmaa Mnebhi <asmaa@nvidia.com>
Reviewed-by: David Thompson <davthompson@nvidia.com>
This seems fine, but your description of the problem still looks like
there may be a more fundamental ordering issue when you enable your RX
pipe here.
quoted
It seems to me like you should enable it from "inner" as in closest to
the CPU/DMA subsystem towards "outer" which is the MAC and finally the
PHY.
quoted
It should be fine to enable your RX DMA as long as you keep the MAC's
RX disabled, and then you can enable your MAC's RX enable and later
start the PHY.
Thanks for your feedback Florian. I will take a look and address your
comments shortly. Sorry for the delayed response, I was OOO.
Hi Florian,
We would like to maintain the code as is because we need to set the RX DMA after the MAC RX filters and the RX rings are setup (in mlxbf_gige_rx_init()).
The PHY start logic needs to be done before that, otherwise, there is a chance we would encounter this bug where our MAC RX consumer index (CI) equals our MAC RX production index (PI) and that results in a MAC state that cannot be solved until we cleanup the MAC again. Note that this bug is difficult to reproduce. Our QA had to run the reboot test and have a setup with really high background traffic.
Thanks.
Asmaa