Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
QL41000 ethernet controller:
BUG: scheduling while atomic: kworker/0:4/531/0x00000200
[qed_probe:488()]hw prepare failed
kernel BUG at mm/vmalloc.c:2355!
Internal error: Oops - BUG: 0 [#1] SMP
CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
pstate: 00400009 (nzcv daif +PAN -UAO)
Call trace:
vunmap+0x4c/0x50
iounmap+0x48/0x58
qed_free_pci+0x60/0x80 [qed]
qed_probe+0x35c/0x688 [qed]
__qede_probe+0x88/0x5c8 [qede]
qede_probe+0x60/0xe0 [qede]
local_pci_probe+0x48/0xa0
work_for_cpu_fn+0x24/0x38
process_one_work+0x1d0/0x468
worker_thread+0x238/0x4e0
kthread+0xf0/0x118
ret_from_fork+0x10/0x18
In this case, qed_hw_prepare() returns error due to hw/fw error, but in
theory work queue should be in process context instead of interrupt.
The root cause might be the unpaired spin_{un}lock_bh() in
_qed_mcp_cmd_and_union(), which causes botton half is disabled incorrectly.
Reported-by: Lijian Zhang <redacted>
Signed-off-by: Jia He <redacted>
---
drivers/net/ethernet/qlogic/qed/qed_mcp.c | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
From: Justin He <hidden> Date: 2021-07-15 23:42:50
-----Original Message-----
From: patchwork-bot+netdevbpf@kernel.org <patchwork-
bot+netdevbpf@kernel.org>
Sent: Friday, July 16, 2021 3:50 AM
To: Justin He <redacted>
Cc: aelior@marvell.com; GR-everest-linux-l2@marvell.com;
davem@davemloft.net; kuba@kernel.org; netdev@vger.kernel.org; linux-
kernel@vger.kernel.org; nd [off-list ref]
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
_qed_mcp_cmd_and_union()
Hello:
This patch was applied to netdev/net.git (refs/heads/master):
On Thu, 15 Jul 2021 16:08:21 +0800 you wrote:
quoted
Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
QL41000 ethernet controller:
BUG: scheduling while atomic: kworker/0:4/531/0x00000200
[qed_probe:488()]hw prepare failed
kernel BUG at mm/vmalloc.c:2355!
Internal error: Oops - BUG: 0 [#1] SMP
CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
Hi Jia,
On Thu, Jul 15, 2021 at 2:28 PM Jia He [off-list ref] wrote:
Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
QL41000 ethernet controller:
BUG: scheduling while atomic: kworker/0:4/531/0x00000200
[qed_probe:488()]hw prepare failed
kernel BUG at mm/vmalloc.c:2355!
Internal error: Oops - BUG: 0 [#1] SMP
CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
pstate: 00400009 (nzcv daif +PAN -UAO)
Call trace:
vunmap+0x4c/0x50
iounmap+0x48/0x58
qed_free_pci+0x60/0x80 [qed]
qed_probe+0x35c/0x688 [qed]
__qede_probe+0x88/0x5c8 [qede]
qede_probe+0x60/0xe0 [qede]
local_pci_probe+0x48/0xa0
work_for_cpu_fn+0x24/0x38
process_one_work+0x1d0/0x468
worker_thread+0x238/0x4e0
kthread+0xf0/0x118
ret_from_fork+0x10/0x18
In this case, qed_hw_prepare() returns error due to hw/fw error, but in
theory work queue should be in process context instead of interrupt.
The root cause might be the unpaired spin_{un}lock_bh() in
_qed_mcp_cmd_and_union(), which causes botton half is disabled incorrectly.
Reported-by: Lijian Zhang <redacted>
Signed-off-by: Jia He <redacted>
---
This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired
spin_{un}lock_bh.
Also,
as per description, looks like you are not sure actual the root-cause.
does this patch really solved the problem?
--pk
From: Justin He <hidden> Date: 2021-07-19 13:17:06
Hi Prabhakar
-----Original Message-----
From: Prabhakar Kushwaha <redacted>
Sent: Monday, July 19, 2021 6:36 PM
To: Justin He <redacted>
Cc: Ariel Elior <redacted>; GR-everest-linux-l2@marvell.com;
David S. Miller [off-list ref]; Jakub Kicinski [off-list ref];
netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
kernel@vger.kernel.org>; nd [off-list ref]; Shai Malin [off-list ref];
Shai Malin [off-list ref]; Prabhakar Kushwaha [off-list ref]
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
_qed_mcp_cmd_and_union()
Hi Jia,
On Thu, Jul 15, 2021 at 2:28 PM Jia He [off-list ref] wrote:
quoted
Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
QL41000 ethernet controller:
BUG: scheduling while atomic: kworker/0:4/531/0x00000200
[qed_probe:488()]hw prepare failed
kernel BUG at mm/vmalloc.c:2355!
Internal error: Oops - BUG: 0 [#1] SMP
CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
Ubuntu
quoted
pstate: 00400009 (nzcv daif +PAN -UAO)
Call trace:
vunmap+0x4c/0x50
iounmap+0x48/0x58
qed_free_pci+0x60/0x80 [qed]
qed_probe+0x35c/0x688 [qed]
__qede_probe+0x88/0x5c8 [qede]
qede_probe+0x60/0xe0 [qede]
local_pci_probe+0x48/0xa0
work_for_cpu_fn+0x24/0x38
process_one_work+0x1d0/0x468
worker_thread+0x238/0x4e0
kthread+0xf0/0x118
ret_from_fork+0x10/0x18
In this case, qed_hw_prepare() returns error due to hw/fw error, but in
theory work queue should be in process context instead of interrupt.
The root cause might be the unpaired spin_{un}lock_bh() in
_qed_mcp_cmd_and_union(), which causes botton half is disabled
incorrectly.
quoted
Reported-by: Lijian Zhang <redacted>
Signed-off-by: Jia He <redacted>
---
This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired
spin_{un}lock_bh.
For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop
if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}
Also,
as per description, looks like you are not sure actual the root-cause.
does this patch really solved the problem?
I don't have that ThunderX2 to verify the patch.
But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore
under driver/.../qlogic, this is the only problematic point I could figure
out. And this might be possible code path of qed_probe().
--
Cheers,
Justin (Jia He)
Hi Justin,
On Mon, Jul 19, 2021 at 6:47 PM Justin He [off-list ref] wrote:
Hi Prabhakar
quoted
-----Original Message-----
From: Prabhakar Kushwaha <redacted>
Sent: Monday, July 19, 2021 6:36 PM
To: Justin He <redacted>
Cc: Ariel Elior <redacted>; GR-everest-linux-l2@marvell.com;
David S. Miller [off-list ref]; Jakub Kicinski [off-list ref];
netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
kernel@vger.kernel.org>; nd [off-list ref]; Shai Malin [off-list ref];
Shai Malin [off-list ref]; Prabhakar Kushwaha [off-list ref]
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
_qed_mcp_cmd_and_union()
Hi Jia,
On Thu, Jul 15, 2021 at 2:28 PM Jia He [off-list ref] wrote:
quoted
Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ
QL41000 ethernet controller:
BUG: scheduling while atomic: kworker/0:4/531/0x00000200
[qed_probe:488()]hw prepare failed
kernel BUG at mm/vmalloc.c:2355!
Internal error: Oops - BUG: 0 [#1] SMP
CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-
Ubuntu
quoted
pstate: 00400009 (nzcv daif +PAN -UAO)
Call trace:
vunmap+0x4c/0x50
iounmap+0x48/0x58
qed_free_pci+0x60/0x80 [qed]
qed_probe+0x35c/0x688 [qed]
__qede_probe+0x88/0x5c8 [qede]
qede_probe+0x60/0xe0 [qede]
local_pci_probe+0x48/0xa0
work_for_cpu_fn+0x24/0x38
process_one_work+0x1d0/0x468
worker_thread+0x238/0x4e0
kthread+0xf0/0x118
ret_from_fork+0x10/0x18
In this case, qed_hw_prepare() returns error due to hw/fw error, but in
theory work queue should be in process context instead of interrupt.
The root cause might be the unpaired spin_{un}lock_bh() in
_qed_mcp_cmd_and_union(), which causes botton half is disabled
incorrectly.
quoted
Reported-by: Lijian Zhang <redacted>
Signed-off-by: Jia He <redacted>
---
This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired
spin_{un}lock_bh.
For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop
I agree till here.
if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}
Because of break, cnt has not been increased.
- cnt is still less than max_retries.
- if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.
I am not seeing any issue here.
quoted
Also,
as per description, looks like you are not sure actual the root-cause.
does this patch really solved the problem?
I don't have that ThunderX2 to verify the patch.
But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore
under driver/.../qlogic, this is the only problematic point I could figure
out. And this might be possible code path of qed_probe().
Without testing and proper root-cause, it is tough to accept the suggested fix.
--pk
From: Justin He <hidden> Date: 2021-07-20 02:10:44
-----Original Message-----
From: Prabhakar Kushwaha <redacted>
Sent: Monday, July 19, 2021 10:51 PM
To: Justin He <redacted>
Cc: Ariel Elior <redacted>; GR-everest-linux-l2@marvell.com;
David S. Miller [off-list ref]; Jakub Kicinski [off-list ref];
netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
kernel@vger.kernel.org>; nd [off-list ref]; Shai Malin [off-list ref];
Shai Malin [off-list ref]; Prabhakar Kushwaha [off-list ref]
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
_qed_mcp_cmd_and_union()
Hi Justin,
On Mon, Jul 19, 2021 at 6:47 PM Justin He [off-list ref] wrote:
quoted
Hi Prabhakar
quoted
-----Original Message-----
From: Prabhakar Kushwaha <redacted>
Sent: Monday, July 19, 2021 6:36 PM
To: Justin He <redacted>
Cc: Ariel Elior <redacted>; GR-everest-linux-l2@marvell.com;
David S. Miller [off-list ref]; Jakub Kicinski [off-list ref];
netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
kernel@vger.kernel.org>; nd [off-list ref]; Shai Malin
[off-list ref];
quoted
quoted
Shai Malin [off-list ref]; Prabhakar Kushwaha
[off-list ref]
quoted
quoted
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in
_qed_mcp_cmd_and_union()
Hi Jia,
On Thu, Jul 15, 2021 at 2:28 PM Jia He [off-list ref] wrote:
quoted
Liajian reported a bug_on hit on a ThunderX2 arm64 server with
FastLinQ
quoted
quoted
quoted
QL41000 ethernet controller:
BUG: scheduling while atomic: kworker/0:4/531/0x00000200
[qed_probe:488()]hw prepare failed
kernel BUG at mm/vmalloc.c:2355!
Internal error: Oops - BUG: 0 [#1] SMP
CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic
#86-
quoted
quoted
Ubuntu
quoted
pstate: 00400009 (nzcv daif +PAN -UAO)
Call trace:
vunmap+0x4c/0x50
iounmap+0x48/0x58
qed_free_pci+0x60/0x80 [qed]
qed_probe+0x35c/0x688 [qed]
__qede_probe+0x88/0x5c8 [qede]
qede_probe+0x60/0xe0 [qede]
local_pci_probe+0x48/0xa0
work_for_cpu_fn+0x24/0x38
process_one_work+0x1d0/0x468
worker_thread+0x238/0x4e0
kthread+0xf0/0x118
ret_from_fork+0x10/0x18
In this case, qed_hw_prepare() returns error due to hw/fw error, but
in
quoted
quoted
quoted
theory work queue should be in process context instead of interrupt.
The root cause might be the unpaired spin_{un}lock_bh() in
_qed_mcp_cmd_and_union(), which causes botton half is disabled
incorrectly.
quoted
Reported-by: Lijian Zhang <redacted>
Signed-off-by: Jia He <redacted>
---
This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired
spin_{un}lock_bh.
For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop
I agree till here.
quoted
if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}
Because of break, cnt has not been increased.
- cnt is still less than max_retries.
- if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.
Sorry, indeed. Let me check other possibilities.
@David S. Miller Sorry for the inconvenience, could you please revert it
in netdev tree?
Apologies again.
--
Cheers,
Justin (Jia He)
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-07-20 09:10:12
On Tue, 20 Jul 2021 02:02:26 +0000, Justin He wrote:
quoted
quoted
For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop
I agree till here.
quoted
if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}
Because of break, cnt has not been increased.
- cnt is still less than max_retries.
- if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.
Sorry, indeed. Let me check other possibilities.
@David S. Miller Sorry for the inconvenience, could you please revert it
in netdev tree?
Please submit a revert patch with the conclusions from the discussion
included in the commit message.