Greetings:
This is a follow up to the previous RFC (linked below).
On two production systems, I saw the following dmesg pattern:
NETDEV WATCHDOG: transmit queue 0 timed out 6073 ms
Resp cmpl intr err msg: 0x51 x20
hwrm_ring_free type 1 failed x12
hwrm_ring_free type 2 failed x8
AMD-Vi: IO_PAGE_FAULT x3
This suggests that, for some currently unknown reason, TX completions stall
and the netdev watchdog fires. The driver asks FW to free the rings, this
times out, but the driver ignores the possible failure and frees ring memory.
Since the FW didn't respond to the ring free command, it is possible that the
FW is still DMAing to the memory which was freed.
Since the RFCv4, I found a third system that hit this repeatedly:
NETDEV WATCHDOG: CPU: 35: transmit queue 0 timed out 5459 ms
hwrm_ring_free type 1 failed. rc:fffffff0 err:0 x64
hwrm_ring_free type 2 failed. rc:fffffff0 err:0 x32
hwrm_ring_free type 0 failed. rc:fffffffb err:1 x64
hwrm_ring_alloc type 1 failed. rc:ffffffe4 err:4
hwrm_ring_free type 1 failed. rc:fffffff0 err:0 x44
Abandoning msg {0x51 0x3e35} len: 0 due to firmware status: 0x408100
nic open fail (rc: fffffffb)
DMAR: [DMA Write NO_PASID] Request device [47:00.0] fault addr
0xffffa07ef000 [fault reason 0x05] PTE Write access is not set
This case differs from the first two in a few ways, but this series would have
still helped:
- The ring frees fail on the close path and again on the error path of
the reopen (bnxt_init_chip()), and the device then writes to memory the
driver has already unmapped. The series stops DMA on both paths.
- The interface came back up later, the watchdog fired again, and the
cycle started over. The host logged 79 DMAR write faults in about 20
hours. Patch 4 breaks this loop by refusing to reopen after DMA has
been stopped.
- The firmware answers each RING_FREE about 5 seconds late, after the
driver has given up, and finally reports an error in its health
register. Patch 2 does not treat these late responses as success because
their sequence ids do not match.
In this most recent case, I suspect the hardware is faulty, but the driver
shouldn't hand memory back to the system while the device could still write to
it.
In all of the above cases, the IOMMU caught the error. If the system had no
IOMMU or the IOMMU was in passthrough mode, data corruption or crash would
have occurred.
In the future, this could possibly be extended to make recovery automatic.
Thanks,
Joe
v5:
- Switched from RFC to actual submission
- Patch 4 updated to use an un-used bit to make backporting easier.
v4: https://lore.kernel.org/netdev/20260925174404.2789072-1-joe@dama.to/ (local)
- No changes to patch 1 or 2
- Patch 3: reworded the message logged when DMA is stopped, a rebind is
what recovers the device, not a firmware reset. No functional change.
- Patch 4 added which adds a new bit (BNXT_STATE_DMA_STOPPED) that is set
when DMA is disabled. When this bit is set, the device requires user
intervention to bring back up.
v3: https://lore.kernel.org/netdev/20260923210744.3406861-1-joe@dama.to/ (local)
- No changes to patch 1
- Patch 2: Don't poll for the valid bit as Michael suggested.
- Patch 3: bnxt_hwrm_ring_free now returns -EIO instead of stopping the
device, so the remaining resources can be freed and the remaining commands
can be sent before stopping the device, as Michael suggested. Note the
switch to using pci_clear_master in this patch instead of
pci_disable_device. This was done so that the normal shutdown paths can
call pci_disable_device without generating a warning.
v2: https://lore.kernel.org/netdev/20260922182405.1290749-1-joe@dama.to/ (local)
- No changes to patch 1
- Patch 2 from v1 dropped
- Patch 2 in the v2 now checks the response and logs state before giving up
- Patch 3 in the v2 disables the device to stop DMA before freeing ring
memory
RFCv1: https://lore.kernel.org/netdev/20260917233218.1160001-1-joe@dama.to/ (local)
Joe Damato (4):
bnxt_en: return the RING_FREE status to callers
bnxt_en: check HWRM response if completion never arrives
bnxt_en: stop DMA before releasing rings the firmware did not free
bnxt_en: refuse to open a device with stopped DMA
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 122 ++++++++++++------
drivers/net/ethernet/broadcom/bnxt/bnxt.h | 1 +
.../net/ethernet/broadcom/bnxt/bnxt_hwrm.c | 33 ++++-
3 files changed, 115 insertions(+), 41 deletions(-)
base-commit: 19a227903e65c7a9c179156485660a4e017575fe
--
2.53.0-Meta