Thread (8 messages) 8 messages, 3 authors, 2026-02-11

Re: [PATCH net] net/mlx5e: Skip NAPI polling when PCI channel is offline

From: Breno Leitao <leitao@debian.org>
Date: 2026-02-11 13:45:08
Also in: linux-rdma, lkml

Hello Tariq,

On Wed, Feb 11, 2026 at 01:26:35PM +0200, Tariq Toukan wrote:
On 09/02/2026 20:01, Breno Leitao wrote:
quoted
When a PCI error (e.g. AER error or DPC containment) marks the PCI
channel as frozen or permanently failed, the IOMMU mappings for the
device may already be torn down. If mlx5e_napi_poll() continues
processing CQEs in this state, every call to dma_unmap_page() triggers
a WARN_ON in iommu_dma_unmap_phys().

In a real-world crash scenario on an NVIDIA Grace (ARM64) platform,
a DPC event froze the PCI channel and the mlx5 NAPI poll continued
processing error CQEs, calling dma_unmap for each pending WQE. Here is
an example:

The DPC event on port 0007:00:00.0 fires and eth1 (on 0017:01:00.0) starts
seeing error CQEs almost immediately:

     pcieport 0007:00:00.0: DPC: containment event, status:0x2009
     mlx5_core 0017:01:00.0 eth1: Error cqe on cqn 0x54e, ci 0xb06, ...

The WARN_ON storm begins ~0.4s later and repeats for every pending WQE:

     WARNING: CPU: 32 PID: 0 at drivers/iommu/dma-iommu.c:1237 iommu_dma_unmap_phys
     Call trace:
      iommu_dma_unmap_phys+0xd4/0xe0
      mlx5e_tx_wi_dma_unmap+0xb4/0xf0
      mlx5e_poll_tx_cq+0x14c/0x438
      mlx5e_napi_poll+0x6c/0x5e0
      net_rx_action+0x160/0x5c0
      handle_softirqs+0xe8/0x320
      run_ksoftirqd+0x30/0x58

After 23 seconds of WARN_ON() storm, the watchdog fires:

     watchdog: BUG: soft lockup - CPU#32 stuck for 23s! [ksoftirqd/32:179]
     Kernel panic - not syncing: softlockup: hung tasks

Each unmap hit the WARN_ON in the IOMMU layer, printing a full stack
trace. With dozens of pending WQEs, this created a storm of WARN_ON
dumps in softirq context that monopolized the CPU for over 23 seconds,
triggering a soft lockup panic.
...
You're introducing an interesting problem, but I am not convinced by this
solution approach.

Why would the driver perform this check if it doesn't guarantee prevention
of invalid access? It only "allows one napi cycle", which happen to be good
enough to prevent the soft lockup in your case.

What if a napi cycle is configured with larger budget?
Very good point. In this case, we will still see some WARN_ON() in DMA, and the
patch might eventually not help much if the AER hits mid-NAPI and there is
still a long budget remaining.
If the problem is that the WARN_ON is being called at a high rate, then it
should be rate-limited.
That would be a solution as well, and I am happy to pursue it, if that one is
more appropriate

Thanks for reviewing it,
--breno
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help