[BUG] atlantic: soft lockup / system hang on PCIe surprise-removal

From: Claas Flint <hidden>
Date: 2026-07-21 11:10:28
Also in: lkml

Hello,

I would like to report a soft lockup and subsequent full system hang caused by the
atlantic driver when the NIC is surprise-removed from the PCIe bus.

Hardware:
- ASUS ProArt X870E-Creator WIFI, AMD (AM5), BIOS 1805 (2025-12-11)
- Onboard Aquantia/Marvell AQC113 [1d6a:04c0] (driver: atlantic), enp11s0
- Kernel 6.17.0-22-generic (also 6.17.0-35), Ubuntu, ZFS root

Summary:
When the AQC113 is surprise-removed from the PCIe bus (device stops responding;
MMIO register reads return 0xffffffff), the atlantic service task does not detect
the dead device and keeps polling its statistics registers. It spins indefinitely,
producing kernel soft lockups on multiple CPUs and ultimately an unrecoverable
full-system freeze that requires a hard reset. A single NIC detach thus takes the
entire machine down.

In our case the trigger is upstream of the NIC: the shared AMD 600-series chipset
PCIe switch downstream port (0000:07:00.0 [1022:43f5]) fails to resume from D3hot
("pcieport 0000:07:00.0: Unable to change power state from D3hot to D0, device
inaccessible"), which detaches both onboard NICs behind that switch. But regardless
of *why* the device disappears, the atlantic driver's reaction (spinning forever on
0xffffffff reads) is what turns a recoverable NIC loss into a dead box.

Sequence (previous boot, timestamps):
  06:36:48  pcieport 0000:07:00.0: Unable to change power state from D3hot to D0, device inaccessible
  06:37:14  watchdog: BUG: soft lockup - CPU#13 stuck for 27s! [kworker/u128:0:3575224]
  ...repeated soft lockups...
  06:43:47  watchdog: BUG: soft lockup - CPU#2 stuck for 392s! [kworker/2:3:3707537]
  (log then goes completely silent: full freeze until manual reset 4 days later)

Representative stack trace:
  RIP: aq_hw_read_reg+0x39/0x60 [atlantic]           (RAX=0x00000000ffffffff, RDI=0xffffffff)
  Call Trace:
   hw_atl_rpb_rx_dma_drop_pkt_cnt_get+0x13/0x20 [atlantic]
   aq_a2_fw_update_stats+0x13e/0x1e0 [atlantic]
   aq_nic_service_task+0x78/0x120 [atlantic]
   process_one_work+0x18e/0x3e0
   worker_thread+0x2e3/0x420
   kthread+0x10a/0x230
  Workqueue: atlantic aq_nic_service_task [atlantic]

Note RAX/RDI = 0xffffffff, the all-ones read of a device that has fallen
off the bus. aq_hw_read_reg() returns 0xffffffff and the stats/service path neither
checks for this nor for pci_channel_offline()/pci_device_is_present(), so it never
stops rescheduling.

Suggested hardening (for discussion):
- In the service task / stats update path, detect a surprise-removed device
  (e.g. pci_channel_offline(pdev) or an all-ones sentinel from aq_hw_read_reg())
  and stop rescheduling aq_nic_service_task instead of spinning, so a NIC detach
  degrades gracefully (interface goes down) rather than hanging the whole system.

Reproducibility: intermittent (~once per 1-2 weeks on this box), tied to the PCIe
D3-resume failure of the shared chipset switch. Full dmesg / lspci topology
available on request; happy to test patches.

Thanks,
Claas Flint

Attachments

Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help