Hi Lukas,

On 7/31/25 10:44 PM, Lukas Wunner wrote:

On Thu, Jul 31, 2025 at 10:04:38AM -0700, Sathyanarayanan Kuppuswamy wrote:

quoted

On 7/31/25 6:01 AM, Lukas Wunner wrote:

quoted

+++ b/drivers/pci/pcie/err.c

@@ -165,6 +165,12 @@ static int report_resume(struct pci_dev *dev, void *data)
   	return 0;
   }
+static int report_disconnect(struct pci_dev *dev, void *data)
+{
+	pci_uevent_ers(dev, PCI_ERS_RESULT_DISCONNECT);
+	return 0;
+}

Since you are notifying the user space, I am wondering whether the drivers
should be notified about the recovery failure?

The drivers are usually *causing* the recovery failure by returning
PCI_ERS_RESULT_DISCONNECT from their pci_error_handlers callbacks
(or by lacking pci_error_handlers, in particular ->error_detected()).

So in principle the drivers should be aware of recovery failure.

There are cases where multiple drivers are involved.  E.g. on GPUs,
there's often a PCIe switch with a graphics device and various sound
or telemetry devices.  Typically errors are reported by the Upstream
Port, so the Secondary Bus Reset occurs at the Root or Downstream Port
above the Upstream Port and affects the switch and all subordinate
devices.  In cases like this, recovery failure may be caused by a
single driver (e.g. GPU) and the other drivers (e.g. telemetry) may
be unaware of it.

Yes, my comment was referring to the scenario mentioned above. If one of the
subordinate devices fails recovery, then recovery effectively fails for all devices
under that downstream port (or root port). Notifying all devices under that port
would allow their drivers to perform the necessary cleanup

The recovery flow documented in Documentation/PCI/pci-error-recovery.rst
was originally conceived for EEH and indeed EEH does notify all drivers
of recovery failures by invoking the ->error_detected() callback with
channel_state pci_channel_io_perm_failure.  See this call ...

	eeh_pe_report("error_detected(permanent failure)", pe,
		      eeh_report_failure, NULL);

... in arch/powerpc/kernel/eeh_driver.c below the recover_failed label
in eeh_handle_normal_event().

Agree. The current implementation does not seem to follow the steps
mentioned in the Documentation/PCI/pci-error-recovery.rst.

STEP 6: Permanent Failure
-------------------------
A "permanent failure" has occurred, and the platform cannot recover
the device.  The platform will call error_detected() with a
pci_channel_state_t value of pci_channel_io_perm_failure.

The device driver should, at this point, assume the worst. It should
cancel all pending I/O, refuse all new I/O, returning -EIO to
higher layers. The device driver should then clean up all of its
memory and remove itself from kernel operations, much as it would
during system shutdown.

I don't know why pcie_do_recovery() doesn't do the same on recovery
failure.  This is one of several annoying deviations between AER and
EEH.  Ideally the behavior should be the same across all platforms
so that drivers don't have to cope with platform-specific quirks.

However I think that's orthogonal to the pci_uevent_ers() invocation
in pcie_do_recovery().

Agree. My thought is, since there is an attempt to fix the user notification
side of things, may be the driver side should also be fixed together .

Thanks,

Lukas

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer