Thread (5 messages) 5 messages, 2 authors, 2024-01-12

PowerNV PCIe hotplug support?

From: Timothy Pearson <tpearson@raptorengineering.com>
Date: 2023-12-28 04:16:17

I've been evaluating some new options for our POWER9-based hardware in the NVMe space, and would like some clarification on the current status of PCIe hotplug for the PowerNV platforms.

From what I understand, the pnv_php driver provides the basic hotplug functionality on PowerNV.  What I'm not clear on is to what extent this is intended to flow downstream to attached PCIe switches.

I have a test setup here that consists of a PMC 8533 switch and several downstream NVMe drives, with the switch attached directly to the PHB4 root port.  After loading the pnv_php module, I can disconnect the downstream NVMe devices by either using echo 0 on /sys/bus/pcu/slots/Snnnnnnn/power, or by doing a physical surprise unplug, however nothing I try can induce a newly plugged device to train and be detected on the bus.  Even trying a echo 0 and subsequent echo 1 to /sys/bus/pcu/slots/Snnnnnnn/power only results in the device going offline, there seems to be no way to bring the device back online short of a reboot.

Hotplug of other devices connected directly to the PHB4 appears to work properly (I can online and offline via the power node); the issue seems to be restricted to downstream devices connected to the (theoretically hotplug capable) PMC 8533 switch.

Is this the intended behavior for downstream (non-IBM) PCIe ports?  Raptor can provide resources to assist in a fix if needed, but I would like to understand if this is a bug or an unimplemented feature first, and if the latter what the main issues are likely to be in implementation.

Thank you!
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help