Re: [RFC PATCH net-next 05/12] PCI: Add device-specific reset for NVIDIA Spectrum devices

From
Ido Schimmel <idosch@nvidia.com>
Date
2023-10-25 11:05:22
Also in
linux-pci
Reply to
Bjorn Helgaas, 2023-10-18
On Wed, Oct 18, 2023 at 03:08:26PM -0500, Bjorn Helgaas wrote:
On Tue, Oct 17, 2023 at 10:42:50AM +0300, Ido Schimmel wrote:
quoted
The PCIe specification defines two methods to trigger a hot reset across
a link: Bus reset and link disablement (r6.0.1, sec 7.1, sec 6.6.1). In
the first method, the Secondary Bus Reset (SBR) bit in the Bridge
Control Register of the Downstream Port is asserted for at least 1ms
(r6.0.1, sec 7.5.1.3.13). In the second method, the Link Disable bit in
the Link Control Register of the Downstream Port is asserted and then
cleared to disable and enable the link (r6.0.1, sec 7.5.3.7).

While the two methods are identical from the perspective of the
Downstream device, they are different as far as the host is concerned.
In the first method, the Link Training and Status State Machine (LTSSM)
of the Downstream Port is expected to be in the Hot Reset state as long
as the SBR bit is asserted. In the second method, the LTSSM of the
Downstream Port is expected to be in the Disabled state as long as the
Link Disable bit is asserted.

This above difference is of importance because the specification
requires the LTTSM to exit from the Hot Reset state to the Detect state
within a 2ms timeout (r6.0.1, sec 4.2.7.11).
I don't read 4.2.7.11 quite that way.  Here's the text (from r6.0):

  • Lanes that were directed by a higher Layer to initiate Hot
    Reset:

    ◦ All Lanes in the configured Link transmit TS1 Ordered Sets
      with the Hot Reset bit asserted and the configured Link and
      Lane numbers.

    ◦ If two consecutive TS1 Ordered Sets are received on any
      Lane with the Hot Reset bit asserted and configured Link
      and Lane numbers, then:

      ▪ LinkUp = 0b (False)

      ▪ If no higher Layer is directing the Physical Layer to
        remain in Hot Reset, the next state is Detect

      ▪ Otherwise, all Lanes in the configured Link continue to
	transmit TS1 Ordered Sets with the Hot Reset bit asserted
	and the configured Link and Lane numbers.

    ◦ Otherwise, after a 2 ms timeout next state is Detect.

I assume that SBR being set constitutes a "higher Layer directing the
Physical Layer to remain in Hot Reset," so I would read this as saying
the LTSSM stays in Hot Reset as long as SBR is set.  Then, *after* a
2 ms timeout (not *within* 2 ms), the next state is Detect.
quoted
NVIDIA Spectrum devices cannot guarantee it and a host enforcing
such a behavior might fail to communicate with the device after
issuing a Secondary Bus Reset.
I don't quite follow this.  What behavior is the host enforcing here?
I guess you're doing an SBR, and the Spectrum device doesn't respond
as expected afterwards?

It looks like pci_reset_secondary_bus() asserts SBR for at least
2 ms.  Then pci_bridge_wait_for_secondary_bus() should wait before
accessing the device, but maybe we don't wait long enough?

I guess this ends up back at d3cold_delay as suggested by Lukas.
I had a meeting with the PCI team before submitting this patch where I
stated that bus reset works fine (tested over 500 iterations) on the
hosts I have access to. They said that bus reset and link toggling are
identical from the perspective of the downstream device, but that in the
past they saw hosts that fail bus reset because of the time it takes the
downstream device to reach the Detect state. This was with a different
line of products that share the same PCI IP as Spectrum.

Given that I'm unable to reproduce this problem with Spectrum and that
your preference seems to be to reuse bus reset (or bus reset plus the
d3cold_delay quirk), I'll drop this patch for now. We can revisit this
patch in the future, if the problem manifests itself.

Regarding the other two PCI patches, I plan to submit this series after
net-next opens for v6.8. Are you OK with them being merged via net-next?

Thanks
Thread (24 messages) 24 messages, 5 authors, last reply 2023-10-25
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help