Re: [PATCH V7 4/9] PCI: Enable 10-Bit Tag support for PCIe Endpoint devices

On Thu, Aug 05, 2021 at 03:47:31PM +0800, Dongdong Liu wrote:

Hi Bjorn

Many thanks for your review.
On 2021/8/5 7:17, Bjorn Helgaas wrote:

quoted

On Wed, Aug 04, 2021 at 09:47:03PM +0800, Dongdong Liu wrote:

quoted

10-Bit Tag capability, introduced in PCIe-4.0 increases the total Tag
field size from 8 bits to 10 bits.

PCIe spec 5.0 r1.0 section 2.2.6.2 "Considerations for Implementing
10-Bit Tag Capabilities" Implementation Note.
For platforms where the RC supports 10-Bit Tag Completer capability,
it is highly recommended for platform firmware or operating software
that configures PCIe hierarchies to Set the 10-Bit Tag Requester Enable
bit automatically in Endpoints with 10-Bit Tag Requester capability. This
enables the important class of 10-Bit Tag capable adapters that send
Memory Read Requests only to host memory.

Quoted material should be set off with a blank line before it and
indented by two spaces so it's clear exactly what comes from the spec
and what you've added.  For example, see
https://git.kernel.org/linus/ec411e02b7a2

Good point, will fix.

quoted

We need to say why we assume it's safe to enable 10-bit tags for all
devices below a Root Port that supports them.  I think this has to do
with switches being required to forward 10-bit tags correctly even if
they were designed before 10-bit tags were added to the spec.

PCIe spec 5.0 r1.0 section 2.2.6.2 "Considerations for Implementing
10-Bit Tag Capabilities" Implementation Note:

  Switches that lack 10-Bit Tag Completer capability are still able to
  forward NPRs and Completions carrying 10-Bit Tags correctly, since the
  two new Tag bits are in TLP Header bits that were formerly Reserved,
  and Switches are required to forward Reserved TLP Header bits without
  modification. However, if such a Switch detects an error with an NPR
  carrying a 10-Bit Tag, and that Switch handles the error by acting as
  the Completer for the NPR, the resulting Completion will have an
  invalid 10-Bit Tag. Thus, it is strongly recommended that Switches
  between any components using 10-Bit Tags support 10-Bit Tag Completer
  capability.  Note that Switches supporting 16.0 GT/s data rates or
  greater must support 10-Bit Tag Completer capability.

This patch also consider to enable 10-Bit Tag for EP device need RP
and Switch device support 10-Bit Tag Completer capability.

quoted

And it should call out any cases where it is *not* safe, e.g., if P2P
traffic is an issue.

Yes, indeed.

quoted

If there are cases where we don't want to enable 10-bit tags, whether
it's to enable P2P traffic or merely to work around device defects,
that ability needs to be here from the beginning.  If somebody needs
to bisect with 10-bit tags disabled, we don't want a bisection hole
between this commit and the commit that adds the control.

We provide sysfs file to disable 10-bit tag for P2P traffic when needed.
The details see PATCH 7/8/9.

A mechanism for avoiding problems needs to be present from the very
beginning so there's no bisection hole.  It should not be added by a
future patch.

The sysfs file is a start, but if we run into an issue, it could mean
that we can't boot and run long enough to use sysfs to disable 10-bit
tags.  So I think we might need a kernel parameter that disables it
(and possibly other things like MPS optimization).

Current we do not know the 10-bit tag defective devices, current may no
need do as 8-bit tag does in quirk_no_ext_tags().