Hi Bjorn

-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas at kernel.org]
Sent: 24 May 2016 00:39
To: Gabriele Paoloni
Cc: Lorenzo Pieralisi; Ard Biesheuvel; Jon Masters; Tomasz Nowicki;
arnd at arndb.de; will.deacon at arm.com; catalin.marinas at arm.com;
rafael at kernel.org; hanjun.guo at linaro.org; okaya at codeaurora.org;
jchandra at broadcom.com; linaro-acpi at lists.linaro.org; linux-
pci at vger.kernel.org; dhdang at apm.com; Liviu.Dudau at arm.com;
ddaney at caviumnetworks.com; jeremy.linton at arm.com; linux-
kernel at vger.kernel.org; linux-acpi at vger.kernel.org;
robert.richter at caviumnetworks.com; Suravee.Suthikulpanit at amd.com;
msalter at redhat.com; Wangyijing; mw at semihalf.com;
andrea.gallo at linaro.org; linux-arm-kernel at lists.infradead.org
Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
controller

On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:

quoted

Hi Lorenzo

quoted

-----Original Message-----
From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi at arm.com]
Sent: 23 May 2016 11:57
To: Ard Biesheuvel
Cc: Gabriele Paoloni; Jon Masters; Tomasz Nowicki;

helgaas at kernel.org;

quoted

quoted

arnd at arndb.de; will.deacon at arm.com; catalin.marinas at arm.com;
rafael at kernel.org; hanjun.guo at linaro.org; okaya at codeaurora.org;
jchandra at broadcom.com; linaro-acpi at lists.linaro.org; linux-
pci at vger.kernel.org; dhdang at apm.com; Liviu.Dudau at arm.com;
ddaney at caviumnetworks.com; jeremy.linton at arm.com; linux-
kernel at vger.kernel.org; linux-acpi at vger.kernel.org;
robert.richter at caviumnetworks.com; Suravee.Suthikulpanit at amd.com;
msalter at redhat.com; Wangyijing; mw at semihalf.com;
andrea.gallo at linaro.org; linux-arm-kernel at lists.infradead.org
Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI

host

quoted

quoted

controller

On Fri, May 20, 2016 at 11:14:03AM +0200, Ard Biesheuvel wrote:

quoted

On 20 May 2016 at 10:40, Gabriele Paoloni

[off-list ref] wrote:

quoted

quoted

Hi Ard

quoted

-----Original Message-----
From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]

[...]

quoted

quoted

Is the PCIe root complex so special that you cannot simply

describe an

quoted

quoted

quoted

implementation that is not PNP0408 compatible as something

else,

quoted

quoted

under

quoted

quoted

quoted

its own unique HID? If everybody is onboard with using ACPI,

how

quoted

quoted

is

quoted

quoted

quoted

this any different from describing other parts of the platform
topology? Even if the SBSA mandates generic PCI, they already

deviated

quoted

quoted

quoted

from that when they built the hardware, so pretending that it

is a

quoted

quoted

quoted

quoted

quoted

PNP0408 with quirks really does not buy us anything.

From my understanding we want to avoid this as this would allow

each

quoted

quoted

vendor to come up with his own code and it would be much more

effort

quoted

quoted

for the PCI maintainer to rework the PCI framework to

accommodate

quoted

quoted

X86

quoted

quoted

and "all" ARM64 Host Controllers...

I guess this approach is too risky and we want to avoid this.

Through

quoted

quoted

standardization we can more easily maintain the code and scale

it

quoted

quoted

to

quoted

quoted

multiple SoCs...

So this is my understanding; maybe Jon, Tomasz or Lorenzo can

give

quoted

quoted

quoted

quoted

a bit more explanation...

OK, so that boils down to recommending to vendors to represent

known

quoted

quoted

quoted

non-compliant hardware as compliant, just so that we don't have

to

quoted

quoted

quoted

change the code to support additional flavors of ECAM ? It's fine

to

quoted

quoted

quoted

be pragmatic, but that sucks.

We keep confusing the x86 case with the ARM case here: for x86,

they

quoted

quoted

quoted

needed to deal with broken hardware *after* the fact, and all

they

quoted

quoted

quoted

could do is find /some/ distinguishing feature in order to guess

which

quoted

exact hardware they might be running on. For arm64, it is the

opposite

quoted

case. We are currently in a position where we can demand vendors

to

quoted

quoted

quoted

comply with the standards they endorsed themselves, and (ab)using

ACPI

quoted

+ DMI as a de facto platform description rather than plain ACPI

makes

quoted

quoted

quoted

me think the DT crowd were actually right from the beginning. It
*directly* violates the standardization principle, since it

requires

quoted

quoted

a

quoted

priori knowledge inside the OS that a certain 'generic' device

must

quoted

quoted

be

quoted

driven in a special way.

So can anyone comment on the feasibility of adding support for

devices

quoted

with vendor specific HIDs (and no generic CIDs) to the current

ACPI

quoted

quoted

quoted

ECAM driver in Linux?

I don't think of ECAM support itself as a "driver".  It's just a
service available to drivers, similar to OF resource parsing.

Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
supports extended config space.  It doesn't specify how we access that
config space, so I think hardware with non-standard ECAM should still
have PNP0A03 and PNP0A08 in _CID or _HID.

"ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
r3.2, sec 4.1) means:

  (a) a memory-mapped model for config space access, and
  (b) a specific mapping of address bits to bus/device/function/
      register

MCFG and _CBA assume both (a) and (b), so I think a device with
non-standard ECAM mappings should not be described in MCFG or _CBA.

If a bridge has ECAM with non-standard mappings, I think either a
vendor-specific _HID or a device-specific method, e.g., _DSM, could
communicate that.

Jon, I agree that we should avoid describing non-standardized hardware
in Linux-specific ways.  Is there a mechanism in use already?  How
does Windows handle this?  DMI is a poor long-term solution because it
requires ongoing maintenance for new platforms, but I think it's OK
for getting started with platforms already shipping.

A _DSM has the advantage that once it is defined and supported, OEMs
can ship new platforms without requiring a new quirk or a new _HID to
be added to a driver.

There would still be the problem of config access before the namespace
is available, i.e., the MCFG use case.  I don't know how important
that is.  Defining an MCFG extension seems like the most obvious
solution.

If we only expect a few non-standard devices, maybe it's enough to
have DMI quirks to statically set up ECAM and just live with the
inconvenience of requiring a kernel change for every new non-standard
device.

quoted

quoted

Host bridges in ACPI are handled through PNP0A08/PNP0A03 ids, and
most of the arch specific code is handled in the respective arch
directories (X86 and IA64, even though IA64 does not rely on

ECAM/MCFG

quoted

quoted

for
PCI ops), it is not a driver per-se, PNP0A08/PNP0A03 are detected
through
ACPI scan handlers and the respective arch code (ie

pci_acpi_scan_root)

quoted

quoted

sets-up resources AND config space on an arch specific basis.

X86 deals with that with code in arch/x86 that sets-up the

pci_raw_ops

quoted

quoted

on a platform specific basis (and it is not nice, but it works

because

quoted

quoted

as you all know the number of platforms in X86 world is contained).

Will this happen for ARM64 in arch/arm64 based on vendor specific
HIDs ?

No.

So given the current state of play (we were requested to move the
arch/arm64 specific ACPI PCI bits to arch/arm64), we would end up
with arch/arm64 code requiring code in /drivers to set-up pci_ops
in a platform specific way, it is horrible, if feasible at all.

The only way this can be implemented is by pretending that the
ACPI/PCI arch/arm64 implementation is generic code (that's what

this

quoted

quoted

series does), move it to /drivers (where it is in this series), and
implement _DSD vendor specific bindings (per HID) to set-up the pci
operations; whether this solution should go upstream, given that it
is just a short-term solution for early platforms bugs, it is

another

quoted

quoted

story and my personal answer is no.

It seems like there should be a way to look for a _DSM before we call
acpi_pci_root_get_mcfg_addr() to look for _CBA.

Currently we call acpi_pci_root_get_mcfg_addr() (to read _CBA) from
the generic acpi_pci_root_add(), but the result (root->mcfg_addr) is
only used in x86-specific code.  I think it would be nicer if the
lookup and the use were together.  Then it would be easier to override
it because the mapping assumptions would all be in one place.

quoted

I think it shouldn't be too bad to move quirk handling mechanism to
arch/arm64. Effectively we would not move platform specific code into
arch/arm64 but just the mechanism checking if there is any quirk that
is defined.

i.e.:

extern struct pci_cfg_fixup __start_acpi_mcfg_fixups[];
extern struct pci_cfg_fixup __end_acpi_mcfg_fixups[];

static struct pci_ecam_ops *pci_acpi_get_ops(struct acpi_pci_root

*root)

quoted

{
        int bus_num = root->secondary.start;
        int domain = root->segment;
        struct pci_cfg_fixup *f;

        /*
         * Match against platform specific quirks and return

corresponding

quoted

         * CAM ops.
         *
         * First match against PCI topology <domain:bus> then use DMI

or

quoted

         * custom match handler.
         */
        for (f = __start_acpi_mcfg_fixups; f <

__end_acpi_mcfg_fixups; f++) {

quoted

                if ((f->domain == domain || f->domain ==

PCI_MCFG_DOMAIN_ANY) &&

quoted

                    (f->bus_num == bus_num || f->bus_num ==

PCI_MCFG_BUS_ANY) &&

quoted

                    (f->system ? dmi_check_system(f->system) : 1) &&
                    (f->match ? f->match(f, root) : 1))
                        return f->ops;
        }
        /* No quirks, use ECAM */
        return &pci_generic_ecam_ops;
}

Such quirks will be defined anyway in drivers/pci/host/ in the vendor
specific quirk implementations.

e.g. in HiSilicon case we would have

DECLARE_ACPI_MCFG_FIXUP(NULL, hisi_pcie_match, &hisi_pcie_ecam_ops,
			PCI_MCFG_DOMAIN_ANY, PCI_MCFG_BUS_ANY);

in "drivers/pci/host/pcie-hisi-acpi.c "

Thanks

Gab

Sorry Gab, I guess I was really responding to earlier messages :)

I don't really have much to say here, except that it doesn't seem
right to have an MCFG that describes a non-standard ECAM mapping.

The ACPI table that this mechanism relies upon is the one discussed
in:
https://lkml.org/lkml/2016/3/9/91

As you can see MCFG describes ECAM mappings, but we have a motherboard
reserved resource outside the MCFG:
Device (RES0)
{
	Name (_HID, "HISI0081") // HiSi PCIe RC config base address
	Name (_CID, "PNP0C02") // Motherboard reserved resource
	Name (_CRS, ResourceTemplate (){
	Memory32Fixed (ReadWrite, 0xb0080000 , 0x10000)
	})
}

This allows us to retrieve the address we need for accessing
the config space on the RC (that is non-ECAM).

I was thinking that such mechanism could fit different vendors
and allow them define their own quirks without spoiling the 
official and standard MCFG; also from the thread discussion
you seemed quite ok with such solution...?

I suppose there's already firmware in the field that does that,
though?

We have "experimental" firmware that is based on the ACPI table
described above, however it is not widely distributed and, obviously,
it is not supported by Linux mainline (so we have room to rework
if we decide that another solution is more appropriate)

Thanks and Regards

Gab

Bjorn