On Mon, Sep 22, 2014 at 3:09 PM, Bjorn Helgaas [off-list ref] wrote:

On Mon, Sep 22, 2014 at 3:33 PM, Tanmay Inamdar [off-list ref] wrote:

quoted

quoted

quoted

+static void xgene_pcie_fixup_bridge(struct pci_dev *dev)
+{
+     int i;
+
+     /* Hide the PCI host BARs from the kernel as their content doesn't
+      * fit well in the resource management
+      */

This needs a better explanation than "doesn't fit well."

I *think* you're probably talking about something similar to the MVEBU
devices mentioned here:
http://lkml.kernel.org/r/CAErSpo56jB1Bf2JtYCGKXZBZqRF1jXFxGmeewPX_e6vSXueGyA at mail.gmail.com

where the device can be configured as either an endpoint or a root port,
and the endpoint BARs are still visible when configured as a root port.

It is true that X-Gene PCIe port can be configured as EP, however the
the FIXUP is required not because of the BARs are still visible when
configured as EP in past. X-Gene PCIe port, when configured as RC,
uses BAR0-BAR1 of RC's configuration space as inbound BARs. Entire DDR
region is mapped in these BARs so that it is accessible for EP devices
for DMA. So if the fixup is not done during enumeration, whole
outbound memory resource space gets assigned to RC and nothing is left
EP devices.

I'm not familiar with the concept of an "inbound BAR"; at least I'm
not aware of anything like this in the PCI specs.  I think it might
reduce confusion if we avoided calling them "BARs".  They happen to be
at the same addresses where real BARs would be, but they certainly
don't behave like real BARs.

It sounds like this is basically a trivial IOMMU that determines which
DMA accesses can reach main memory.

Yes. That is correct.

quoted

quoted

In any event, I'd like a description of exactly what these BARs are and wha
the problem is.

Is it ok to put above description in comment to explain the fixup?

quoted

Presumably the BARs exist and were sized by the PCI core
in __pci_read_base().  That will generate some log messages and possibly
some warnings, depending on how the host bridge windows are set up.

Instead of doing this in a fixup, can you change the PCI config
accessors so that when accessing these inbound DMA mapping registers,
they drop writes and return zeros for reads?  Then the PCI core won't
think there are BARs there, and it won't mistakenly corrupt them when
it tries to size them.

I suppose this can be done. I will fix this in next revision.

Bjorn