Dear Jason Gunthorpe,

On Mon, 10 Dec 2012 11:44:39 -0700, Jason Gunthorpe wrote:

Fundamentally this is similar to what Marvell did, the routing
registers are functionally identical to PCI-E bridges, but don't use
PCI-E standard configuration.

Look at how Intel models their PCI-E and you can see the same
functional elements:

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b4)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b4)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b4)

1c.x are *physical* PCI-E ports, while 00 represents the internal ring
bus. All of the above devices are on the CPU SOC.

The address decoding windows of the physical ports are configured via
standard PCI-E bridge configuration space accesses and control which
addresses from the internal ring bus go out the port, as well as which
bus number range the port handles. This the same function as the
address decoding windows, just expressed via a standard register
layout instead of as a SOC specific feature.

By providing actual PCI configuration space to control all this it is
automatically compatible with standard PCI configuration code.

The mistake these SOCs are making is not using standard PCI modeling
to describe their PCI functional blocks :|

What you'd want to see is the same arrangement as Intel:

00:00.0 Host bridge: SOC software emulated host bridge
00:01.1 PCI bridge: SOC software emulated PCI-E root port 1
00:01.2 PCI bridge: SOC software emulated PCI-E root port 2
00:02.0 Ethernet device..

With a tree like:
  00:00.0 -> 0:01.1
          -> 0:01.2 -> 00:02.0

Discovery is started from 00:00.0 and flows through all the ports in
one pass.

Hum, ok, this makes sense. I have no idea at the moment how to achieve
that, but I'll try to have a look. Do you have pointers to code doing
this?

It sounds like the concept of a PCI-E bridge with internal
configuration could be generalized for more types of hardware that the
Marvell case. Pretty much all socs will have a similar design, a
number of PCI-E ports, a collection of address decoding/routing
windows and some registers to control them, someplace...

Indeed. But for example, in Marvell's case, the address decoding
windows mechanism is not specific to PCIe, it is also used for other
devices, so the management of those decoding windows cannot be entirely
left to the PCIe driver.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com