On 08/14/2012 05:51 PM, Stephen Warren wrote:

On 08/14/2012 04:58 PM, Stephen Warren wrote:

quoted

On 08/14/2012 03:55 PM, Bjorn Helgaas wrote:

quoted

On Tue, Aug 14, 2012 at 12:58 PM, Thierry Reding
[off-list ref] wrote:

quoted

On Tue, Aug 14, 2012 at 01:39:23PM -0600, Stephen Warren wrote:

quoted

On 08/13/2012 05:18 PM, Bjorn Helgaas wrote:

quoted

On Mon, Aug 13, 2012 at 11:47 AM, Stephen Warren [off-list ref] wrote:

...

quoted

quoted

whereas for a device tree boot:

(same):

quoted

[    2.112217] pci 0000:01:00.0: reg 10: [io  0x0000-0x00ff]
[    2.117635] pci 0000:01:00.0: reg 18: [mem 0x00000000-0x00000fff 64bit pref]
[    2.124690] pci 0000:01:00.0: reg 20: [mem 0x00000000-0x00003fff 64bit pref]
[    2.131731] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]

... (request region happens early)

quoted

[    2.179838] r8169 0000:01:00.0: BAR 0: requesting [io  0x0000-0x00ff]
[    2.193312] r8169 0000:01:00.0: BAR 2: requesting [mem 0x00000000-0x00000fff 64bit pref]
[    2.201397] r8169 0000:01:00.0: BAR 2: can't reserve [mem 0x00000000-0x00000fff 64bit pref]
[    2.209742] r8169 0000:01:00.0: (unregistered net_device): could not request regions

... (same, just happens too late)

quoted

[    2.236818] pci 0000:01:00.0: BAR 6: assigned [mem 0xa0000000-0xa001ffff pref]
[    2.244027] pci 0000:01:00.0: BAR 4: assigned [mem 0xa0020000-0xa0023fff 64bit pref]
[    2.251794] pci 0000:01:00.0: BAR 2: assigned [mem 0xa0024000-0xa0024fff 64bit pref]
[    2.259542] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]

I suspect this is all still related to the PCI devices themselves being
probed much earlier in the overall PCI initialization sequence when the
PCI controller is probed later in the boot sequence, whereas PCI device
probe is deferred until the overall PCI initialization sequence is
complete if the PCI controller is probed very early in the boot sequence.

I don't know what to apply your patches to (they don't apply cleanly
to v3.6-rc2), so I can't see exactly what you're doing.  But it looks
like you might be calling pci_bus_add_devices() before
pci_bus_assign_resource(), which isn't going to work.

Yes, that's exactly what is happening.

PCIe initialization starts in arch/arm/mach-tegra/pci.e
tegra_pcie_init() which calls arch/arm/kernel/bios32.c
pci_common_init(). That function first calls pcibios_init_hw() (in the
same file, more about this later) and then loops over PCI buses, calling
amongst other things pci_bus_assign_resources() then pci_bus_add_devices().

The problem is that ARM's pcibios_init_hw() calls pci_scan_root_bus()
(or a host-driver-specific function which that also calls
pci_scan_root_bus() in Tegra's case) which in turn calls
pci_bus_add_devices() right at the end, before control has returned to
pci_common_init() and hence before pci_bus_assign_resources() has been
called.

If I modify pci_scan_root_bus() and remove the call to
pci_bus_add_devices(), everything works as expected.

So, I guess the question is: Should ARM's pcibios_init_hw() not be
calling pci_scan_root_bus(), or at least presumably the ARM PCI code
needs to do things in a slightly different order?

I think you need to do something like this instead of using pci_scan_root_bus():

    pci_create_root_bus()
    pci_scan_child_bus()
    pci_bus_assign_resources()
    pci_bus_add_devices()

This is the effective order used by most of the pci_create_root_bus() callers.

That would pretty much duplicate everything in pci_scan_root_bus(). That
might cause divergence down the road.

Can't we make the call to pci_bus_add_devices() optional in
pci_scan_root_bus() somehow; one of:

Sigh, that turns out not to work correctly; it solves at least this part
of the problem when booting using device tree, but when booting using a
board file, it causes the IRQ number passed to the PCIe device to be
bogus:-(

The reason this fails is because the following happens after
pci_scan_bus_root():

	pci_fixup_irqs(pcibios_swizzle, pcibios_map_irq);

(in arch/arm/kernel/bios32.c:pci_common_init())

So, if I replace the call to pci_scan_root_bus() with the code you wrote
above, then the IRQ numbers aren't assigned into the pci_dev structures
until after they're device_add()ed, and hence probed().

Ideally, I could just add a call to pci_fixup_irqs() right before the
call to pci_bus_add_devices() in the code you wrote above. However, that
won't work, because pci_fixup_irqs() finds all the PCI devices to fix up
by searching the list of devices that pci_bus_add_devices() adds to:-(

I guess it's a pretty basic premise of the current PCI code that all the
PCI scanning happens well before any device drivers are registered,
which in turn means that device_add() doesn't trigger the device's
probe() until much later, after all the fixups and resource assignments
are done?