On Mon, Aug 13, 2012 at 11:47 AM, Stephen Warren [off-list ref] wrote:
On 08/13/2012 11:40 AM, Thierry Reding wrote:
quoted
On Mon, Aug 06, 2012 at 01:42:21PM -0600, Stephen Warren wrote:
quoted
On 07/26/2012 01:55 PM, Thierry Reding wrote:
quoted
This patch series adds support for device tree based probing of
the PCIe controller found on Tegra SoCs.
Thierry, I just tested all Tegra boards in v3.6-rc1, and noticed
that PCIe doesn't work on TrimSlice when booting use device tree.
I think I found the cause, and I can't see why the same problem
doesn't affect this series. Perhaps you can enlighten me?
...
quoted
quoted
PCI: Device 0000:01:00.0 not available because of resource
collisions
...
quoted
I've looked into this a bit, and it seems like ARM is using an
open- coded version of the pci_enable_resources() function here,
with the only difference being the unconditional enabling of both
I/O and memory- mapped access for bridges. On Tegra there is
already a PCI fixup to do this, so pci_enable_resources() can be
used as-is.
I'd prefer that bridge I/O & memory access enabling be done in a
mainline path, not in a fixup. Fixups are intended for working around
defects in specific devices, not for the normal path. I know various
architectures have fixups that are used in the normal path, but I've
been working on eliminating them.
The patch did alter the behavior a little for TrimSlice, but didn't
solve the problem. The old error messages:
quoted
[ 2.173971] PCI: Device 0000:01:00.0 not available because of resource collisions
[ 2.181453] r8169 0000:01:00.0: (unregistered net_device): enable failure
[ 2.188254] r8169: probe of 0000:01:00.0 failed with error -22
Were replaced with the following with your patch:
quoted
[ 2.174010] r8169 0000:01:00.0: device not available (can't reserve [io 0x0000-0x00ff])
[ 2.182098] r8169 0000:01:00.0: (unregistered net_device): enable failure
[ 2.188900] r8169: probe of 0000:01:00.0 failed with error -22
This message appears from drivers/pci/setup-res.c pci_enable_resources()
due to:
quoted
if (!r->parent) {
dev_err(&dev->dev, "device not available "
"(can't reserve %pR)\n", r);
return -EINVAL;
}
That check doesn't appear in ARM's custom pcibios_enable_device().
Disabling that check yields:
quoted
[ 2.174192] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[ 2.180041] r8169 0000:01:00.0: BAR 2: can't reserve [mem 0x00000000-0x00000fff 64bit pref]
[ 2.188386] r8169 0000:01:00.0: (unregistered net_device): could not request regions
[ 2.196140] r8169: probe of 0000:01:00.0 failed with error -16
I think that's because the pci_dev's resources are initially assigned
PCI-aperture-relative addresses, and then these are later patched up to
take account of where the aperture is mapped into the CPU's address space.
We definitely shouldn't be calling the driver probe routine before the
device BARs are assigned.
Boot log using board files:
quoted
[ 1.146145] pci 0000:01:00.0: reg 10: [io 0x0000-0x00ff]
[ 1.151745] pci 0000:01:00.0: reg 18: [mem 0x00000000-0x00000fff 64bit pref]
[ 1.159007] pci 0000:01:00.0: reg 20: [mem 0x00000000-0x00003fff 64bit pref]
[ 1.166270] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
...
quoted
[ 1.217829] pci 0000:01:00.0: BAR 6: assigned [mem 0xa0000000-0xa001ffff pref]
[ 1.225264] pci 0000:01:00.0: BAR 4: assigned [mem 0xa0020000-0xa0023fff 64bit pref]
[ 1.233236] pci 0000:01:00.0: BAR 2: assigned [mem 0xa0024000-0xa0024fff 64bit pref]
[ 1.241206] pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff]
... (I added some extra printks:)
quoted
[ 1.488007] r8169 0000:01:00.0: BAR 0: requesting [io 0x1000-0x10ff]
[ 1.501483] r8169 0000:01:00.0: BAR 2: requesting [mem 0xa0024000-0xa0024fff 64bit pref]
[ 1.516611] r8169 0000:01:00.0: BAR 4: requesting [mem 0xa0020000-0xa0023fff 64bit pref]
whereas for a device tree boot:
(same):
quoted
[ 2.112217] pci 0000:01:00.0: reg 10: [io 0x0000-0x00ff]
[ 2.117635] pci 0000:01:00.0: reg 18: [mem 0x00000000-0x00000fff 64bit pref]
[ 2.124690] pci 0000:01:00.0: reg 20: [mem 0x00000000-0x00003fff 64bit pref]
[ 2.131731] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
... (request region happens early)
quoted
[ 2.179838] r8169 0000:01:00.0: BAR 0: requesting [io 0x0000-0x00ff]
[ 2.193312] r8169 0000:01:00.0: BAR 2: requesting [mem 0x00000000-0x00000fff 64bit pref]
[ 2.201397] r8169 0000:01:00.0: BAR 2: can't reserve [mem 0x00000000-0x00000fff 64bit pref]
[ 2.209742] r8169 0000:01:00.0: (unregistered net_device): could not request regions
... (same, just happens too late)
quoted
[ 2.236818] pci 0000:01:00.0: BAR 6: assigned [mem 0xa0000000-0xa001ffff pref]
[ 2.244027] pci 0000:01:00.0: BAR 4: assigned [mem 0xa0020000-0xa0023fff 64bit pref]
[ 2.251794] pci 0000:01:00.0: BAR 2: assigned [mem 0xa0024000-0xa0024fff 64bit pref]
[ 2.259542] pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff]
I suspect this is all still related to the PCI devices themselves being
probed much earlier in the overall PCI initialization sequence when the
PCI controller is probed later in the boot sequence, whereas PCI device
probe is deferred until the overall PCI initialization sequence is
complete if the PCI controller is probed very early in the boot sequence.
I don't know what to apply your patches to (they don't apply cleanly
to v3.6-rc2), so I can't see exactly what you're doing. But it looks
like you might be calling pci_bus_add_devices() before
pci_bus_assign_resource(), which isn't going to work.
I don't know what it means to probe PCI devices before probing the PCI
controller (the host bridge) -- that shouldn't happen either. In
order to probe PCI devices, we have to first know about the host
bridge so we know how to do config accesses and what the bridge
apertures are.
On Mon, Aug 13, 2012 at 04:18:16PM -0700, Bjorn Helgaas wrote:
On Mon, Aug 13, 2012 at 11:47 AM, Stephen Warren [off-list ref] wrote:
quoted
On 08/13/2012 11:40 AM, Thierry Reding wrote:
quoted
On Mon, Aug 06, 2012 at 01:42:21PM -0600, Stephen Warren wrote:
quoted
On 07/26/2012 01:55 PM, Thierry Reding wrote:
quoted
This patch series adds support for device tree based probing of
the PCIe controller found on Tegra SoCs.
Thierry, I just tested all Tegra boards in v3.6-rc1, and noticed
that PCIe doesn't work on TrimSlice when booting use device tree.
I think I found the cause, and I can't see why the same problem
doesn't affect this series. Perhaps you can enlighten me?
...
quoted
quoted
PCI: Device 0000:01:00.0 not available because of resource
collisions
...
quoted
I've looked into this a bit, and it seems like ARM is using an
open- coded version of the pci_enable_resources() function here,
with the only difference being the unconditional enabling of both
I/O and memory- mapped access for bridges. On Tegra there is
already a PCI fixup to do this, so pci_enable_resources() can be
used as-is.
I'd prefer that bridge I/O & memory access enabling be done in a
mainline path, not in a fixup. Fixups are intended for working around
defects in specific devices, not for the normal path. I know various
architectures have fixups that are used in the normal path, but I've
been working on eliminating them.
I understand. Perhaps it should be added to the pci_enable_resources()
function?
quoted
The patch did alter the behavior a little for TrimSlice, but didn't
solve the problem. The old error messages:
quoted
[ 2.173971] PCI: Device 0000:01:00.0 not available because of resource collisions
[ 2.181453] r8169 0000:01:00.0: (unregistered net_device): enable failure
[ 2.188254] r8169: probe of 0000:01:00.0 failed with error -22
Were replaced with the following with your patch:
quoted
[ 2.174010] r8169 0000:01:00.0: device not available (can't reserve [io 0x0000-0x00ff])
[ 2.182098] r8169 0000:01:00.0: (unregistered net_device): enable failure
[ 2.188900] r8169: probe of 0000:01:00.0 failed with error -22
This message appears from drivers/pci/setup-res.c pci_enable_resources()
due to:
quoted
if (!r->parent) {
dev_err(&dev->dev, "device not available "
"(can't reserve %pR)\n", r);
return -EINVAL;
}
That check doesn't appear in ARM's custom pcibios_enable_device().
Disabling that check yields:
quoted
[ 2.174192] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[ 2.180041] r8169 0000:01:00.0: BAR 2: can't reserve [mem 0x00000000-0x00000fff 64bit pref]
[ 2.188386] r8169 0000:01:00.0: (unregistered net_device): could not request regions
[ 2.196140] r8169: probe of 0000:01:00.0 failed with error -16
I think that's because the pci_dev's resources are initially assigned
PCI-aperture-relative addresses, and then these are later patched up to
take account of where the aperture is mapped into the CPU's address space.
We definitely shouldn't be calling the driver probe routine before the
device BARs are assigned.
quoted
Boot log using board files:
quoted
[ 1.146145] pci 0000:01:00.0: reg 10: [io 0x0000-0x00ff]
[ 1.151745] pci 0000:01:00.0: reg 18: [mem 0x00000000-0x00000fff 64bit pref]
[ 1.159007] pci 0000:01:00.0: reg 20: [mem 0x00000000-0x00003fff 64bit pref]
[ 1.166270] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
...
quoted
[ 1.217829] pci 0000:01:00.0: BAR 6: assigned [mem 0xa0000000-0xa001ffff pref]
[ 1.225264] pci 0000:01:00.0: BAR 4: assigned [mem 0xa0020000-0xa0023fff 64bit pref]
[ 1.233236] pci 0000:01:00.0: BAR 2: assigned [mem 0xa0024000-0xa0024fff 64bit pref]
[ 1.241206] pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff]
... (I added some extra printks:)
quoted
[ 1.488007] r8169 0000:01:00.0: BAR 0: requesting [io 0x1000-0x10ff]
[ 1.501483] r8169 0000:01:00.0: BAR 2: requesting [mem 0xa0024000-0xa0024fff 64bit pref]
[ 1.516611] r8169 0000:01:00.0: BAR 4: requesting [mem 0xa0020000-0xa0023fff 64bit pref]
whereas for a device tree boot:
(same):
quoted
[ 2.112217] pci 0000:01:00.0: reg 10: [io 0x0000-0x00ff]
[ 2.117635] pci 0000:01:00.0: reg 18: [mem 0x00000000-0x00000fff 64bit pref]
[ 2.124690] pci 0000:01:00.0: reg 20: [mem 0x00000000-0x00003fff 64bit pref]
[ 2.131731] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
... (request region happens early)
quoted
[ 2.179838] r8169 0000:01:00.0: BAR 0: requesting [io 0x0000-0x00ff]
[ 2.193312] r8169 0000:01:00.0: BAR 2: requesting [mem 0x00000000-0x00000fff 64bit pref]
[ 2.201397] r8169 0000:01:00.0: BAR 2: can't reserve [mem 0x00000000-0x00000fff 64bit pref]
[ 2.209742] r8169 0000:01:00.0: (unregistered net_device): could not request regions
... (same, just happens too late)
quoted
[ 2.236818] pci 0000:01:00.0: BAR 6: assigned [mem 0xa0000000-0xa001ffff pref]
[ 2.244027] pci 0000:01:00.0: BAR 4: assigned [mem 0xa0020000-0xa0023fff 64bit pref]
[ 2.251794] pci 0000:01:00.0: BAR 2: assigned [mem 0xa0024000-0xa0024fff 64bit pref]
[ 2.259542] pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff]
I suspect this is all still related to the PCI devices themselves being
probed much earlier in the overall PCI initialization sequence when the
PCI controller is probed later in the boot sequence, whereas PCI device
probe is deferred until the overall PCI initialization sequence is
complete if the PCI controller is probed very early in the boot sequence.
I don't know what to apply your patches to (they don't apply cleanly
to v3.6-rc2), so I can't see exactly what you're doing. But it looks
like you might be calling pci_bus_add_devices() before
pci_bus_assign_resource(), which isn't going to work.
The patch was on top of this series, which in turn is based on -next.
However the patch doesn't actually change anything in the ordering of
the initialization sequence, it just replaces the ARM implementation
of pcibios_enable_device() to reuse pci_enable_resources().
ARM's main PCI entry point, arch/arm/kernel/bios32.c:pci_common_init(),
does call the correct sequence: pci_bus_assign_resources() followed by
pci_bus_add_devices(), so the sequence looks correct. However, judging
by the logs Stephen posted, the BAR assignment is too late, after the
driver has been probed.
I don't know what it means to probe PCI devices before probing the PCI
controller (the host bridge) -- that shouldn't happen either. In
order to probe PCI devices, we have to first know about the host
bridge so we know how to do config accesses and what the bridge
apertures are.
Right. I don't think this can happen. If the bridge hasn't been probed,
then the devices shouldn't be there anyway.
Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20120814/8c368b33/attachment.sig>
On 08/13/2012 05:18 PM, Bjorn Helgaas wrote:
On Mon, Aug 13, 2012 at 11:47 AM, Stephen Warren [off-list ref] wrote:
...
quoted
whereas for a device tree boot:
(same):
quoted
[ 2.112217] pci 0000:01:00.0: reg 10: [io 0x0000-0x00ff]
[ 2.117635] pci 0000:01:00.0: reg 18: [mem 0x00000000-0x00000fff 64bit pref]
[ 2.124690] pci 0000:01:00.0: reg 20: [mem 0x00000000-0x00003fff 64bit pref]
[ 2.131731] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
... (request region happens early)
quoted
[ 2.179838] r8169 0000:01:00.0: BAR 0: requesting [io 0x0000-0x00ff]
[ 2.193312] r8169 0000:01:00.0: BAR 2: requesting [mem 0x00000000-0x00000fff 64bit pref]
[ 2.201397] r8169 0000:01:00.0: BAR 2: can't reserve [mem 0x00000000-0x00000fff 64bit pref]
[ 2.209742] r8169 0000:01:00.0: (unregistered net_device): could not request regions
... (same, just happens too late)
quoted
[ 2.236818] pci 0000:01:00.0: BAR 6: assigned [mem 0xa0000000-0xa001ffff pref]
[ 2.244027] pci 0000:01:00.0: BAR 4: assigned [mem 0xa0020000-0xa0023fff 64bit pref]
[ 2.251794] pci 0000:01:00.0: BAR 2: assigned [mem 0xa0024000-0xa0024fff 64bit pref]
[ 2.259542] pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff]
I suspect this is all still related to the PCI devices themselves being
probed much earlier in the overall PCI initialization sequence when the
PCI controller is probed later in the boot sequence, whereas PCI device
probe is deferred until the overall PCI initialization sequence is
complete if the PCI controller is probed very early in the boot sequence.
I don't know what to apply your patches to (they don't apply cleanly
to v3.6-rc2), so I can't see exactly what you're doing. But it looks
like you might be calling pci_bus_add_devices() before
pci_bus_assign_resource(), which isn't going to work.
Yes, that's exactly what is happening.
PCIe initialization starts in arch/arm/mach-tegra/pci.e
tegra_pcie_init() which calls arch/arm/kernel/bios32.c
pci_common_init(). That function first calls pcibios_init_hw() (in the
same file, more about this later) and then loops over PCI buses, calling
amongst other things pci_bus_assign_resources() then pci_bus_add_devices().
The problem is that ARM's pcibios_init_hw() calls pci_scan_root_bus()
(or a host-driver-specific function which that also calls
pci_scan_root_bus() in Tegra's case) which in turn calls
pci_bus_add_devices() right at the end, before control has returned to
pci_common_init() and hence before pci_bus_assign_resources() has been
called.
If I modify pci_scan_root_bus() and remove the call to
pci_bus_add_devices(), everything works as expected.
So, I guess the question is: Should ARM's pcibios_init_hw() not be
calling pci_scan_root_bus(), or at least presumably the ARM PCI code
needs to do things in a slightly different order?