AMCC PPC460EX Canyonlands does not see PCIe end point with only non-prefetchable memory (both 2.6.27.7 and -next)

Hello,

AMCC PPC460EX Canyonlands development board with PCI Express x1 card
in PCIe x4 slot labeled 'PCIE-1'. The PCI Express card has a
soft-programmable PCIe end point.

Problem: The end point with only non-prefetchable BAR does not appear
behind the PPC SoC internal bridge. It does show up on two other
setups: MPC8315E-RDB PPC as well as a PC x86.
It has BARs 0-5 each as a 32-bit non-prefetchable memory region. This
is the kernel boot:

pci 0001:80:00.0: PCI bridge, secondary bus 0001:81
pci 0001:80:00.0:   IO window: disabled
pci 0001:80:00.0:   MEM window: disabled
pci 0001:80:00.0:   PREFETCH window: disabled
...
bus: 81 index 0 mmio: [0xfffe0000-0xfffe0fff]
bus: 81 index 1 mmio: [0xeb8000000-0xeb80fffff]
bus: 81 index 2 mmio: [0x0-0x0]
bus: 81 index 3 mmio: [0x0-0x0]

lspci -v does not show the device.


The default end point design does have a prefetchable region and does
show up properly:

pci 0001:80:00.0: PCI bridge, secondary bus 0001:81
pci 0001:80:00.0:   IO window: disabled
pci 0001:80:00.0:   MEM window: 0x81000000-0x810fffff
pci 0001:80:00.0:   PREFETCH window: 0x00000080000000-0x00000080ffffff
...
bus: 81 index 0 mmio: [0xfffe0000-0xfffe0fff]
bus: 81 index 1 mmio: [0xe81000000-0xe810fffff]
bus: 81 index 2 mmio: [0xe80000000-0xe80ffffff]
bus: 81 index 3 mmio: [0x0-0x0]

Working and non-working behaviour is identical on 2.6.27.7 and today's
-next tree.

12 more lines

Hello,

I enabled more DEBUG output, the online files have been updated. Snippets:

The detected end point boot:

pci 0001:80:00.0: scanning behind bridge, config bf8180, pass 0
PCI: Scanning bus 0001:81
pci 0001:81:00.0: found [1a0b:5331] class 00ff00 header type 00
pci 0001:81:00.0: reg 10 64bit mmio: [0xb8000000-0xb8ffffff]
pci 0001:81:00.0: reg 18 32bit mmio: [0xb9000000-0xb903ffff]
pci 0001:81:00.0: calling pcibios_fixup_resources+0x0/0xec
pci 0001:81:00.0: calling fixup_ppc4xx_pci_bridge+0x0/0x154
pci 0001:81:00.0: disabling ASPM on pre-1.1 PCIe device.  You can
enable it with 'pcie_aspm=force'
PCI: Fixups for bus 0001:81

The non-detected end point boot:

pci 0001:80:00.0: scanning behind bridge, config bf8180, pass 0
PCI: Scanning bus 0001:81
PCI: Fixups for bus 0001:81

Regards,
-- 
Leon

On Wed, Nov 26, 2008 at 4:25 PM, Leon Woestenberg
[off-list ref] wrote:

The non-detected end point boot:

pci 0001:80:00.0: scanning behind bridge, config bf8180, pass 0
PCI: Scanning bus 0001:81
PCI: Fixups for bus 0001:81

Further debugging.

drivers/pci/probe.c:
static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn)
{
...
	if (pci_bus_read_config_dword(bus, devfn, PCI_VENDOR_ID, &l))
		return NULL;
...
	/* some broken boards return 0 or ~0 if a slot is empty: */
	if (l == 0xffffffff || l == 0x00000000 ||
	    l == 0x0000ffff || l == 0xffff0000) {
  	printk(KERN_WARNING "--- slot empty?! l == 0x%lx\n", (u32)l);
		return NULL;
	}

It ends up here with l == 0xffffffff.

Note that u-boot finds the end point vendor ID, and Linux on x86 and
Linux on  mpc8316e powerpc also.

Regards,
-- 
Leon

Hello all,

On Wed, Nov 26, 2008 at 7:26 PM, Leon Woestenberg
[off-list ref] wrote:

On Wed, Nov 26, 2008 at 4:25 PM, Leon Woestenberg
[off-list ref] wrote:

quoted

The non-detected end point boot:

pci 0001:80:00.0: scanning behind bridge, config bf8180, pass 0
PCI: Scanning bus 0001:81
PCI: Fixups for bus 0001:81

Progress on this issue:

I found out that re-programming the end point FPGA again *just* after
*uboot* read it, and before Linux kernel has started, makes the end
point appear properly.

Either u-boot leaves the end point in bad state, or the root complex
reset also requires an end point reset. (or a third option I could not
think of yet).

I'll pick this up on ppc and u-boot mailing list with new info,
especially with the ppc4xx maintainers.

Consider this thread closed.

Thanks for ignoring my ranting :-)

Regards,
-- 
Leon

On Fri, 2008-11-28 at 11:54 +0100, Leon Woestenberg wrote:

I found out that re-programming the end point FPGA again *just* after
*uboot* read it, and before Linux kernel has started, makes the end
point appear properly.

Either u-boot leaves the end point in bad state, or the root complex
reset also requires an end point reset. (or a third option I could not
think of yet).

I think you are getting a reset yes, maybe your FPGA loses its
programming or something similar ?

I'll pick this up on ppc and u-boot mailing list with new info,
especially with the ppc4xx maintainers.

CC me,

Ben.