Thread (53 messages) 53 messages, 9 authors, 2014-02-23

pci-mvebu driver on km_kirkwood

From: Gerlando Falauto <hidden>
Date: 2014-02-19 09:39:07

Hi Thomas,

spoiler first: SUCCESS!!!!

On 02/19/2014 10:26 AM, Thomas Petazzoni wrote:
Dear Gerlando Falauto,
[...]
quoted
# hexdump -C /proc/device-tree/ocp at f1000000/pcie-controller/ranges
   | cut -c1-58
00000000  82 00 00 00 00 00 00 00  00 04 00 00 00 04 00 00
00000010  00 00 00 00 00 00 20 00  82 00 00 00 00 00 00 00
00000020  e0 00 00 00 e0 00 00 00  00 00 00 00 10 00 00 00
                                                 ^^^^^^^^^^^
Wow, that's an old DT representation that you have here :)
Indeed... ;-)
quoted hunk ↗ jump to hunk
But ok, let me try to explain. The 256 MB value that you define in the
DT is the global PCIe memory aperture: it is the maximum amount of
memory that we allow the PCIe driver to allocate for PCIe windows. But
depending on which PCIe devices you have plugged in, and how large
their BARs are, not necessarily all of these 256 MB will be used.

So, you can very well have a 256 MB global PCIe memory aperture, and
still have only one 1 MB PCIe memory window for PCIe 0.0 and a 256 KB
PCIe memory window for PCIe 1.0, and that's it.

Now, the 192 MB comes from the enumeration of your device. Linux
enumerates the BAR of your device:

pci 0000:01:00.0: BAR 1: assigned [mem 0xe0000000-0xe7ffffff]
pci 0000:01:00.0: BAR 3: assigned [mem 0xe8000000-0xe87fffff]
pci 0000:01:00.0: BAR 4: assigned [mem 0xe8800000-0xe8801fff]
pci 0000:01:00.0: BAR 0: assigned [mem 0xe8802000-0xe8802fff]
pci 0000:01:00.0: BAR 2: assigned [mem 0xe8803000-0xe8803fff]
pci 0000:01:00.0: BAR 5: assigned [mem 0xe8804000-0xe8804fff]

and then concludes that at the emulated bridge level, the memory region
to be created is:

pci 0000:00:01.0:   bridge window [mem 0xe0000000-0xebffffff]

which corresponds to the 192 MB window that we see created.

But I believe a 192 MB memory window cannot work with MBus, it should
be rounded up to the next power of 2. Can you try the below patch (not
tested, not even compiled, might need some tweaks to apply to your 3.10
kernel) :
diff --git a/drivers/pci/host/pci-mvebu.c b/drivers/pci/host/pci-mvebu.c
index 13478ec..002229a 100644
--- a/drivers/pci/host/pci-mvebu.c
+++ b/drivers/pci/host/pci-mvebu.c
@@ -372,6 +372,11 @@ static void mvebu_pcie_handle_membase_change(struct mvebu_pcie_port *port)
                 (((port->bridge.memlimit & 0xFFF0) << 16) | 0xFFFFF) -
                 port->memwin_base;

+       pr_info("PCIE %d.%d: creating window at 0x%x, size 0x%x rounded up to 0x%x\n",
+               port->port, port->lane, port->memwin_base,
+               port->memwin_size, roundup_pow_of_two(port->memwin_size));
+       port->memwin_size = roundup_pow_of_two(port->memwin_size);
+
         mvebu_mbus_add_window_by_id(port->mem_target, port->mem_attr,
                                     port->memwin_base, port->memwin_size);
  }
I'm obviously interested in seeing the message that gets shown, as well
as the new mvebu-mbus debugfs output.
----------
pci 0000:00:01.0:   bridge window [mem 0xe0000000-0xebffffff]
PCIE 0.0: creating window at 0xe0000000, size 0xbffffff rounded up to 
0x10000000
----------

  cat /sys/kernel/debug/mvebu-mbus/devices
[00] disabled
[01] disabled
[02] disabled
[03] disabled
[04] 00000000ff000000 - 00000000ff010000 : nand
[05] 00000000f4000000 - 00000000f8000000 : vpcie
[06] 00000000fe000000 - 00000000fe010000 : dragonite
[07] 00000000e0000000 - 00000000f0000000 : pcie0.0
For good measure, if you could also dump the registers of the PCIe
window. In your case, it was window 7, so dumping 0xf1020070 and
0xf1020074 would be useful.
Isn't that where the output of debugfs comes from?
quoted
But apart from that, what I still don't understand is how that could
have anything to do with my problem. The memory area I'm not able to
access starts at 0xe4000000.
BAR0, on the other hand, spawns 0xe8802000-0xe8802fff and seems to work
fine.
I am not sure, but since we are configuring an invalid memory size,
maybe the MBus behavior is undefined, and we get some completely funky
behavior, where parts of the 192 MB window are actually work, but parts
of it are not.
And... Ladies and gentlemen... it turns out YOU'RE RIGHT!!!
With your patch now everything works fine!!!

No words (or quads, for that matter) can express how grateful I am! ;-)

Thank you so much!!!
Gerlando
Thomas
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help