Accessing PCI-E resources on 460EX

Hi,

I'm currently struggling with some application
that tries to access PCI-E memory by mmaping
address found in /sys/bus/pci/devices/<device>/resource

mmap() fails with "invalid argument" error, apparently
because on 460EX PCI addresses are 36-bit.

What is the correct way of accessing PCI-E from
userspace on this platform.

Thanks a lot.

Felix Radensky
Embedded Solutions Ltd.
http://www.embedded-sol.com

linuxppc-embedded-bounces+aymane=3Dtanisys.com@ozlabs.org <> said on :

I'm currently struggling with some application that tries to
access PCI-E memory by mmaping address found in
/sys/bus/pci/devices/<device>/resource
=20
mmap() fails with "invalid argument" error, apparently
because on 460EX PCI addresses are 36-bit.
=20
What is the correct way of accessing PCI-E from userspace on this
platform.=20

What works for me is using the mmap64 as in the following example to
map the physical address, fd is the file descriptor for /dev/mem

off64_t offset =3D static_cast<off64_t>(your physical address);
void * const p =3D
mmap64(0,256,PROT_WRITE|PROT_READ,MAP_SHARED,fd,offset);

The only thing to note is that the address you pass should be aligned=20
to the page boundry even though some devices are not.

Ayman El-Khashab wrote:

What works for me is using the mmap64 as in the following example to
map the physical address, fd is the file descriptor for /dev/mem

off64_t offset = static_cast<off64_t>(your physical address);
void * const p =
mmap64(0,256,PROT_WRITE|PROT_READ,MAP_SHARED,fd,offset);

The only thing to note is that the address you pass should be aligned to the page boundry even though some devices are not.
  

Thanks, Ayman. I've tried that, but I'm still getting the same error.
My physical address is 0xe80000000, length is 0x100000.

Felix.

Did you try mmap2.  It probable is not in glibc so you will have to call
it using syscall.

#include <sys/syscall.h>

unsigned *pdata  = (unsigned *)syscall(__NR_mmap2,0,0x100000,PROT_READ|
PROT_WRITE,MAP_SHARED,fd, 0xe80000 );

The address is in page offsets that why the address is 0xe80000.


On Tue, 2008-11-04 at 00:53 +0200, Felix Radensky wrote:

Ayman El-Khashab wrote:

quoted

What works for me is using the mmap64 as in the following example to
map the physical address, fd is the file descriptor for /dev/mem

off64_t offset = static_cast<off64_t>(your physical address);
void * const p =
mmap64(0,256,PROT_WRITE|PROT_READ,MAP_SHARED,fd,offset);

The only thing to note is that the address you pass should be aligned 
to the page boundry even though some devices are not.
  

Thanks, Ayman. I've tried that, but I'm still getting the same error.
My physical address is 0xe80000000, length is 0x100000.

Felix.
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

On Mon, 2008-11-03 at 23:09 +0200, Felix Radensky wrote:

Hi,

I'm currently struggling with some application
that tries to access PCI-E memory by mmaping
address found in /sys/bus/pci/devices/<device>/resource

mmap() fails with "invalid argument" error, apparently
because on 460EX PCI addresses are 36-bit.

What is the correct way of accessing PCI-E from
userspace on this platform.

I fixed a lot of these problems recently. Can you try again with
2.6.28-latest-rc ?

Cheers,
Ben.

Benjamin Herrenschmidt wrote:

On Mon, 2008-11-03 at 23:09 +0200, Felix Radensky wrote:
  

quoted

Hi,

I'm currently struggling with some application
that tries to access PCI-E memory by mmaping
address found in /sys/bus/pci/devices/<device>/resource

mmap() fails with "invalid argument" error, apparently
because on 460EX PCI addresses are 36-bit.

What is the correct way of accessing PCI-E from
userspace on this platform.
    

I fixed a lot of these problems recently. Can you try again with
2.6.28-latest-rc ?
  

Thanks a lot for this input.
The problem is that I have a custom board with quite a few kernel modifications,
and I'm afraid it can take me some time to move from 2.6.26 to 2.6.28.
Any chance I can backport you modifications to 2.6.26 ? Have you modified
other files except arch/powerpc/sysdev/ppc4xx_pci.c to fix these problems ?

Cheers,
Ben.

_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-embedded
  

On Mon, 2008-12-01 at 13:08 +0200, Felix Radensky wrote:

and I'm afraid it can take me some time to move from 2.6.26 to 2.6.28.
Any chance I can backport you modifications to 2.6.26 ? Have you
modified
other files except arch/powerpc/sysdev/ppc4xx_pci.c to fix these
problems ?

Yes, quite a few changes in the PCI layer etc...

Board support changes shouldn't be too hard to port over tho, it depends
how you did them.

Cheers,
Ben.

Benjamin Herrenschmidt wrote:

On Mon, 2008-12-01 at 13:08 +0200, Felix Radensky wrote:
  

quoted

and I'm afraid it can take me some time to move from 2.6.26 to 2.6.28.
Any chance I can backport you modifications to 2.6.26 ? Have you
modified
other files except arch/powerpc/sysdev/ppc4xx_pci.c to fix these
problems ?

    

Yes, quite a few changes in the PCI layer etc...

Board support changes shouldn't be too hard to port over tho, it depends
how you did them.
  

I've managed to run 2.6.28-rc6 on this platform, it was not that difficult after all.
I only had to remove PCIX and PCIE0 from Canyonlands DTS, as they are not
used on this board. However this did not solve my problem. Below are relevant
bits from dmesg

PCIE1: Checking link...
PCIE1: Device detected, waiting for link...
PCIE1: link is up !
PCI host bridge /plb/pciex@d20000000 (primary) ranges:
MEM 0x0000000e80000000..0x0000000effffffff -> 0x0000000080000000
 IO 0x0000000f80010000..0x0000000f8001ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCIE1: successfully set as root-complex
PCI: Probing PCI hardware
PCI: Hiding 4xx host bridge resources 0000:80:00.0
pci 0000:81:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
pci 0000:80:00.0: PCI bridge, secondary bus 0000:81
pci 0000:80:00.0:   IO window: disabled
pci 0000:80:00.0:   MEM window: 0x80000000-0x800fffff
pci 0000:80:00.0:   PREFETCH window: disabled
bus: 80 index 0 io port: [0x00-0xffff]
bus: 80 index 1 mmio: [0xe80000000-0xeffffffff]
bus: 81 index 0 mmio: [0x0-0xfff]
bus: 81 index 1 mmio: [0xe80000000-0xe800fffff]
bus: 81 index 2 mmio: [0x0-0x0]
bus: 81 index 3 mmio: [0x0-0x0]

The application code that mmaps PCI memory is attached.
The attempt to access /sys/bus/pci/devices/0000:81:00.0/resource0
results in I/O error. The device in question is Mellanox Infiniband
switch. I will gladly test any other ideas you may have.

Felix.

Cheers,
Ben.


_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-embedded
  

Hi, Ben

Sorry for confusion. With 2.6.28-rc6 mmap() of resource0  works fine.
The application still doesn't work, but that's because it's not reading what
it expects from resource file. I guess it's no longer a kernel problem.

Thanks a lot for your help.

Felix.

Felix Radensky wrote:

Benjamin Herrenschmidt wrote:

quoted

On Mon, 2008-12-01 at 13:08 +0200, Felix Radensky wrote:
 

quoted

and I'm afraid it can take me some time to move from 2.6.26 to 2.6.28.
Any chance I can backport you modifications to 2.6.26 ? Have you
modified
other files except arch/powerpc/sysdev/ppc4xx_pci.c to fix these
problems ?

    

Yes, quite a few changes in the PCI layer etc...

Board support changes shouldn't be too hard to port over tho, it depends
how you did them.
  

I've managed to run 2.6.28-rc6 on this platform, it was not that difficult after all.
I only had to remove PCIX and PCIE0 from Canyonlands DTS, as they are not
used on this board. However this did not solve my problem. Below are relevant
bits from dmesg

PCIE1: Checking link...
PCIE1: Device detected, waiting for link...
PCIE1: link is up !
PCI host bridge /plb/pciex@d20000000 (primary) ranges:
MEM 0x0000000e80000000..0x0000000effffffff -> 0x0000000080000000
 IO 0x0000000f80010000..0x0000000f8001ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCIE1: successfully set as root-complex
PCI: Probing PCI hardware
PCI: Hiding 4xx host bridge resources 0000:80:00.0
pci 0000:81:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
pci 0000:80:00.0: PCI bridge, secondary bus 0000:81
pci 0000:80:00.0:   IO window: disabled
pci 0000:80:00.0:   MEM window: 0x80000000-0x800fffff
pci 0000:80:00.0:   PREFETCH window: disabled
bus: 80 index 0 io port: [0x00-0xffff]
bus: 80 index 1 mmio: [0xe80000000-0xeffffffff]
bus: 81 index 0 mmio: [0x0-0xfff]

26 more quoted lines