PCI Memory mapping

I am currently trying the following

 +-----+        +-----+
 | mem |        | mem |
 +-----+        +-----+
    |              |
 +-----+        +-----+
 | ppc |        | dsp |
 +-----+        +-----+
    |     pci      |
    +--------------+

For a number of reasons (load on ppc, dsp silicon bug), we are trying to
transform the ppc dma transfers (ppc master write) to transfers
initiated by the dsp; where the memory block that is the result of
processing by the ppc is fetched by the dsp (ppc slave read).

Mapping pci devices registers and memory  into ppc kernel memory is well
documented, but I seem to be missing information about the other way
around: I (ppc) want to signal the dsp that data is ready at address
0xXXXXXXXX, come and get it (virt_to_bus(0xXXXXXXXX)?).

Most likely I have been searching archives with the wrong keywords since
I haven't found a good reference next to some notes in IO-mapping.txt
which seem to indicate that this is possible.

Any pointers about making cpu memory visible on the pci bus would greatly
be appreciated.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

On Tue, 2004-03-16 at 06:40, Marc Leeman wrote:

Any pointers about making cpu memory visible on the pci bus would greatly
be appreciated.

Check your PPC documentation for the configuration of the PITARx,
PIBARx, and PICMRx.  Once you read about these it should all make sense.

Then look at your bootloader source code (hopefully you are using
u-boot) to see how these registers are being configured.

HTH,
Jeff Angielski
The PTR Group


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Check your PPC documentation for the configuration of the PITARx,
PIBARx, and PICMRx. Once you read about these it should all make
sense. Then look at your bootloader source code (hopefully you are
using u-boot) to see how these registers are being configured.

Tnx for the info.

I am working on a solution, so instead of just posing a question on
the list, maybe the current working solution is a good contribution to
the lesser divine kernel hackers on the list :)

pci_map_single(driver_data.dev,driver_data.device[i]->\
    pciaddress,PCI_TRANSFER_SIZE,PCI_DMA_TODEVICE)

and

pci_unmap_single(driver_data.dev,driver_data.device[i]->\
    bus_addr,PCI_TRANSFER_SIZE,PCI_DMA_TODEVICE);

The data is transferred in blocks of 2k and first copied into a kernel
buffer from user space and then mapped on the PCI address range. Ideally
the user-space buffer would be mapped onto PCI (don't know if this is
possible, I did a quick, negative, test, but I suspect that I should
check the addresses again). However, this would put some logic at user
level, which is currently (in the transformation process) not desired.

The bus address (result of pci_map_single) is transferred to the DSP,
which it uses to fetch the data.

  +------+       +--------+     +------------+
  | user |------>| kernel |---->| pci-mapped |
  +------+       +--------+     +------------+

In the current kernel code (2.4.17 mvl2.1), pci_sync_single does not
seem to be implemented. In the code I checked, the successive mapping
and unmapping of the buffers does result in the same addresses.

I would assume that pci_sync_single would allow me to pass the mapped
address only once to the DSP, awaking it with an appropriate interrupt
when new data is available. Due to the absence of pci_sync_single, the
pci-mapped address needs to be passed to the DSP each time.

The observation based question is: to what degree can I be certain that
in a streaming data application, the successive mapping and unmapping of
the pci mapped addresses will result in the same bus addresses?

--
  Marc Leeman                                          Hardware R&D Engineer
  Barco Controlrooms Division                Noordlaan 5, B-8520 Kuurne (BE)
  Tel. +32 56 368 428 http://www.barcocontrolrooms.com marc.leeman@barco.com

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

the list, maybe the current working solution is a good contribution to
the lesser divine kernel hackers on the list :)

euhm, which include me :)

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

  +------+       +--------+     +------------+
  | user |------>| kernel |---->| pci-mapped |
  +------+       +--------+     +------------+

Hm, I'm not quite out of the woods yet it seems :-/

The system contains (amongst other components) a MPC8245 PPC and a
TMS320C6415 DSP. The PPC handles incoming data and passes it along to
the DSP. Partially due to a problem on the DSP, we want the DMA to be
initiated by the DSP.

As such, we prepare a kernel buffer:
if(!(driver_data.device[i]->kern_addr= \
  pci_alloc_consistent(driver_data.dev,PCI_TRANSFER_SIZE,\
  &(driver_data.device[i]->bus_addr)))){...}

that links/maps it on the PCI memory range. This dma_t (bus address) is
passed to the DSP. The DSP then reads the data over the PCI buss and
acknowledges this to the (currently polling) PPC.

The PPC prepares the data, and calls an ioctl that copies part of the
user buffer into kernel space (driver_data.device[i]->kern_addr);
following this, the DSP is warned that new data is available.

 ioctl(...,MAP_CONSISTENT,...);

 while(ptr<end){
   movebuftokernel(ptr,size); /* ioctl */
   ptr+=size;
 }

 ioctl(...,MAPOFF_CONSISTENT,...);

This process is repeated until the entire userspace buffer has been
processed.

Even though documentation indicates that consistent DMA mapping should
not have caching issues we have a problem that reflects this very much:

When data is passed to the DSP and every buffer is confirmed with an
[Enter] (i.e. __very__ slow_), data is passed correctly. When we try to
pass the same data without user intervention (comment out
waitforenter()),
parts of the transferred data are copied from other locations: most of
the time, part of the buffer (e.g. 2048 bytes) is not update yet (old
data).

e.g.

8 * 32 bit: correct
96 * 32 bit: wrong
... rest is OK.

Any ideas what might go wrong since consistent mappings 'should' not
have problems with caching...

--
  Marc Leeman                          Hardware R&D Engineer
  Barco Controlrooms Division Noordlaan 5, B-8520 Kuurne (BE)
  Tel. +32 56 368 428        http://www.barcocontrolrooms.com=20

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Any ideas what might go wrong since consistent mappings 'should' not
have problems with caching...

OK, more tests.

I tried to make certain that cache was disabled for these pages by
including CONFIG_NOT_COHERENT_CACHE (which defaults back to

  consistent_alloc
instead of
  __get_free_pages

But we did not notice a difference. Adding a user space buffer of 2 Megs
that just got filled with a counter makes the DMA transfers OK (this
should have purged the cache I think).

But so does adding a delay of 1 second between 2 transfers.

Between two transfers, I even filled the kernel buffer inbetween with
0xCA in kernel space and in user space (by copying a user buffer into
the kernel buffer again).

The strange thing is that the second buffer is always corrupted with
seemingly 'old' data, at an offset of 4 words (32 bit) and this for only
24 words, the rest of the buffer is fine.

The common factor seems to be 'timing', but unfortunately, this does not
yet pinpoint the problem to either the DSP or the PPC side. Any
experienced insight or well educated guesses?

--
  Marc Leeman                          Hardware R&D Engineer
  Barco Controlrooms Division Noordlaan 5, B-8520 Kuurne (BE)
  Tel. +32 56 368 428        http://www.barcocontrolrooms.com

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

On Tue, 2004-03-23 at 22:17, Marc Leeman wrote:

When data is passed to the DSP and every buffer is confirmed with an
[Enter] (i.e. __very__ slow_), data is passed correctly. When we try to
pass the same data without user intervention (comment out
waitforenter()),
parts of the transferred data are copied from other locations: most of
the time, part of the buffer (e.g. 2048 bytes) is not update yet (old
data).

One thing is: after having written your data to the consistent buffer,
you should make sure you have proper ordering with the MMIO write you
do to kick the DSP. On most CPUs, eieio appear to be enough and I
wouldn't suppose the 8245 to heavily reorder, but just in case, try
adding a sync.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

On Mar 23, 2004, at 11:01 AM, Marc Leeman wrote:

The strange thing is that the second buffer is always corrupted with
seemingly 'old' data, at an offset of 4 words (32 bit) and this for
only
24 words, the rest of the buffer is fine.

The common factor seems to be 'timing', but unfortunately, this does
not
yet pinpoint the problem to either the DSP or the PPC side. Any
experienced insight or well educated guesses?

Are you putting in sync instructions where appropriate?

--
----------------------------------------------
  Michael Zucca - mrz5149@acm.org
----------------------------------------------
  "I'm too old to use Emacs." -- Rod MacDonald
----------------------------------------------


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

I just added a [1]
  flush_cache_all();

after copying the buffer from user space, no change.

The same for
__asm__ __volatile__("sync\n");

However, more and more things seem to point to either the DSP or 'the
hardware':

For testing purposes, I allocate a 4k buffer and pass the address of the
buffer +0x400 to the DSP (the corrupted data always occur in the firsts
28 words: from word 5 to word 28).

In userspace, I fill the 0x400 bytes with 0xFF and only use the last
0xC00 bytes with 'userful' data. The 4k buffer is passed to kernel space
where it is mapped on the PCI.

  PPC user
  +-------+
  | 0x400 |
  +-------+
  |       |
  | 0xC00 |
  |       |
  +-------+

     || copy_from_user
     \/

  PPC kern            DSP
  +-------+
  | 0x400 |
  +-------+        +-------+
  |       |  PCI   |       |
  | 0xC00 |  ===>  | 0xC00 |
  |       |        |       |
  +-------+        +-------+



The DSP only reads the last 3k bytes but still, it gets the corrupted
data from word 5 to word 32 when no (excessive) delays are used.

Next to this, consistent_alloc() already uses _PAGE_NO_CACHE to avoid
caching of the buffer (called by pci_consistent_alloc()) in cachemap.c
(yes, the 8245 kernel configuration does not enable
CONFIG_NOT_COHERENT_CACHE normally [2], but I added it but the behaviour
is still identical).

Filling the user buffer with 0xCA after copy_from_user and re-copying it
into the kernel buffer (without interrupting the DSP this time) still
has no effect: the 'corrupted' data is still old data from the last
PCI transferred data on the DSP side (after a subsequent copy_from_user
with useful data and DSP interrupt).

However the first PCI transferred buffer is ALWAYS correct.

On the other hand, this DSP-type did read via PCI master read
operations and a PCI plugin card memory from w32 on an x86 arch (instead
of our internal PCI bus) a couple of yrs ago I'm told.

[1] hmmmmm, this is helpful code for ppc :)

/*
 * No cache flushing is required when address mappings are
 * changed, because the caches on PowerPCs are physically
 * addressed.  -- paulus
 * Also, when SMP we use the coherency (M) bit of the
 * BATs and PTEs.  -- Cort
 */
#define flush_cache_all()               do { } while (0)


[2]
6xx/7xx/74xx/8260      CONFIG_6xx
instead of
CONFIG_8xx

as Processor Type.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

This is getting weird, I found a solution to the problem, well kind of.

Let's introduce the major players in this drama:

 /* communication struct between user and kernel space , in driver.h
  * kernel include */
 typedef struct SStreamPci {
         unsigned int pciaddress;
         unsigned int useraddress;
         unsigned char stream;
 }TSStreamPci;

 /* user space program */
 /* as long as not all data is transferred */
 while(userptr < (userbuffer + (TRANSFER_SIZE>>2))){
   /* currentstream is a struct that is passed to the kernel of
    * type TSStreamPci */
   currentstream.useraddress = (unsigned)userptr;

   /* copy our buffer to the kernel (PCI_TRANSFER_SIZE), 4k */
   ioctl(dsphandle,PPC2DSP_CONSISTENT_COPY_TO_KERNEL,&currentstream);

   /* Interrupt the DSP */
   WriteToDSP(PacketReady,0x1);
   ioctl(dsphandle,PPC2DSP_SET_HDCR,&interrupt);

   /* Wait until the DSP acknowledges transfer */
   while(ReadFromDSP(PacketReady)){}

   /* Read kernel buffer back */
   ioctl(dsphandle,PPC2DSP_CONSISTENT_COPY_FROM_KERNEL,&currentstream);

   /* increment pointer */
   userptr += (BLOCK_TRANSFER_SIZE >> 2);
 }

The strange thing is that this works only and only if I do a transfer
back from kernel space into the userbuffer AFTER the DSP has
acknowledged transferring the data. Note that this data is never used
again. Doing this copy_to_user before DSP acknowledgement still results
in the afore mentioned data corruption.

The associated ioctls are:
case PPC2DSP_CONSISTENT_COPY_TO_KERNEL:
  {
    TSStreamPci currentstream;
    if(copy_from_user(&currentstream,(TSStreamPci*) arg,sizeof(TSStreamPci))){
      return -EFAULT;
    }
    if(copy_from_user(driver_data.device[currentstream.stream]->kern_addr,(char*)currentstream.useraddress, PCI_TRANSFER_SIZE)){
      return -EFAULT;
    }
    break;
  }
case PPC2DSP_CONSISTENT_COPY_FROM_KERNEL:
  {
    TSStreamPci currentstream;
    if(copy_from_user(&currentstream,(TSStreamPci*) arg, sizeof(TSStreamPci))){
      return -EFAULT;
    }
    if(copy_to_user((char*)currentstream.useraddress,driver_data.device[currentstream.stream]->kern_addr, PCI_TRANSFER_SIZE)){
      return -EFAULT;
    }
    break;
  }

There is about a 10% bandwith reduction (due to PPC utilisation I would
assume) over the PCI, so I can live with that; but what bothers me is
not knowing _why_, ... Especially since I want my first module not to
contain too large beginners mistakes :)

  marc.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/