RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

From: Pravin Bathija <hidden>
Date: 2009-09-10 20:40:41

Tom Burns wrote=20
Hi,
=20
Thank you everyone for your help.
=20
I've been looking into the other dma/pci API calls

(dma_alloc_coherent,

pci_alloc_consistent).  I don't see how either of these return memory
mapped to a TLB with the I bit set to 1 in kernel 2.6.24.  In our
kernel
code, the only use of the PPC44x_TLB_I define is in head_44x.S in
_start.  We have CONFIG_NON_COHERENT_CACHE enabled.
=20
We changed our code to use dma_alloc_coherent, removed our manual
cacheline flushing, and saw the corrupted data return.  To me this
means
dma_alloc_coherent cannot be setting the I=3D1 bit in the TLB entry.
=20
I tried, using our JTAG debugger (BDI3000), to pause operation after
calling dma_alloc_coherent to examine the TLB entry for the memory
returned by the call (which was just past
CONFIG_CONSISTENT_START=3D0xff100000).  The TLB list loaded at the =

time

that I paused operation did not show a mapping for this area.  I guess
the kernel swaps TLB entries on the fly so it isn't limited to only 64
entries?  I will try to sleep in the same context as the
dma_alloc_coherent call to try to catch the TLB entry while loaded to
see if it has the I bit set.
=20
If that fails, any ideas?
=20
Thanks,
Tom Burns
International Datacasting Corporation
=20

There is also a patch that was submitted for 440EPX a couple of years
back. The 440EPX SOC causes hangs with Memory Read Multiple (MRM)
commands. Whether MRM is used or not depends on the value of
PCI_CACHE_LINE_SIZE register. I see that the changes are no longer
present in linux 2.6.30+ kernels. Although the patch certainly resolved
the hang issue with Silicon Image 680 PATA card as the 680 driver
attempts to use MRM commands - I don't know if it would resolve the data
corruption issue. It is certainly worth trying in my opinion. Below is a
link to the patch submission:

http://git.denx.de/?p=3Dlinux-2.6-denx.git;a=3Dcommit;h=3Dcffefde924123e6=
85327
48dd58fcb780eab5e219

Mikhail Zolotaryov wrote:

quoted

Hi Tom,

possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).


Tom Burns wrote:

quoted

Hi,

With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync().  This OOPses on PPC440 since it tries to call

directly

quoted

the assembly instruction dcbi, which can only be executed in
supervisor mode.  We tried that before resorting to manual cache

line

quoted

management with usermode-safe assembly calls.

Regards,
Tom Burns
International Datacasting Corporation

Mikhail Zolotaryov wrote:

quoted

Hi,

Why manage cache lines  manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)

Prodyut Hazarika wrote:

quoted

Hi Adam,

quoted

Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus

(using

quoted

the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on

timing)

quoted

end up being partially corrupted when we try to parse the data

in

the
quoted
quoted
quoted
quoted
quoted
virtual page. We have confirmed the data is good before the PCI-
IDE
quoted
quoted
quoted
quoted
quoted
bridge. We are creating two 8K pages and map them to physical

DMA

quoted

memory

quoted

using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of

16byte

quoted

cache lines) is overwritten with old data from the last time the

buffer

quoted

was used.

This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA

region

quoted

has
the CacheInhibit bit set.
You will need a BDI connected to your system.

Also, you will need to invalidate and flush the lines

appropriately,

quoted

since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for

example on

quoted

how to do it.

Thanks
Prodyut

On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:

quoted

Hi Adam,

quoted

Are you sure there is L2 cache on the 440?

It depends on the SoC you are using. SoC like 460EX (Canyonlands

board)

quoted

have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there

is

quoted

L2C.
For example, in canyonlands.dts (460EX based board), we have the

L2C

quoted

entry.
        L2C0: l2c {
              ...
        }

quoted

I am seeing this problem with our custom IDE driver which is
based on

quoted

pretty old code. Our driver uses pci_alloc_consistent() to

allocate

quoted

quoted
quoted
quoted
quoted
quoted
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual

page.

quoted

It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you

suggest.

quoted

quoted
quoted
quoted
quoted
quoted
Could you give more details on the consistency problem? It is a

good

quoted

idea to change to the new DMA APIs, but pci_alloc_consistent()

should

quoted

work too

Thanks
Prodyut  On Thu, 2009-09-03 at 19:57 +1000, Benjamin

Herrenschmidt

quoted

wrote:

quoted

On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:

quoted

Hi Adam,

If you have a look in include/asm-ppc/pgtable.h for the

following

quoted

quoted

section:

quoted

#ifdef CONFIG_44x
#define _PAGE_BASE    (_PAGE_PRESENT | _PAGE_ACCESSED |

_PAGE_GUARDED)

quoted

#else
#define _PAGE_BASE    (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif

Try adding _PAGE_COHERENT to the appropriate line above and

see

if
quoted
quoted
quoted
quoted
quoted
quoted
quoted
that
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the
page
quoted
quoted
quoted
quoted
quoted
quoted
quoted
which
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to
check
quoted
quoted
quoted
quoted
quoted
quoted
quoted
the
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was

originally

quoted

quoted
quoted
quoted
quoted
quoted
quoted
masked
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the

cache

quoted

quoted
quoted

quoted

coherency issues with non-SMP systems were resolved).

I have some doubts about the usefulness of doing that for 4xx.

AFAIK,

quoted

the 440 core just ignores M.

The problem lies probably elsewhere. Maybe the L2 cache

coherency

quoted

isn't

quoted

enabled or not working ?

The L1 cache on 440 is simply not coherent, so drivers have to

make

quoted

quoted
quoted
quoted
quoted
quoted
sure
quoted
they use the appropriate DMA APIs which will do cache flushing

when

quoted

needed.

Adam, what driver is causing you that sort of problems ?

Cheers,
Ben.

=20
=20
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help