On Thu, Mar 25, 2004 at 10:34:14AM -0600, linas@austin.ibm.com wrote:
On Thu, Mar 25, 2004 at 04:48:46PM +0100, Marc Leeman wrote:
quoted
As ever, insights are welcome :-/
OK, another wild guess: are the cache lines properly invalidated?
I think you said that the corrpution was a copy of earlier data ...
well, where could 'earlier data' be hiding? The offsets you're
reporting sound all wrong, but hey ...
actually, if you have 32-byte cache lines on your cpu, and you
have, umm, something maybe 4-way set associative (I've forgotten
how it works) that might explain it. I don't know what cpu's
have what cache sizes.
Don't know how hard it would be for you to run the test with cachine disabled,
but it might be worth a try.
--linas
From: Marc Leeman <hidden> Date: 2004-03-26 08:00:13
actually, if you have 32-byte cache lines on your cpu, and you have,
umm, something maybe 4-way set associative (I've forgotten how it
works) that might explain it. I don't know what cpu's have what cache
sizes.
This was also one of the paths we were considering, but the following
tests have been performed in order to validate or invalidate this
assumption:
1. I was under the impression that 'consistent_alloc' (which I
activited by defining CONFIG_NOT_COHERENT_CACHE in the kernel
configuration for our board) assured that the pages were not using
cache, at least the following lines lead me to this:
[malm@bpscltpd linux-2.4.17_bl21]$ vim ./arch/ppc/mm/cachemap.c
<...>
/*
* we need to ensure that there are no cachelines in use,
* or worse dirty in this area.
*/
invalidate_dcache_range(page, page + size);
<...>
and
<...>
flags = _PAGE_KERNEL | _PAGE_NO_CACHE;
err = 0;
for (i = 0; i < size && err == 0; i += PAGE_SIZE)
err = map_page(va+i, pa+i, flags);
<...>
As told before, no change with or without this define enabled that uses
consistent_alloc.
2. Next I tried to invalidate the cache the naive way:
I refilled the used buffer with 0xCA in userspace and copied it back to
kernelspace (in the PCI mappded buffer). Next I transferred the next
'useful' data from user to kernel space. Still completely the same: the
data showing up (corrupted) on the DSP side was the 'old' lines of the
buffer last transferred (i.e. the buffer before bringing 0xCA into
kernel space).
3. I took the previous point one step further: I filled the kernel
buffer with 0xCA before copying in the userbuffer. Still the same old
data is showing up on the DSP side. Still, I understand that none of
these two techniques assure purging the cache.
4. I added sync's and the like which should sync cache and mem, still no
change.
Apparently, the closest I can come to pinpointing the location of the
problem at the moment is that 'something' must be happening at the point
where the DSP starts reading from the PPC memory. When no such read is
performed, the problem does not occur. The other way around, the DSP can
write into the PPC memory without corrupting data.
Maybe reading of the DSP over PCI and the resulting DMA setup causes a
[hardware|software] timing problem with purging the cache (which should
not be used for these pages to start with (see [1]), or the DSP ack's
too early on and reading these 97 bytes after the DSP ACK causes some
hardware lock to be released, giving the DSP just enough time to
correctly fetch the data before the PPC starts pumping in new, ...
marc.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Jeff Angielski <hidden> Date: 2004-03-30 19:49:39
Marc,
How is the PICMRx configured? Do you have snoop enabled or disabled?
This will sound obvious, but have you stuck a PCI bus analyzer into the
system to watch the actual PCI transactions? They more than make up for
their cost in time savings. :)
Jeff
On Fri, 2004-03-26 at 03:00, Marc Leeman wrote:
quoted
actually, if you have 32-byte cache lines on your cpu, and you have,
umm, something maybe 4-way set associative (I've forgotten how it
works) that might explain it. I don't know what cpu's have what cache
sizes.
This was also one of the paths we were considering, but the following
tests have been performed in order to validate or invalidate this
assumption:
1. I was under the impression that 'consistent_alloc' (which I
activited by defining CONFIG_NOT_COHERENT_CACHE in the kernel
configuration for our board) assured that the pages were not using
cache, at least the following lines lead me to this:
[malm@bpscltpd linux-2.4.17_bl21]$ vim ./arch/ppc/mm/cachemap.c
<...>
/*
* we need to ensure that there are no cachelines in use,
* or worse dirty in this area.
*/
invalidate_dcache_range(page, page + size);
<...>
and
<...>
flags = _PAGE_KERNEL | _PAGE_NO_CACHE;
err = 0;
for (i = 0; i < size && err == 0; i += PAGE_SIZE)
err = map_page(va+i, pa+i, flags);
<...>
As told before, no change with or without this define enabled that uses
consistent_alloc.
2. Next I tried to invalidate the cache the naive way:
I refilled the used buffer with 0xCA in userspace and copied it back to
kernelspace (in the PCI mappded buffer). Next I transferred the next
'useful' data from user to kernel space. Still completely the same: the
data showing up (corrupted) on the DSP side was the 'old' lines of the
buffer last transferred (i.e. the buffer before bringing 0xCA into
kernel space).
3. I took the previous point one step further: I filled the kernel
buffer with 0xCA before copying in the userbuffer. Still the same old
data is showing up on the DSP side. Still, I understand that none of
these two techniques assure purging the cache.
4. I added sync's and the like which should sync cache and mem, still no
change.
Apparently, the closest I can come to pinpointing the location of the
problem at the moment is that 'something' must be happening at the point
where the DSP starts reading from the PPC memory. When no such read is
performed, the problem does not occur. The other way around, the DSP can
write into the PPC memory without corrupting data.
Maybe reading of the DSP over PCI and the resulting DMA setup causes a
[hardware|software] timing problem with purging the cache (which should
not be used for these pages to start with (see [1]), or the DSP ack's
too early on and reading these 97 bytes after the DSP ACK causes some
hardware lock to be released, giving the DSP just enough time to
correctly fetch the data before the PPC starts pumping in new, ...
marc.
From: Marc Leeman <hidden> Date: 2004-03-31 15:56:25
How is the PICMRx configured? Do you have snoop enabled or disabled?
This is what I like about my job: problem based digging deeper into code
internals :)
Anyway, the time of day difference between us enables me to try to come
up with some kind of founded answer. We are using ppcboot (yeah, I know,
but unfortunately, there are too many things in the todo queue to
upgraced to uboot).
After some digging into kernel code, ppcboot code and the mpc8245
documentation, I assume you are trying to point me to the following:
In Section 7.5:
"Because a snoop transaction is required to establish a lock, the
MPC8245 does not honour the assertion of a LOCK when PICR1[NO_SNOOP_EN]
is set."; when describing exclusive access to data.
In cpu/mpc824x/cpu_init.c:
CONFIG_WRITE_HALFWORD(PCICR, 0x06);
/* bus master, respond to PCI accesses */
CONFIG_READ_WORD(PICR1, val);
CONFIG_WRITE_WORD( PICR1,
(val & (PICR1_RCS0)) |
PICR1_PROC_TYPE_603E |
PICR1_FLASH_WR_EN | PICR1_MCP_EN |
PICR1_CF_DPARK | PICR1_NO_BUSW_CK |
PICR1_DEC| PICR1_CF_APARK | 0x10);
/* 8245 UM says bit 4 must be set */
#define PICR1_RCS0 0x00100000
#define PICR1_PROC_TYPE_603E 0x00040000
#define PICR1_FLASH_WR_EN 0x00001000
#define PICR1_MCP_EN 0x00000800
#define PICR1_CF_DPARK 0x00000200
#define PICR1_NO_BUSW_CK 0x00000080 /* no bus width check for
flash writes */
#define PICR1_DEC 0x00000100 /* Time Base enable on
8245/8241 */
#define PICR1_CF_APARK 0x00000008
And for PICR2:
CONFIG_READ_WORD(PICR2, val);
val= val & ~ (PICR2_CF_SNOOP_WS_MASK | PICR2_CF_APHASE_WS_MASK);
/*mask off waitstate bits*/
#define PICR2_CF_APHASE_WS_MASK 0x0000000c
#define PICR2_CF_SNOOP_WS_MASK 0x000c0000
k. I got the (relevant) registers states at bootup now, I am still
looking for the NO_SNOOP_EN bit, it must be somewhere in the docs...
4.7: Speculative PCI reads see Chapter 13;
but NO_SNOOP_EN is defined for PICR2 on bit 27, could it be that the
the description in section (note) 7.5 is referring to PICR2 instead of
the mentioned PICR1?
which should be 0x8000000
I'll masking PICR2 with this tomorrow...
will sound obvious, but have you stuck a PCI bus analyzer into the
system to watch the actual PCI transactions? They more than make up
for their cost in time savings. :)
Not yet, mainly because we want to interactively PPC/DSP rule out other
possibilities (an infamous DSP reset which is extremely difficult to
pinpoint); combined with the usual growing to do list with continuous
interrupts for 'paramount' things to do...
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Marc Leeman <hidden> Date: 2004-03-31 16:02:55
which should be 0x8000000 I'll masking PICR2 with this tomorrow...
Of course, I should't enable this bit, but at least I'll be able to see
if the situation gets worse by enabling it...
I need more coffee...
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Marc Leeman <hidden> Date: 2004-04-01 12:33:40
I'll masking PICR2 with this tomorrow...
I have obtained some results by setting some 8245 registers at bootup.
I tested the following:
. enabling NO_SNOOP_EN (27 in PICR2): nfs does not boot anymore
. increasing the snoop and address wait cycles to 5 (default ppc bootup),
no real change.
. disable speculative reads (2 in PICR1): doesn't boot nfs anymore
(via eepro100?)
No real success, until I came accross the PCMBCR register which defines
the number of PCI to Local Memory Read and Write Buffers. When I set
both to 1 ( & 0xF0), the data copy back is no longer needed to assure
data consistency between the PPC and DSP (default is 4 32 byte buffers).
include/mpc824x.h:
#define PCMBCR 0x800000e1 /* PCI/Memory Buffer Configuration
Register */
cpu/mpc824x/cpu_init.c:
CONFIG_READ_BYTE(PCMBCR,val);
CONFIG_WRITE_BYTE(PCMBCR,(val | 0xF0));
Double checking this by setting both on 4 again (during cpu_init of
ppcboot) resulted in severe MPEG data corruption. These results at least
seems to point in the direction you suggested.
This is the only way I seem to be able to assure data consistency (the
kernel copy back is no longer required), but the documentation also
suggests that (obviously) this degrades performance and is mainly used
for debugging.0
Still, when reading the explanations that could case this behaviour
(copy back buffer and filling the PCMRBs) , they point to cache
operations, which, I thought, were disabled in the linux kernel for
these particular PCI mapped buffers.
By adding __asm__ __volatile__("eieio"); in user and kernel space,
which lets the CCU buffers to be flushed, no change is observed (sect.
5.4.3.1; CCU Responses to the Processor Transactions).
It looks to me as if I should disable the bus snooping, but as mentioned
before, initialising this in ppc boot inhibits the NFS boot process.
Can this be disabled at runtime (possibly with re-enabling it) and if
so, is this a good practice to do so?
marc.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2004-04-04 22:53:55
On Thu, 2004-04-01 at 22:33, Marc Leeman wrote:
No real success, until I came accross the PCMBCR register which defines
the number of PCI to Local Memory Read and Write Buffers. When I set
both to 1 ( & 0xF0), the data copy back is no longer needed to assure
data consistency between the PPC and DSP (default is 4 32 byte buffers).
include/mpc824x.h:
#define PCMBCR 0x800000e1 /* PCI/Memory Buffer Configuration
Register */
cpu/mpc824x/cpu_init.c:
CONFIG_READ_BYTE(PCMBCR,val);
CONFIG_WRITE_BYTE(PCMBCR,(val | 0xF0));
Double checking this by setting both on 4 again (during cpu_init of
ppcboot) resulted in severe MPEG data corruption. These results at least
seems to point in the direction you suggested.
This is the only way I seem to be able to assure data consistency (the
kernel copy back is no longer required), but the documentation also
suggests that (obviously) this degrades performance and is mainly used
for debugging.0
Still, when reading the explanations that could case this behaviour
(copy back buffer and filling the PCMRBs) , they point to cache
operations, which, I thought, were disabled in the linux kernel for
these particular PCI mapped buffers.
By adding __asm__ __volatile__("eieio"); in user and kernel space,
which lets the CCU buffers to be flushed, no change is observed (sect.
5.4.3.1; CCU Responses to the Processor Transactions).
It looks to me as if I should disable the bus snooping, but as mentioned
before, initialising this in ppc boot inhibits the NFS boot process.
Can this be disabled at runtime (possibly with re-enabling it) and if
so, is this a good practice to do so?
Looks like your north bridge is horribly buggy...
Ben.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Adrian Cox <hidden> Date: 2004-04-05 08:46:09
On Thu, 2004-04-01 at 13:33, Marc Leeman wrote:
By adding __asm__ __volatile__("eieio"); in user and kernel space,
which lets the CCU buffers to be flushed, no change is observed (sect.
5.4.3.1; CCU Responses to the Processor Transactions).
It looks to me as if I should disable the bus snooping, but as mentioned
before, initialising this in ppc boot inhibits the NFS boot process.
Here's one possible scenario:
Your user space buffer is cleared with dcbz operations. This may be done
by standard library functions, rather than explicitly in your code. (If
that's not true, what follows could still bite someone else.)
The 603e core will only broadcast the dcbz when the page is marked
coherent. The page is only marked coherent when the kernel is compiled
for SMP.
If the above hypothesis is correct, the 824x CPUs will need to have
CPU_FTR_NEED_COHERENT set in this patch:
http://lists.linuxppc.org/linuxppc-embedded/200403/msg00218.html
- Adrian Cox
http://www.humboldt.co.uk/
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/