From: Brian Waite <hidden> Date: 2003-02-19 20:52:01
Hi all,
I am trying to hunt down a memory controller configuration problem and I have
been asked to disable all caching so as to remove it from the equation. I can
easl disable L2 but when I start ucking with the WIMG bits to set cache
inhibit, The kernel panics with stack overflows. Does anyone know where or
what I have to set to disable caching?
Thanks
Brian
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2003-02-19 21:07:59
On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
Hi all,
I am trying to hunt down a memory controller configuration problem and I have
been asked to disable all caching so as to remove it from the equation. I can
easl disable L2 but when I start ucking with the WIMG bits to set cache
inhibit, The kernel panics with stack overflows. Does anyone know where or
what I have to set to disable caching?
Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
Ben.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Gary Thomas <hidden> Date: 2003-02-19 22:48:50
On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
quoted
Hi all,
I am trying to hunt down a memory controller configuration problem and I have
been asked to disable all caching so as to remove it from the equation. I can
easl disable L2 but when I start ucking with the WIMG bits to set cache
inhibit, The kernel panics with stack overflows. Does anyone know where or
what I have to set to disable caching?
Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
There are other problems with this. With the caches disabled
(via HID0) all data cache instructions (like DCBF) will fail.
There are some of these in the kernel itself, but beware that
GLIBC has it's own set.
... totally disabling the data CACHE with Linux is non-trivial.
--
------------------------------------------------------------
Gary Thomas |
MLB Associates | Consulting for the
+1 (970) 229-1963 | Embedded world
http://www.mlbassoc.com/ |
email: [off-list ref] |
gpg: http://www.chez-thomas.org/gary/gpg_key.asc
------------------------------------------------------------
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Brian Waite <hidden> Date: 2003-02-20 13:55:12
According to The User's Manual, the data cache instructions become no-ops if
the data cache is disabled.
Thanks
Brian
On Wednesday 19 February 2003 5:48 pm, Gary Thomas wrote:
On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
quoted
On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
quoted
Hi all,
I am trying to hunt down a memory controller configuration problem and
I have been asked to disable all caching so as to remove it from the
equation. I can easl disable L2 but when I start ucking with the WIMG
bits to set cache inhibit, The kernel panics with stack overflows. Does
anyone know where or what I have to set to disable caching?
Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
There are other problems with this. With the caches disabled
(via HID0) all data cache instructions (like DCBF) will fail.
There are some of these in the kernel itself, but beware that
GLIBC has it's own set.
... totally disabling the data CACHE with Linux is non-trivial.
From: Gary Thomas <hidden> Date: 2003-02-20 14:01:39
On Thu, 2003-02-20 at 06:55, Brian Waite wrote:
According to The User's Manual, the data cache instructions become no-ops if
the data cache is disabled.
There are "User Manuals" and then there is "Real Life". My
experience has been that trying to execute these instructions
with the cache disabled was treated like an invalid instruction
(hence causing a trap). Maybe this isn't always true with all
PowerPC processors, but I was just pointing out that you might
run into trouble with it.
Thanks
Brian
On Wednesday 19 February 2003 5:48 pm, Gary Thomas wrote:
quoted
On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
quoted
On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
quoted
Hi all,
I am trying to hunt down a memory controller configuration problem and
I have been asked to disable all caching so as to remove it from the
equation. I can easl disable L2 but when I start ucking with the WIMG
bits to set cache inhibit, The kernel panics with stack overflows. Does
anyone know where or what I have to set to disable caching?
Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
There are other problems with this. With the caches disabled
(via HID0) all data cache instructions (like DCBF) will fail.
There are some of these in the kernel itself, but beware that
GLIBC has it's own set.
... totally disabling the data CACHE with Linux is non-trivial.
--
------------------------------------------------------------
Gary Thomas |
MLB Associates | Consulting for the
+1 (970) 229-1963 | Embedded world
http://www.mlbassoc.com/ |
email: [off-list ref] |
gpg: http://www.chez-thomas.org/gary/gpg_key.asc
------------------------------------------------------------
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
According to The User's Manual, the data cache instructions become no-ops if
the data cache is disabled.
There are "User Manuals" and then there is "Real Life". My
experience has been that trying to execute these instructions
with the cache disabled was treated like an invalid instruction
(hence causing a trap). Maybe this isn't always true with all
PowerPC processors, but I was just pointing out that you might
run into trouble with it.
There may be a different behavior for `disabling the data cache globally' and
`using e.g. dcbf on uncached memory' (with the data cache enabled globally).
quoted
On Wednesday 19 February 2003 5:48 pm, Gary Thomas wrote:
quoted
On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
quoted
On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
quoted
Hi all,
I am trying to hunt down a memory controller configuration problem and
I have been asked to disable all caching so as to remove it from the
equation. I can easl disable L2 but when I start ucking with the WIMG
bits to set cache inhibit, The kernel panics with stack overflows. Does
anyone know where or what I have to set to disable caching?
Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
There are other problems with this. With the caches disabled
(via HID0) all data cache instructions (like DCBF) will fail.
There are some of these in the kernel itself, but beware that
GLIBC has it's own set.
... totally disabling the data CACHE with Linux is non-trivial.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Brian Waite <hidden> Date: 2003-02-20 14:09:50
Gary,
Thanks for the clarification. I appreciate the real life experiences. This is
exactly why I said "in the User Manual". I was hoping you would come back
with the exact answer you provided. :) I'll keep my eyes open.
Thanks
Brian
On Thursday 20 February 2003 9:01 am, Gary Thomas wrote:
On Thu, 2003-02-20 at 06:55, Brian Waite wrote:
quoted
According to The User's Manual, the data cache instructions become no-ops
if the data cache is disabled.
There are "User Manuals" and then there is "Real Life". My
experience has been that trying to execute these instructions
with the cache disabled was treated like an invalid instruction
(hence causing a trap). Maybe this isn't always true with all
PowerPC processors, but I was just pointing out that you might
run into trouble with it.
quoted
Thanks
Brian
On Wednesday 19 February 2003 5:48 pm, Gary Thomas wrote:
quoted
On Wed, 2003-02-19 at 14:07, Benjamin Herrenschmidt wrote:
quoted
On Wed, 2003-02-19 at 21:52, Brian Waite wrote:
quoted
Hi all,
I am trying to hunt down a memory controller configuration problem
and I have been asked to disable all caching so as to remove it
from the equation. I can easl disable L2 but when I start ucking
with the WIMG bits to set cache inhibit, The kernel panics with
stack overflows. Does anyone know where or what I have to set to
disable caching?
Hrm... set L2 and L3 off, then hack HID0 to disable L1 ?
There are other problems with this. With the caches disabled
(via HID0) all data cache instructions (like DCBF) will fail.
There are some of these in the kernel itself, but beware that
GLIBC has it's own set.
... totally disabling the data CACHE with Linux is non-trivial.
From: Dan Malek <hidden> Date: 2003-02-20 15:23:27
Geert Uytterhoeven wrote:
There may be a different behavior for `disabling the data cache globally' and
`using e.g. dcbf on uncached memory' (with the data cache enabled globally).
I decided to take a diversion and read some processor manuals. :-)
The behavior of the cache instructions on areas that are not cacheable
for any reason depends upon the cache instruction used. Some have
no effect, some have unpredictable behavior, some cause alignment/access
traps. It all depends upon how they would translate/access the cache
and the operation they would perform on a cache line. We use all
instructions in Linux that would trigger any of the behavior. :-)
The bottom line appears to be you really don't want to execute cache
instructions on areas that are not cached. This includes global disable
or any method of marking pages uncached.
For Brian, I'm also curious why you think running Linux with disabled caches
will assist in debugging memory controller problems? If you are looking for
such fine control of the bus cycles for debugging it seems a simple
memory diagnostic used in conjuction with hardware debug tools is a
better approach.
Thanks.
-- Dan
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Brian Waite <hidden> Date: 2003-02-20 15:47:02
Well to preface, some people around here think disabling cache is a good idea.
I report to one of those :). What I am trying to debug is why my custom board
hangs. Basically, what I see is when a PCI interface starts reading/writing
SDRAM, my 745x processor locks up. I see no more decrementer interrupts being
serviced and using a BDI 2000 JTAG interface I see the processors stalled on
some memory access instruciton. The memory controller (gt64260) seems to be
doing something very wrong and it is thought to be a cache coherency issue.
So to prove this, I was asked to try running without any sort of cache.
Hopefully between Gary's information and yours, Dan I can convince people
that it is futile.
Thanks
Brian
On Thursday 20 February 2003 10:23 am, Dan Malek wrote:
Geert Uytterhoeven wrote:
quoted
There may be a different behavior for `disabling the data cache globally'
and `using e.g. dcbf on uncached memory' (with the data cache enabled
globally).
I decided to take a diversion and read some processor manuals. :-)
The behavior of the cache instructions on areas that are not cacheable
for any reason depends upon the cache instruction used. Some have
no effect, some have unpredictable behavior, some cause alignment/access
traps. It all depends upon how they would translate/access the cache
and the operation they would perform on a cache line. We use all
instructions in Linux that would trigger any of the behavior. :-)
The bottom line appears to be you really don't want to execute cache
instructions on areas that are not cached. This includes global disable
or any method of marking pages uncached.
For Brian, I'm also curious why you think running Linux with disabled
caches will assist in debugging memory controller problems? If you are
looking for such fine control of the bus cycles for debugging it seems a
simple memory diagnostic used in conjuction with hardware debug tools is a
better approach.
Thanks.
-- Dan
From: Gary Thomas <hidden> Date: 2003-02-20 15:54:19
On Thu, 2003-02-20 at 08:47, Brian Waite wrote:
Well to preface, some people around here think disabling cache is a good idea.
I report to one of those :). What I am trying to debug is why my custom board
hangs. Basically, what I see is when a PCI interface starts reading/writing
SDRAM, my 745x processor locks up. I see no more decrementer interrupts being
serviced and using a BDI 2000 JTAG interface I see the processors stalled on
some memory access instruciton. The memory controller (gt64260) seems to be
doing something very wrong and it is thought to be a cache coherency issue.
So to prove this, I was asked to try running without any sort of cache.
Hopefully between Gary's information and yours, Dan I can convince people
that it is futile.
I agree with Dan that you should try and reduce the scope of your test.
Try to [re]create the failure with as little as possible, hopefully
without Linux in the way, etc. (I'm recommending that you write a
separate test program) In this way, your test can have complete control
over the environment and doing things like verifying cache issues, etc,
should be simpler.
Thanks
Brian
On Thursday 20 February 2003 10:23 am, Dan Malek wrote:
quoted
Geert Uytterhoeven wrote:
quoted
There may be a different behavior for `disabling the data cache globally'
and `using e.g. dcbf on uncached memory' (with the data cache enabled
globally).
I decided to take a diversion and read some processor manuals. :-)
The behavior of the cache instructions on areas that are not cacheable
for any reason depends upon the cache instruction used. Some have
no effect, some have unpredictable behavior, some cause alignment/access
traps. It all depends upon how they would translate/access the cache
and the operation they would perform on a cache line. We use all
instructions in Linux that would trigger any of the behavior. :-)
The bottom line appears to be you really don't want to execute cache
instructions on areas that are not cached. This includes global disable
or any method of marking pages uncached.
For Brian, I'm also curious why you think running Linux with disabled
caches will assist in debugging memory controller problems? If you are
looking for such fine control of the bus cycles for debugging it seems a
simple memory diagnostic used in conjuction with hardware debug tools is a
better approach.
Thanks.
-- Dan
--
------------------------------------------------------------
Gary Thomas |
MLB Associates | Consulting for the
+1 (970) 229-1963 | Embedded world
http://www.mlbassoc.com/ |
email: [off-list ref] |
gpg: http://www.chez-thomas.org/gary/gpg_key.asc
------------------------------------------------------------
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2003-02-20 15:55:37
On Thu, 2003-02-20 at 16:47, Brian Waite wrote:
Well to preface, some people around here think disabling cache is a good idea.
I report to one of those :). What I am trying to debug is why my custom board
hangs. Basically, what I see is when a PCI interface starts reading/writing
SDRAM, my 745x processor locks up. I see no more decrementer interrupts being
serviced and using a BDI 2000 JTAG interface I see the processors stalled on
some memory access instruciton. The memory controller (gt64260) seems to be
doing something very wrong and it is thought to be a cache coherency issue.
So to prove this, I was asked to try running without any sort of cache.
Hopefully between Gary's information and yours, Dan I can convince people
that it is futile.
It could be the gt doing something wrong, it could also be the
CPU, 745x have known lockup conditions, though at least some of
them have workarounds enabled in the kernel.
Look closely at the CPU errata.
Ben.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Dan Malek <hidden> Date: 2003-02-20 16:14:03
Brian Waite wrote:
> .... The memory controller (gt64260) seems to be
doing something very wrong and it is thought to be a cache coherency issue.
The problem is cache coherency is central to the operation of processor
bus. One possibility, especially in the example you described, is the gt
is generating incorrect cache coherency control cycles on the bus. Disabling
the processor cache isn't going to affect this.
So to prove this, I was asked to try running without any sort of cache.
Make sure you disable it in the memory controller as well.
Hopefully between Gary's information and yours, Dan I can convince people
that it is futile.
Linux is a very poor diagnostic tool. When I run Linux and discover this
type of problem, I reduce it down into a non-Linux diagnostic that can
generate the same type of bus cycles. That way you can just flash it into
rom and let some hardware engineer run it over and over without having to
boot up something as complex as Linux and sorting through millions of
bus cycles for the failure mode. With these types of problems you have
to determine exactly what causes the failure. Since the processor hangs
so completely it should be easy to see the condition of the failure. If
it is a memory controller timing problem, you should see exactly what
failed and it should be clear how to program the controller to correct it.
Of course, like Ben said, check the errata.
Good Luck.
-- Dan
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
From: Mark A. Greer <hidden> Date: 2003-02-20 16:51:23
Brian,
I know you are aware of this but I *always* experience cache coherency
problems with the GT64260A when Sysclk/Tclk are anything above 100MHz
and I believe that your board is clocked over that. Try a 100MHz
crystal and see if the problem goes away.
Mark
--
Dan Malek wrote:
Brian Waite wrote:
quoted
.... The memory controller (gt64260) seems to be
quoted
doing something very wrong and it is thought to be a cache coherency
issue.
The problem is cache coherency is central to the operation of processor
bus. One possibility, especially in the example you described, is the gt
is generating incorrect cache coherency control cycles on the bus.
Disabling
the processor cache isn't going to affect this.
quoted
So to prove this, I was asked to try running without any sort of cache.
Make sure you disable it in the memory controller as well.
quoted
Hopefully between Gary's information and yours, Dan I can convince
people
that it is futile.
Linux is a very poor diagnostic tool. When I run Linux and discover this
type of problem, I reduce it down into a non-Linux diagnostic that can
generate the same type of bus cycles. That way you can just flash it
into
rom and let some hardware engineer run it over and over without having to
boot up something as complex as Linux and sorting through millions of
bus cycles for the failure mode. With these types of problems you have
to determine exactly what causes the failure. Since the processor hangs
so completely it should be easy to see the condition of the failure. If
it is a memory controller timing problem, you should see exactly what
failed and it should be clear how to program the controller to correct
it.
Of course, like Ben said, check the errata.
Good Luck.
-- Dan