Disabling L1 D-cache and side effects

11 messages, 3 authors, 2008-10-01 · open the first message on its own page

Disabling L1 D-cache and side effects

From: Tirumala Reddy Marri <hidden>
Date: 2008-09-29 17:05:45

 
Hi,
  I had to bring up a PPC based SOC with L1 dcache disabled.  I did that
and tried to boot Linux using RAMDISK/NFS mount. In RAMDISK I see the
file system errors. In case of NFS mount I see error saying failed to
load ld.so library.
 
 Could you guys please share thoughts what are the different side
effects might be causing this.
Thanks,
Marri


________________________________

From: Tirumala Reddy Marri 
Sent: Saturday, September 27, 2008 2:37 PM
To: <redacted>
Subject: rootfs mount problem


Hi ,
  I am trying to bring up a new SOC. I am seeing the following error.
Has any one seen this error before. I am pretty sure RAMDISK is not
corrupted.
Thanks,
Marri
------------------- LOG ---------------
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
prepare_namespace line 361 
prepare_namespace line 385 
RAMDISK: Compressed image found at block 0
invalid compressed format (err=1)
EXT3-fs error (device ram0): ext3_check_descriptors: Block bitmap for
group 0 not in group (block 14728687)!
EXT3-fs: group descriptors corrupted!
EXT2-fs error (device ram0): ext2_check_descriptors: Block bitmap for
group 0 not in group (block 14728687)!

24 more lines

Re: Disabling L1 D-cache and side effects

From: Olof Johansson <hidden>
Date: 2008-09-29 18:00:47

On Mon, Sep 29, 2008 at 10:05:41AM -0700, Tirumala Reddy Marri wrote:
 
Hi,
  I had to bring up a PPC based SOC with L1 dcache disabled.  I did that
and tried to boot Linux using RAMDISK/NFS mount. In RAMDISK I see the
file system errors. In case of NFS mount I see error saying failed to
load ld.so library.
 
 Could you guys please share thoughts what are the different side
effects might be causing this.
There are a number of things you have to be careful about when you
disable caches. Depending on your implementation, you likely can't use
lwarx/stwcx, cache ops will not work, etc. Also, are you 100% sure that
this is caused only by disabling L1D, and not by any other problems with
your silicon?

If you're doing early bringup of you'll have a whole lot of debug work
in front of you, and this mailing list is probably not the best place
to bring your homework.


-Olof

RE: Disabling L1 D-cache and side effects

From: Tirumala Reddy Marri <hidden>
Date: 2008-09-29 21:00:11

 Olof,
  Thanks for the response.

  Is there a piece of code in Linux which does the Machine check
recovery and continue normal execution ?
Thanks and Regards,
Marri

-----Original Message-----
From: Olof Johansson [mailto:<redacted>]=20
Sent: Monday, September 29, 2008 11:05 AM
To: Tirumala Reddy Marri
Cc: <redacted>
Subject: Re: Disabling L1 D-cache and side effects

On Mon, Sep 29, 2008 at 10:05:41AM -0700, Tirumala Reddy Marri wrote:
=20
Hi,
  I had to bring up a PPC based SOC with L1 dcache disabled.  I did=20
that and tried to boot Linux using RAMDISK/NFS mount. In RAMDISK I see
the file system errors. In case of NFS mount I see error saying failed
to load ld.so library.
=20
 Could you guys please share thoughts what are the different side=20
effects might be causing this.
There are a number of things you have to be careful about when you
disable caches. Depending on your implementation, you likely can't use
lwarx/stwcx, cache ops will not work, etc. Also, are you 100% sure that
this is caused only by disabling L1D, and not by any other problems with
your silicon?

If you're doing early bringup of you'll have a whole lot of debug work
in front of you, and this mailing list is probably not the best place to
bring your homework.


-Olof

Re: Disabling L1 D-cache and side effects

From: Olof Johansson <hidden>
Date: 2008-09-29 21:09:51

On Mon, Sep 29, 2008 at 02:00:06PM -0700, Tirumala Reddy Marri wrote:
 Olof,
  Thanks for the response.

  Is there a piece of code in Linux which does the Machine check
recovery and continue normal execution ?
Yes, there is. Several pieces, actually.


-Olof

RE: Disabling L1 D-cache and side effects

From: Tirumala Reddy Marri <hidden>
Date: 2008-09-29 21:38:53


Hi Olof,
   I see code in arch/powerpc/kernel/traps.c . But only
cbe_machine_check_handler() is the only function assigned to
ppc_md.machine_check_exception. Which is also not doing any recovery. It
just dumps the registers and return 0. Which would cause system Panic.


Could you please point me to the which does the Critical error (Machine
Check) recovery. BTW I am successful booting the Linux until rootfs is
being mounted. It fails to mount the Linux saying that blocks are
corrupted in file system. I had to modify lots of initial bring up code
to disable D-cache and make sure all TLB's are cache inhibited. Ando
also made sure none of the misc_32.S , entry_32.S and head.S makes any
references to d-cache.


Thanks,
Marri

-----Original Message-----
From: Olof Johansson [mailto:<redacted>]=20
Sent: Monday, September 29, 2008 2:14 PM
To: Tirumala Reddy Marri
Cc: <redacted>
Subject: Re: Disabling L1 D-cache and side effects

On Mon, Sep 29, 2008 at 02:00:06PM -0700, Tirumala Reddy Marri wrote:
 Olof,
  Thanks for the response.
=20
  Is there a piece of code in Linux which does the Machine check=20
recovery and continue normal execution ?
Yes, there is. Several pieces, actually.


-Olof

RE: Disabling L1 D-cache and side effects

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: 2008-09-30 07:16:30

On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
Could you please point me to the which does the Critical error (Machine
Check) recovery. BTW I am successful booting the Linux until rootfs is
being mounted. It fails to mount the Linux saying that blocks are
corrupted in file system. I had to modify lots of initial bring up code
to disable D-cache and make sure all TLB's are cache inhibited. Ando
also made sure none of the misc_32.S , entry_32.S and head.S makes any
references to d-cache.
Why the heck are you doing that btw ? AFAIK, as Olof says, things like
atomic operations will not work, dcbz neither etc... it's likely that
even if you manage to plaster around all of this in the kernel, whatever
userspace code you'll try to run in userspace will blow up too...

Cheers,
Ben.

RE: Disabling L1 D-cache and side effects

From: Tirumala Reddy Marri <hidden>
Date: 2008-09-30 16:57:11

=20
Ben,
  Thanks for the response. I am wondering how user space would get
affected by absence of L1 Dcache.
Thanks,
Marri

-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]=20
Sent: Tuesday, September 30, 2008 12:16 AM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects

On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
Could you please point me to the which does the Critical error=20
(Machine
Check) recovery. BTW I am successful booting the Linux until rootfs is
being mounted. It fails to mount the Linux saying that blocks are=20
corrupted in file system. I had to modify lots of initial bring up=20
code to disable D-cache and make sure all TLB's are cache inhibited.=20
Ando also made sure none of the misc_32.S , entry_32.S and head.S=20
makes any references to d-cache.
Why the heck are you doing that btw ? AFAIK, as Olof says, things like
atomic operations will not work, dcbz neither etc... it's likely that
even if you manage to plaster around all of this in the kernel, whatever
userspace code you'll try to run in userspace will blow up too...

Cheers,
Ben.

RE: Disabling L1 D-cache and side effects

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: 2008-09-30 21:30:46

On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote:
Ben,
  Thanks for the response. I am wondering how user space would get
affected by absence of L1 Dcache.
You didn't answer my question :-)

Well, as I said, things like lwarx/stwcx not working, dcbz taking
alignment exceptions, etc...

Ben.
Thanks,
Marri

-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org] 
Sent: Tuesday, September 30, 2008 12:16 AM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects

On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
quoted
Could you please point me to the which does the Critical error 
(Machine
Check) recovery. BTW I am successful booting the Linux until rootfs is
quoted
being mounted. It fails to mount the Linux saying that blocks are 
corrupted in file system. I had to modify lots of initial bring up 
code to disable D-cache and make sure all TLB's are cache inhibited. 
Ando also made sure none of the misc_32.S , entry_32.S and head.S 
makes any references to d-cache.
Why the heck are you doing that btw ? AFAIK, as Olof says, things like
atomic operations will not work, dcbz neither etc... it's likely that
even if you manage to plaster around all of this in the kernel, whatever
userspace code you'll try to run in userspace will blow up too...

Cheers,
Ben.

RE: Disabling L1 D-cache and side effects

From: Tirumala Reddy Marri <hidden>
Date: 2008-09-30 22:27:09

Ben,
    I got to bring up Linux on one of the 440 processors with out L1
dcache to do some bench marking  and compare with  L1 d-cache enabled. =20

I am avoiding any references to dcbz ,dcbt and dcbst .   Also the TLB's
are created with cache inhibited. I looked at lwarx/stwcx description,
there seem to be no dependency on L1 cache.

I don't see any critical exceptions or traps. All I  see is /init/bin
failing to execute because data is corrupted.=20

Thanks,
Marri


-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]=20
Sent: Tuesday, September 30, 2008 2:31 PM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects

On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote:
Ben,
  Thanks for the response. I am wondering how user space would get=20
affected by absence of L1 Dcache.
You didn't answer my question :-)

Well, as I said, things like lwarx/stwcx not working, dcbz taking
alignment exceptions, etc...

Ben.
Thanks,
Marri
=20
-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
Sent: Tuesday, September 30, 2008 12:16 AM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects
=20
On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
quoted
Could you please point me to the which does the Critical error=20
(Machine
Check) recovery. BTW I am successful booting the Linux until rootfs=20
is
=20
quoted
being mounted. It fails to mount the Linux saying that blocks are=20
corrupted in file system. I had to modify lots of initial bring up=20
code to disable D-cache and make sure all TLB's are cache inhibited.
Ando also made sure none of the misc_32.S , entry_32.S and head.S=20
makes any references to d-cache.
=20
Why the heck are you doing that btw ? AFAIK, as Olof says, things like
atomic operations will not work, dcbz neither etc... it's likely that=20
even if you manage to plaster around all of this in the kernel,=20
whatever userspace code you'll try to run in userspace will blow up
too...
=20
Cheers,
Ben.

RE: Disabling L1 D-cache and side effects

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: 2008-09-30 22:56:26

On Tue, 2008-09-30 at 15:26 -0700, Tirumala Reddy Marri wrote:
Ben,
    I got to bring up Linux on one of the 440 processors with out L1
dcache to do some bench marking  and compare with  L1 d-cache enabled.  

I am avoiding any references to dcbz ,dcbt and dcbst .   Also the TLB's
are created with cache inhibited. I looked at lwarx/stwcx description,
there seem to be no dependency on L1 cache.
Ok. Well, they are generally implemented at the L2 level but maybe not
on 440, architecturally, they must be used on cacheable memory but it's
possible that 440 being not SMP coherent, the actual implementation of
those is too dumb to care.
I don't see any critical exceptions or traps. All I  see is /init/bin
failing to execute because data is corrupted. 
Have you properly replaced dcbz with multiple stores ? I did some bring
up work internally on some stuff where dcbz wasn't quite there yet and
one pitfall to be careful is that if you force-enable the alternate
CONFIG_8xx implementation in the various copy & memset routines in
arch/powerpc/lib, you also need to fix those implementations to copy
or clear 32 bytes instead of just 16, as 8xx has 16 byte cache lines.

Typically failing to do so causes things like memset to fail to properly
clear things such as page tables and thus random crap occurs.

Cheers,
Ben.
Thanks,
Marri
-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org] 
Sent: Tuesday, September 30, 2008 2:31 PM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects

On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote:
quoted
Ben,
  Thanks for the response. I am wondering how user space would get 
affected by absence of L1 Dcache.
You didn't answer my question :-)

Well, as I said, things like lwarx/stwcx not working, dcbz taking
alignment exceptions, etc...

Ben.
quoted
Thanks,
Marri

-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
Sent: Tuesday, September 30, 2008 12:16 AM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects

On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
quoted
Could you please point me to the which does the Critical error 
(Machine
Check) recovery. BTW I am successful booting the Linux until rootfs 
is
quoted
being mounted. It fails to mount the Linux saying that blocks are 
corrupted in file system. I had to modify lots of initial bring up 
code to disable D-cache and make sure all TLB's are cache inhibited.
Ando also made sure none of the misc_32.S , entry_32.S and head.S 

11 more quoted lines

RE: Disabling L1 D-cache and side effects

From: Tirumala Reddy Marri <hidden>
Date: 2008-10-01 18:07:23


Ben,
  You are right. After I corrected copy_page and __copy_tofrom_user
functions Linux booted normally. Thanks a lot for the suggestions.
Thanks,
Marri

-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]=20
Sent: Tuesday, September 30, 2008 3:56 PM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects

On Tue, 2008-09-30 at 15:26 -0700, Tirumala Reddy Marri wrote:
Ben,
    I got to bring up Linux on one of the 440 processors with out L1=20
dcache to do some bench marking  and compare with  L1 d-cache enabled.
=20
I am avoiding any references to dcbz ,dcbt and dcbst .   Also the
TLB's
are created with cache inhibited. I looked at lwarx/stwcx description,
there seem to be no dependency on L1 cache.
Ok. Well, they are generally implemented at the L2 level but maybe not
on 440, architecturally, they must be used on cacheable memory but it's
possible that 440 being not SMP coherent, the actual implementation of
those is too dumb to care.
I don't see any critical exceptions or traps. All I  see is /init/bin=20
failing to execute because data is corrupted.
Have you properly replaced dcbz with multiple stores ? I did some bring
up work internally on some stuff where dcbz wasn't quite there yet and
one pitfall to be careful is that if you force-enable the alternate
CONFIG_8xx implementation in the various copy & memset routines in
arch/powerpc/lib, you also need to fix those implementations to copy or
clear 32 bytes instead of just 16, as 8xx has 16 byte cache lines.

Typically failing to do so causes things like memset to fail to properly
clear things such as page tables and thus random crap occurs.

Cheers,
Ben.
Thanks,
Marri
=20
-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
Sent: Tuesday, September 30, 2008 2:31 PM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects
=20
On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote:
quoted
Ben,
  Thanks for the response. I am wondering how user space would get=20
affected by absence of L1 Dcache.
=20
You didn't answer my question :-)
=20
Well, as I said, things like lwarx/stwcx not working, dcbz taking=20
alignment exceptions, etc...
=20
Ben.
=20
quoted
Thanks,
Marri
=20
-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
Sent: Tuesday, September 30, 2008 12:16 AM
To: Tirumala Reddy Marri
Cc: Olof Johansson; <redacted>
Subject: RE: Disabling L1 D-cache and side effects
=20
On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
quoted
Could you please point me to the which does the Critical error=20
(Machine
Check) recovery. BTW I am successful booting the Linux until=20
rootfs is
=20
quoted
being mounted. It fails to mount the Linux saying that blocks are=20
corrupted in file system. I had to modify lots of initial bring up
quoted
quoted
code to disable D-cache and make sure all TLB's are cache
inhibited.
quoted
quoted
Ando also made sure none of the misc_32.S , entry_32.S and head.S=20
makes any references to d-cache.
=20
Why the heck are you doing that btw ? AFAIK, as Olof says, things=20
like
=20
quoted
atomic operations will not work, dcbz neither etc... it's likely=20
that even if you manage to plaster around all of this in the kernel,
quoted
whatever userspace code you'll try to run in userspace will blow up
too...
quoted
=20
Cheers,
Ben.
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help