Hang with isync

11 messages, 4 authors, 2006-09-21 · open the first message on its own page

Hang with isync

From: Manoj Sharma <hidden>
Date: 2006-09-19 01:28:25

Hi,

We use linux kernel 2.4.20 on ppc405 and the system hangs once in a while
when isync gets called in this function:

_GLOBAL(_nmask_and_or_msr)
    mfmsr   r0      /* Get current msr */
    andc    r0,r0,r3    /* And off the bits set in r3 (first parm) */
    or  r0,r0,r4    /* Or on the bits in r4 (second parm) */
    sync            /* Some chip revs have problems here... */
    isync
    mtmsr   r0      /* Update machine state */
    isync
    blr         /* Done */

 2.5 onwards, I find that "sync; isync" has been replaced by a macro SYNC
(defined only for 601). I don't find it in any changelog and reason for the
change.

Can someone give some information on this change?

Appreciate any help.
Manoj

[off-list ref]

Re: Hang with isync

From: Manoj Sharma <hidden>
Date: 2006-09-20 01:16:08

Hi,

We use linux kernel 2.4.20 on ppc405 and the system hangs once in a while
when isync gets called in this function:

_GLOBAL(_nmask_and_or_msr)
    mfmsr   r0      /* Get current msr */
    andc    r0,r0,r3    /* And off the bits set in r3 (first parm) */
    or  r0,r0,r4    /* Or on the bits in r4 (second parm) */
    sync            /* Some chip revs have problems here... */
    isync
    mtmsr   r0      /* Update machine state */
    isync
    blr         /* Done */

 2.5 onwards, I find that "sync; isync" has been replaced by a macro SYNC
(defined only for 601). I don't find it in any changelog and reason for the
change.

Can someone give some information on this change?

Appreciate any help.
Manoj

[off-list ref]

Re: Hang with isync

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: 2006-09-20 21:35:22

On Tue, 2006-09-19 at 18:16 -0700, Manoj Sharma wrote:
        Hi,
        
        We use linux kernel 2.4.20 on ppc405 and the system hangs once
        in a while when isync gets called in this function: 
        
        _GLOBAL(_nmask_and_or_msr)
            mfmsr   r0      /* Get current msr */
            andc    r0,r0,r3    /* And off the bits set in r3 (first
        parm) */ 
            or  r0,r0,r4    /* Or on the bits in r4 (second parm) */
            sync            /* Some chip revs have problems here... */
            isync
            mtmsr   r0      /* Update machine state */
            isync
            blr         /* Done */ 
        
         2.5 onwards, I find that "sync; isync" has been replaced by a
        macro SYNC (defined only for 601). I don't find it in any
        changelog and reason for the change.
        
        Can someone give some information on this change? 
Regardless of the change... on 2.4, _nmask_and_or_msr() was used for a
number of things. We would need to know where it was called from with
what values as arguments to have an idea of what's going wrong. It's
probably not dying on the isync, but rather on the following mtmsr due
to a problem with the values passed in....

Ben.

Re: Hang with isync

From: Manoj Sharma <hidden>
Date: 2006-09-20 22:31:56

This is the stack trace.

Registers:
GPR00: 00069030 C01F3000 C01F1080 00000000 00048000 C0639F48 C01F1080
FFFFFC18
GPR08: C02203FC 00000020 C0638000 C01F31B0 42FEE022 1056A7F8 00FE502A
00000000
GPR16: 00000000 FFC44232 00000000 00000000 FFC441EC 00080000 00010000
0000000A
GPR24: 00000000 0007CD80 00000CE0 00000000 00000000 C02B0000 00000000
C02B0000

NIP; c0005da4 _<_nmask_and_or_msr+0x18/0x20 [kernel]>
Trace; c0025328 _<check_pgt_cache+0x20/0x30 [kernel]>
Trace; c0004f4c _<idled+0x58/0x70 [kernel]>
Trace; c0004f74 _<cpu_idle+0x10/0x24 [kernel]>
Trace; c00012b0 _<rest_init+0x30/0x40 [kernel]>
Trace; c02a45a4 _<start_kernel+0x168/0x17c [kernel]>
Trace; c0000250 _<skpinv+0x1f8/0x234 [kernel]>


On 9/20/06, Benjamin Herrenschmidt [off-list ref] wrote:
On Tue, 2006-09-19 at 18:16 -0700, Manoj Sharma wrote:
quoted
        Hi,

        We use linux kernel 2.4.20 on ppc405 and the system hangs once
        in a while when isync gets called in this function:

        _GLOBAL(_nmask_and_or_msr)
            mfmsr   r0      /* Get current msr */
            andc    r0,r0,r3    /* And off the bits set in r3 (first
        parm) */`
            or  r0,r0,r4    /* Or on the bits in r4 (second parm) */
            sync            /* Some chip revs have problems here... */
            isync
            mtmsr   r0      /* Update machine state */
            isync
            blr         /* Done */

         2.5 onwards, I find that "sync; isync" has been replaced by a
        macro SYNC (defined only for 601). I don't find it in any
        changelog and reason for the change.

        Can someone give some information on this change?
Regardless of the change... on 2.4, _nmask_and_or_msr() was used for a
number of things. We would need to know where it was called from with
what values as arguments to have an idea of what's going wrong. It's
probably not dying on the isync, but rather on the following mtmsr due
to a problem with the values passed in....

Ben.


Re: Hang with isync

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: 2006-09-20 22:38:23

On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
This is the stack trace.

Registers:
GPR00: 00069030 C01F3000 C01F1080 00000000 00048000 C0639F48 C01F1080
FFFFFC18
GPR08: C02203FC 00000020 C0638000 C01F31B0 42FEE022 1056A7F8 00FE502A
00000000
GPR16: 00000000 FFC44232 00000000 00000000 FFC441EC 00080000 00010000
0000000A 
GPR24: 00000000 0007CD80 00000CE0 00000000 00000000 C02B0000 00000000
C02B0000

NIP; c0005da4 _<_nmask_and_or_msr+0x18/0x20 [kernel]>
Trace; c0025328 _<check_pgt_cache+0x20/0x30 [kernel]>
Trace; c0004f4c _<idled+0x58/0x70 [kernel]> 
Trace; c0004f74 _<cpu_idle+0x10/0x24 [kernel]>
Trace; c00012b0 _<rest_init+0x30/0x40 [kernel]>
Trace; c02a45a4 _<start_kernel+0x168/0x17c [kernel]>
Trace; c0000250 _<skpinv+0x1f8/0x234 [kernel]> 
Is this upstream 2.6.20 or do you have any additional patches ? (Like
Montavista stuff or RT linux or whatever ?)

It's unclear to me from just that backtrace what mask it is... it could
just be re-enabling interrupt and you have a stale IRQ line asserted....

Have you also checked the errata list for your 405 core, in case it has
a known issue ?

Ben.
On 9/20/06, Benjamin Herrenschmidt [off-list ref] wrote:
        On Tue, 2006-09-19 at 18:16 -0700, Manoj Sharma wrote:
        >         Hi,
        >
        >         We use linux kernel 2.4.20 on ppc405 and the system
        hangs once
        >         in a while when isync gets called in this function: 
        >
        >         _GLOBAL(_nmask_and_or_msr)
        >             mfmsr   r0      /* Get current msr */
        >             andc    r0,r0,r3    /* And off the bits set in
        r3 (first
        >         parm) */`
        >             or  r0,r0,r4    /* Or on the bits in r4 (second
        parm) */ 
        >             sync            /* Some chip revs have problems
        here... */
        >             isync
        >             mtmsr   r0      /* Update machine state */
        >             isync
        >             blr         /* Done */ 
        >
        >          2.5 onwards, I find that "sync; isync" has been
        replaced by a
        >         macro SYNC (defined only for 601). I don't find it
        in any
        >         changelog and reason for the change. 
        >
        >         Can someone give some information on this change?
        
        Regardless of the change... on 2.4, _nmask_and_or_msr() was
        used for a
        number of things. We would need to know where it was called
        from with 
        what values as arguments to have an idea of what's going
        wrong. It's
        probably not dying on the isync, but rather on the following
        mtmsr due
        to a problem with the values passed in....
        
        Ben.
        
        
        

Re: Hang with isync

From: Linas Vepstas <hidden>
Date: 2006-09-20 23:04:10

On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin Herrenschmidt wrote:
On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
quoted
This is the stack trace.

Registers:
GPR00: 00069030 
This is the MSR and it has the user-mode bit set, which is surely wrong. 
This is not how one gets to user space.

00048000 

The MSR had this or'ed into it, which is setting the user-mode bit. 
Surely that's wrong.

--linas

Re: Hang with isync

From: Manoj Sharma <hidden>
Date: 2006-09-21 00:59:29

No MSR is 00029030 and user mode bit is not set here.

I had missed it in the prev mail:

NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000 REGS: c01eff30 TRAP:
1020    Not tainted
MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c01f1080[0] 'swapper' Last syscall: 120
last math 00000000 last altivec 00000000
PLB0: bear= 0x08000000 acr=   0xbb000000 besr=  0x00000000


On 9/20/06, Linas Vepstas [off-list ref] wrote:
On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin Herrenschmidt wrote:
quoted
On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
quoted
This is the stack trace.

Registers:
GPR00: 00069030
This is the MSR and it has the user-mode bit set, which is surely wrong.
This is not how one gets to user space.

00048000

The MSR had this or'ed into it, which is setting the user-mode bit.
Surely that's wrong.

--linas

RE: Hang with isync

From: Liu Dave-r63238 <hidden>
Date: 2006-09-21 04:24:56

No MSR is 00029030 and user mode bit is not set here.

I had missed it in the prev mail:

NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000 REGS: c01eff30
TRAP: 1020    Not tainted
MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 
TASK = c01f1080[0] 'swapper' Last syscall: 120
last math 00000000 last altivec 00000000
PLB0: bear= 0x08000000 acr=   0xbb000000 besr=  0x00000000
 
Dave>I notice that MSR and TRAP, MSR is 00029030- the critical interrupt
enable.
Dave>TRAP is 1020. --WatchDog timer exception is happening
Dave>You can disable the MSR[CE] bit to no critical exception or disable
the WD timer

 

	On 9/20/06, Linas Vepstas [off-list ref] wrote: 

		On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin
Herrenschmidt wrote:
		> On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
		> > This is the stack trace.
		> >
		> > Registers:
		> > GPR00: 00069030 
		
		This is the MSR and it has the user-mode bit set, which
is surely wrong.
		This is not how one gets to user space.
		
		00048000
		
		The MSR had this or'ed into it, which is setting the
user-mode bit.
		Surely that's wrong. 
		
		--linas
		

Re: Hang with isync

From: Manoj Sharma <hidden>
Date: 2006-09-21 06:17:54

Dave, watchdog timeout is around one second and no cpu activity for that
long is something wrong. Is it ok to disable it to hide the problem lying
somewhere else? Do you think it can be because of sync-isync
instructions and moving to 2.6 might resolve it?


On 9/20/06, Liu Dave-r63238 [off-list ref] wrote:

No MSR is 00029030 and user mode bit is not set here.

I had missed it in the prev mail:

NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000 REGS: c01eff30
TRAP: 1020    Not tainted
MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c01f1080[0] 'swapper' Last syscall: 120
last math 00000000 last altivec 00000000
PLB0: bear= 0x08000000 acr=   0xbb000000 besr=  0x00000000

Dave>I notice that MSR and TRAP, MSR is 00029030- the critical interrupt
enable.
Dave>TRAP is 1020. --WatchDog timer exception is happening
Dave>You can disable the MSR[CE] bit to no critical exception or disable
the WD timer



On 9/20/06, Linas Vepstas [off-list ref] wrote:
quoted
On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin Herrenschmidt wrote:
quoted
On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
quoted
This is the stack trace.

Registers:
GPR00: 00069030
This is the MSR and it has the user-mode bit set, which is surely wrong.
This is not how one gets to user space.

00048000

The MSR had this or'ed into it, which is setting the user-mode bit.
Surely that's wrong.

--linas

RE: Hang with isync

From: Liu Dave-r63238 <hidden>
Date: 2006-09-21 06:27:22

First, you must make sure if it really happen at watchdog timer
exception.
if it is, you need select one suitable way to fix it.
Second, I don't believe the sync-isync instructions make it happen.
you can try the 2.6, I don't know if 2.6 kernel can resolve your
problem.
 
-Dave



________________________________

	Dave, watchdog timeout is around one second and no cpu activity
for that long is something wrong. Is it ok to disable it to hide the
problem lying somewhere else? Do you think it can be because of
sync-isync instructions and moving to 2.6 might resolve it?

	 
	On 9/20/06, Liu Dave-r63238 [off-list ref] wrote: 

		

		No MSR is 00029030 and user mode bit is not set here.
		
		I had missed it in the prev mail:
		
		NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000
REGS: c01eff30 TRAP: 1020    Not tainted 
		MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 
		TASK = c01f1080[0] 'swapper' Last syscall: 120
		last math 00000000 last altivec 00000000
		PLB0: bear= 0x08000000 acr=   0xbb000000 besr=
0x00000000
		 
		Dave>I notice that MSR and TRAP, MSR is 00029030- the
critical interrupt enable.
		Dave>TRAP is 1020. --WatchDog timer exception is
happening
		Dave>You can disable the MSR[CE] bit to no critical
exception or disable the WD timer
		

		 

			On 9/20/06, Linas Vepstas <linas@austin.ibm.com
wrote: 
				On Thu, Sep 21, 2006 at 08:38:13AM
+1000, Benjamin Herrenschmidt wrote:
				> On Wed, 2006-09-20 at 15:31 -0700,
Manoj Sharma wrote: 
				> > This is the stack trace.
				> >
				> > Registers:
				> > GPR00: 00069030 
				
				This is the MSR and it has the user-mode
bit set, which is surely wrong.
				This is not how one gets to user space. 
				
				00048000
				
				The MSR had this or'ed into it, which is
setting the user-mode bit.
				Surely that's wrong. 
				
				--linas
				


Re: Hang with isync

From: Manoj Sharma <hidden>
Date: 2006-09-21 07:11:04

The hang has trigerred watchdog timer exception. It points that there is
problem somewhere but don't know what is it.
Even with 2.4, it does not happen regularly. It occurs once in a while and
not reproducable.

On 9/20/06, Liu Dave-r63238 [off-list ref] wrote:
 First, you must make sure if it really happen at watchdog timer
exception.
if it is, you need select one suitable way to fix it.
Second, I don't believe the sync-isync instructions make it happen.
you can try the 2.6, I don't know if 2.6 kernel can resolve your problem.

-Dave

 ------------------------------
  Dave, watchdog timeout is around one second and no cpu activity for that
long is something wrong. Is it ok to disable it to hide the problem lying
somewhere else? Do you think it can be because of sync-isync
instructions and moving to 2.6 might resolve it?


On 9/20/06, Liu Dave-r63238 [off-list ref] wrote:
quoted

No MSR is 00029030 and user mode bit is not set here.

I had missed it in the prev mail:

NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000 REGS: c01eff30
TRAP: 1020    Not tainted
MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c01f1080[0] 'swapper' Last syscall: 120
last math 00000000 last altivec 00000000
PLB0: bear= 0x08000000 acr=   0xbb000000 besr=  0x00000000

Dave>I notice that MSR and TRAP, MSR is 00029030- the critical interrupt
enable.
Dave>TRAP is 1020. --WatchDog timer exception is happening
Dave>You can disable the MSR[CE] bit to no critical exception or disable
the WD timer



On 9/20/06, Linas Vepstas <linas@austin.ibm.com > wrote:
quoted
On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin Herrenschmidt
wrote:
quoted
On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
quoted
This is the stack trace.

Registers:
GPR00: 00069030
This is the MSR and it has the user-mode bit set, which is surely
wrong.
This is not how one gets to user space.

00048000

The MSR had this or'ed into it, which is setting the user-mode bit.
Surely that's wrong.

--linas
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help