Re: Machine Check in P2010(e500v2)

From: Joakim Tjernlund <hidden>
Date: 2017-09-20 16:45:26

On Sat, 2017-09-09 at 14:45 +0200, Joakim Tjernlund wrote:

On Fri, 2017-09-08 at 22:27 +0000, Leo Li wrote:

quoted

-----Original Message-----
From: Joakim Tjernlund [mailto:Joakim.Tjernlund@infinera.com]
Sent: Friday, September 08, 2017 7:51 AM
To: linuxppc-dev@lists.ozlabs.org; Leo Li <redacted>; York =

Sun

quoted

[off-list ref]
Subject: Re: Machine Check in P2010(e500v2)
=20
On Fri, 2017-09-08 at 11:54 +0200, Joakim Tjernlund wrote:

quoted

On Thu, 2017-09-07 at 18:54 +0000, Leo Li wrote:

quoted

-----Original Message-----
From: Joakim Tjernlund [mailto:Joakim.Tjernlund@infinera.com]
Sent: Thursday, September 07, 2017 3:41 AM
To: linuxppc-dev@lists.ozlabs.org; Leo Li <redacted>;
York Sun [off-list ref]
Subject: Re: Machine Check in P2010(e500v2)
=20
On Thu, 2017-09-07 at 00:50 +0200, Joakim Tjernlund wrote:

quoted

On Wed, 2017-09-06 at 21:13 +0000, Leo Li wrote:

quoted

-----Original Message-----
From: Joakim Tjernlund
[mailto:Joakim.Tjernlund@infinera.com]
Sent: Wednesday, September 06, 2017 3:54 PM
To: linuxppc-dev@lists.ozlabs.org; Leo Li
[off-list ref]; York Sun [off-list ref]
Subject: Re: Machine Check in P2010(e500v2)
=20
On Wed, 2017-09-06 at 20:28 +0000, Leo Li wrote:

quoted

-----Original Message-----
From: Joakim Tjernlund
[mailto:Joakim.Tjernlund@infinera.com]
Sent: Wednesday, September 06, 2017 3:17 PM
To: linuxppc-dev@lists.ozlabs.org; Leo Li
[off-list ref]; York Sun [off-list ref]
Subject: Re: Machine Check in P2010(e500v2)
=20
On Wed, 2017-09-06 at 19:31 +0000, Leo Li wrote:

quoted

-----Original Message-----
From: York Sun
Sent: Wednesday, September 06, 2017 10:38 AM
To: Joakim Tjernlund
[off-list ref];
linuxppc- dev@lists.ozlabs.org; Leo Li
[off-list ref]
Subject: Re: Machine Check in P2010(e500v2)
=20
Scott is no longer with Freescale/NXP. Adding Leo=

quoted

=20
On 09/05/2017 01:40 AM, Joakim Tjernlund wrote:

quoted

So after some debugging I found this bug:

@@ -996,7 +998,7 @@ int

fsl_pci_mcheck_exception(struct pt_regs

=20
*regs)

quoted

         if (is_in_pci_mem_space(addr)) {
                 if (user_mode(regs)) {
                         pagefault_disable();
-                       ret =3D get_user(regs->=

nip, &inst);

quoted

+                       ret =3D get_user(inst,
+ (__u32 __user *)regs->nip);
                         pagefault_enable();
                 } else {
                         ret =3D
probe_kernel_address(regs->nip, inst);
=20
However, the kernel still locked up after fixin=

g that.

quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
Now I wonder why this fixup is there in the fir=

st place?

quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
The routine will not really fixup the insn, jus=

quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
return 0xffffffff for the failing read and then=

 advance the

quoted
quoted
=20
process NIP.
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
=20
You are right.  The code here only gives 0xffffffff=

to

quoted

the load instructions and

=20
continue with the next instruction when the load
instruction is causing the machine check.  This will
prevent a system lockup when reading from PCI/RapidIO=

 device

quoted

=20
which is link down.

quoted

=20
I don't know what is actual problem in your case.
Maybe it is a write

=20
instruction instead of read?   Or the code is in a in=

finite loop

quoted
quoted
=20
waiting for
quoted
quoted
quoted
=20
a
quoted
quoted
quoted
=20
valid
quoted
quoted
read result?  Are you able to do some further debuggi=

ng

quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
with the NIP correctly printed?
quoted
=20
=20
According to the MC it is a Read and the NIP also lea=

ds

quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
to a read in the
=20
program.
quoted
quoted
ATM, I have disabled the fixup but I will enable that=

 again.

quoted

Question, is it safe add a small printk when this MC
happens(after fixing up)? I need to see that it has
happened as the error is somewhat

=20
random.

quoted

=20
I think it is safe to add printk as the current machine
check handlers are also

=20
using printk.
=20
I hope so, but if the fixup fires there is no printk at a=

ll so I was a bit

quoted
quoted
=20
unsure.
quoted
quoted
quoted
quoted
quoted
quoted
Don't like this fixup though, is there not a better way t=

han

quoted

faking a read to user space(or kernel for that matter) ?

=20
I don't have a better idea.  Without the fixup, the offendi=

ng

quoted

load instruction

=20
will never finish if there is anything wrong with the backing
device and freeze the whole system.  Do you have any suggestion=

 in mind?

quoted

=20

=20
But it never finishes the load, it just fakes a load of
0xfffffffff, for user space I rather have it signal a SIGBUS =

but

quoted
quoted
quoted
quoted
quoted
quoted
that does not seem to work either, at least not for us but th=

at

quoted

could be a bug in general MC code

=20
maybe.

quoted

This fixup might be valid for kernel only as it has never wor=

ked

quoted

for user space

=20
due to the bug I found.

quoted

=20
Where can I read about this errata ?

=20
I have look high and low an cannot find an errata which maps to=

 this fixup.

quoted

The closest I get is A-005125 which seems to have another
workaround, I cannot find any evidence that this workaround has=

 been

quoted

=20
applied in Linux, can you?

quoted

=20
This is not A-005125.  There was an erratum for this issue with o=

lder silicons

quoted

=20
(e.g. erratum PCI-ex 3 for MPC8572).

quoted

" When its link goes down, the PCI Express controller clears all
outstanding transactions with an error indicator and sends a link
down exception to the interrupt controller if PEX_PME_MES_DISR[LD=

DD]

quoted

=3D 0. If, however, any transactions are sent to the controller a=

fter

quoted

the link down event, they are accepted by the controller and wait
for the link to come back up before starting any timeout counters=

 (for

quoted

=20
example, completion timeout). There is no mechanism to cancel the new
transactions short of a device HRESET. "

quoted

=20
But it was removed in newer silicon like P2020/P2010 probably bec=

ause a

quoted

=20
Machine Check will be triggered in this situation to deal with the st=

alled

quoted

instruction and no longer considered it as a hardware issue.

quoted

=20

=20
Maybe this fixup should be configurable then?

=20
No.  My point is that the problem was no longer considered a hardware i=

ssue because of the machine check mechanism is in place to handle it.  If t=
here is no handling of this special case, we would still experience a syste=
m hang if this situation really occurs.

quoted

=20

quoted

=20

quoted

The A-005125 is dealt with in u-boot.

=20
https://emea01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2F=

lists.de

quoted

nx.de%2Fpipermail%2Fu-boot%2F2013-
August%2F161185.html&data=3D01%7C01%7Cleoyang.li%40nxp.com%7Ccb8a93e
0090e48eb53a008d4f6b84235%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0&
sdata=3D8sR4yoXA4adqMHz6TY%2BvmYpfCBTcYEZHjPuANjz%2F1EQ%3D&reserve
d=3D0

quoted

=20
Yes, I found it eventually :)
=20
However, I cannot return to normal execution. I can follow the code=

to

quoted

returning from
machine_check_exception() and moving into ASM handler for returning
from a ME but then I am a bit lost. It does not seem to be any prob=

lem

quoted

executing, it feels more like a SW bug dealing with machine checks.=

 Don't

quoted

=20
known how to diagnose this further and could use some pointers.

=20
Is the execution returned to the user application?  I doubt the system =

hang is caused by the machine check handling.

quoted

You can try to comment out the machine check handling code and check if=

 there is any improvement and see if

quoted

this is related to the machine check handling.

=20
It tries to return to user app but I cannot see what happens as the syste=

m lock up when the

MC returns.
How do you mean comment out MC handling? The simplest path is the PCI fix=

up which will

just do regs->nip +=3D 4; and then return to user space. That still does =

not work as

as soon MC handling returns, the system is locked up.
=20

quoted

=20
Machine check is a serious situation and not always possible to be reco=

vered from.=20

=20
This one should at least not kill the whole system. It is a simple bus er=

ror in user space and

the app should get SIGBUS and the the system should carry on.=20
=20

quoted

I would focus more on debugging why the machine check is triggered by t=

he user space application.

quoted

Can you locate what code is causing this machine check from user space?=

=20

quoted

Is it accessing some hardware related space which is not ready?=20
Or is it accessing address that it shouldn't have accessed?

=20
of course, this is ongoing and getting closer a solution. The MC looking =

the machine completely

does not make this any easier though.
These are 2 separate things, fixing the cause and not having a simple bus=

 error lock up the machine.

I am focusing on fixing the lockup.
=20
I have been following the execution in the kernel and I always end up in =

the ASM returning

from the MC.
The other day we got a similar PCI MC(bus error) on T1042 CPU(e5500/e500m=

c) and there

the system survived. The one thing I see different there is that MSR RI i=

s set

when entering MC, why is that?
=20
 Jocke

Got some more info now, this is a new errata I think, adding EDAC to the mi=
x yields:
[   28.372574] LTSSM:16
[   28.377197] Machine check in kernel mode.
[   28.381201] Caused by (from MCSR=3D10008, MCAR:0x8003e000): Bus - Read D=
ata Bus Error
[   28.388861] Oops: Machine check, sig: 7 [#1]
[   28.393125] P2010 E500v2
[   28.395651] Modules linked in: linux_bcm_knet(PO) linux_user_bde(PO) lin=
ux_kernel_bde(PO)
[   28.403842] CPU: 0 PID: 485 Comm: emxp2_hw_bl Tainted: P           O    =
4.1.43+ #19
[   28.411499] task: db13a0f0 ti: df17c000 task.ti: df17c000
[   28.416894] NIP: 10a66954 LR: 10a66a88 CTR: 0f9e7f44
[   28.421855] REGS: df17df10 TRAP: 0204   Tainted: P           O     (4.1.=
43+)
[   28.428901] MSR: 0002d000 <CE,EE,PR,ME>  CR: 44002428  XER: 20000000
[   28.435267] DEAR: b73cc000 ESR: 00000000=20
GPR00: 10a66a88 bfc21bc0 b7eee4a0 136eb4a0 00000000 00000000 00000000 00000=
000=20
GPR08: 0002d000 0003e000 b738e000 00000000 24002422 11db7334 00000000 00000=
000=20
GPR16: 10f8b054 10f895e5 10f8a8bf 0000b541 0000b541 11ddd380 00000011 00000=
001=20
GPR24: 01a9985e 136f1010 07000000 136eb4a0 00006000 07006000 00000000 00000=
000=20
[   28.467506] NIP [10a66954] 0x10a66954
[   28.471162] LR [10a66a88] 0x10a66a88
[   28.474730] Call Trace:
[   28.477170] ---[ end trace b25436dea505b49d ]---
[   28.481781]=20
[   28.483267] PCIe error(s) detected
[   28.486662] PCIe ERR_DR register: 0x00800000
[   28.490927] PCIe ERR_CAP_STAT register: 0x00000023
[   28.495713] PCIe ERR_CAP_R0 register: 0x00000000
[   28.500324] PCIe ERR_CAP_R1 register: 0x00000000
[   28.504936] PCIe ERR_CAP_R2 register: 0x00000000
[   28.509548] PCIe ERR_CAP_R3 register: 0x00000000

I logged LTSSM and it is 16(link up) and Ref. manual says this about ERR_DR=
 =3D 0x00800000:

PCIe ERR_DR: PCT bit
PCI Express completion time-out. A completion time-out condition was detect=
ed for a non-posted,
outbound PCI Express transaction. An error response is sent back to the req=
uestor. Note that a
completion timeout counter only starts when the non-posted request was able=
 to send to the link partner.
-
A completion time-out on the PCI Express link was detected. Note that a com=
pletion timeout error is a
fatal error. If a completion timeout error is detected, the system has beco=
me unstable. Hot reset is
recommended to restore stability of the system.

This error is not described in any errata I can find, how to workaround thi=
s?

   Jocke

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help