[powerpc] Intermittent crashes ( link_path_walk) with linux-next

Started by Sachin Sant on Mar 26, 2020 · 4 messages from 3 authors · Last reply Mar 26, 2020

open the first message on its own page

Sachin Sant

Message details and related content
Open message details and related content
I am running into intermittent crashes with linux-next on POWER 9 PowerVM LPAR
First it was against next-20200324 while running LTP tests. With next-20200325
I ran into similar crash (a different stack trace but same failure point — link_path_walk)
while running sosreport command.

BUG: Kernel NULL pointer dereference on read at 0x00000000
Faulting instruction address: 0xc00000000043f278
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in: loop iscsi_target_mod target_core_mod macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc overlay dm_mod nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip6_tables nft_compat ip_set rfkill nf_tables nfnetlink sunrpc sg pseries_rng uio_pdrv_genirq uio sch_fq_codel ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom t10_pi ibmvscsi scsi_transport_srp ibmveth
CPU: 26 PID: 7771 Comm: avocado Not tainted 5.6.0-rc7-next-20200324-autotest #1
NIP:  c00000000043f278 LR: c00000000043f330 CTR: 00000000000001fb
REGS: c00000082802f830 TRAP: 0300   Not tainted  (5.6.0-rc7-next-20200324-autotest)
MSR:  8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 28248442  XER: 20040000
CFAR: c00000000000dec4 DAR: 0000000000000000 DSISR: 40000000 IRQMASK: 0 
GPR00: c00000000043f330 c00000082802fac0 c00000000155e900 0000000000000000 
GPR04: 0000000000000002 0000000000000000 0000000000000002 c0000008b3400000 
GPR08: 0000000000031419 0000000000000000 0000000000000000 ffffffffffff0000 
GPR12: 0000000000008000 c00000001ec48600 00007fffa08a53f8 0000000000000001 
GPR16: 00007fff9faf9a63 00000100073bec00 00007fff9f2493b0 0000000000000000 
GPR20: 00007fffa1143bf8 00007fffa1103b18 c00000087f547cb3 2f2f2f2f2f2f2f2f 
GPR24: 0000000000000003 0000000000000000 c00000082802fbc8 fffffffffffff000 
GPR28: 0000000000200000 ffffffffffffffff 61c8864680b583eb 0000000000000000 
NIP [c00000000043f278] link_path_walk.part.49+0x228/0x400
LR [c00000000043f330] link_path_walk.part.49+0x2e0/0x400
Call Trace:
[c00000082802fac0] [c00000000043f330] link_path_walk.part.49+0x2e0/0x400 (unreliable)
[c00000082802fb50] [c00000000043f5a4] path_lookupat.isra.51+0x64/0x1f0
[c00000082802fba0] [c000000000441c00] filename_lookup.part.69+0xa0/0x1b0
[c00000082802fce0] [c00000000042ff38] vfs_statx+0xa8/0x190
[c00000082802fd60] [c0000000004302a0] __do_sys_newstat+0x40/0x90
[c00000082802fe20] [c00000000000b278] system_call+0x5c/0x68
Instruction dump:
3bffffff e93a0058 38800000 7f43d378 7fff07b4 1d5f0030 7d295214 eac90020 
4bfffb21 2fa30000 409e00c8 e93a0008 <81290000> 55290256 7f89e000 419efecc 
 ---[ end trace 34abf29ebd56e423 ]—

Relevant snippet from obj dump:

31 more lines

Michael Ellerman replied to Sachin Sant

Message details and related content
Open message details and related content
Sachin Sant [off-list ref] writes:
I am running into intermittent crashes with linux-next on POWER 9 PowerVM LPAR
First it was against next-20200324 while running LTP tests. With next-20200325
I ran into similar crash (a different stack trace but same failure point — link_path_walk)
while running sosreport command.

BUG: Kernel NULL pointer dereference on read at 0x00000000
Faulting instruction address: 0xc00000000043f278
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in: loop iscsi_target_mod target_core_mod macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc overlay dm_mod nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip6_tables nft_compat ip_set rfkill nf_tables nfnetlink sunrpc sg pseries_rng uio_pdrv_genirq uio sch_fq_codel ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom t10_pi ibmvscsi scsi_transport_srp ibmveth
CPU: 26 PID: 7771 Comm: avocado Not tainted 5.6.0-rc7-next-20200324-autotest #1
NIP:  c00000000043f278 LR: c00000000043f330 CTR: 00000000000001fb
REGS: c00000082802f830 TRAP: 0300   Not tainted  (5.6.0-rc7-next-20200324-autotest)
MSR:  8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 28248442  XER: 20040000
CFAR: c00000000000dec4 DAR: 0000000000000000 DSISR: 40000000 IRQMASK: 0 
GPR00: c00000000043f330 c00000082802fac0 c00000000155e900 0000000000000000 
GPR04: 0000000000000002 0000000000000000 0000000000000002 c0000008b3400000 
GPR08: 0000000000031419 0000000000000000 0000000000000000 ffffffffffff0000 
GPR12: 0000000000008000 c00000001ec48600 00007fffa08a53f8 0000000000000001 
GPR16: 00007fff9faf9a63 00000100073bec00 00007fff9f2493b0 0000000000000000 
GPR20: 00007fffa1143bf8 00007fffa1103b18 c00000087f547cb3 2f2f2f2f2f2f2f2f 
GPR24: 0000000000000003 0000000000000000 c00000082802fbc8 fffffffffffff000 
GPR28: 0000000000200000 ffffffffffffffff 61c8864680b583eb 0000000000000000 
NIP [c00000000043f278] link_path_walk.part.49+0x228/0x400
LR [c00000000043f330] link_path_walk.part.49+0x2e0/0x400
Call Trace:
[c00000082802fac0] [c00000000043f330] link_path_walk.part.49+0x2e0/0x400 (unreliable)
[c00000082802fb50] [c00000000043f5a4] path_lookupat.isra.51+0x64/0x1f0
[c00000082802fba0] [c000000000441c00] filename_lookup.part.69+0xa0/0x1b0
[c00000082802fce0] [c00000000042ff38] vfs_statx+0xa8/0x190
[c00000082802fd60] [c0000000004302a0] __do_sys_newstat+0x40/0x90
[c00000082802fe20] [c00000000000b278] system_call+0x5c/0x68
Instruction dump:
3bffffff e93a0058 38800000 7f43d378 7fff07b4 1d5f0030 7d295214 eac90020 
4bfffb21 2fa30000 409e00c8 e93a0008 <81290000> 55290256 7f89e000 419efecc 
 ---[ end trace 34abf29ebd56e423 ]—

Relevant snippet from obj dump:

17 more quoted lines

Yeah.

And r9 is NULL.

It came from r26 + 8, r26 is c00000082802fbc8  which looks OK.

r26 is nd, the nameidata:

struct nameidata {
	struct path	path;

struct path {
	struct vfsmount *mnt;
	struct dentry *dentry;
} __randomize_layout;

r26 + 8 should be the dentry in the struct path.

So we have an nd.path with a NULL dentry.


    6ddc:       56 02 29 55     rlwinm  r9,r9,0,9,11
                if (unlikely(!d_can_lookup(nd->path.dentry))) {
    6de0:       00 e0 89 7f     cmpw    cr7,r9,r28

The code in question (link_path_walk() in fs/namei.c ) was recently changed by
following commit:

commit 881386f7e46a: 
  link_path_walk(): sample parent's i_uid and i_mode for the last component
That and about 10 other commits.

Unless Al can give us a clue we'll need to bisect.

cheers


Crash while running sosreport command:

[ 1917.926113] BUG: Kernel NULL pointer dereference on read at 0x00000000
[ 1917.926126] Faulting instruction address: 0xc00000000043f638
[ 1917.926131] Oops: Kernel access of bad area, sig: 11 [#1]
[ 1917.926136] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
[ 1917.926147] Dumping ftrace buffer:
[ 1917.926157]    (ftrace buffer empty)
[ 1917.926161] Modules linked in: iscsi_target_mod target_core_mod macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc overlay dm_mod nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip6_tables nft_compat ip_set rfkill nf_tables nfnetlink sunrpc sg pseries_rng uio_pdrv_genirq uio sch_fq_codel ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod t10_pi ibmvscsi scsi_transport_srp ibmveth
[ 1917.926198] CPU: 12 PID: 20734 Comm: busctl Not tainted 5.6.0-rc7-next-20200325-autotest #1
[ 1917.926203] NIP:  c00000000043f638 LR: c00000000043f6f0 CTR: 0000000000000000
[ 1917.926209] REGS: c00000082bc1f720 TRAP: 0300   Not tainted  (5.6.0-rc7-next-20200325-autotest)
[ 1917.926215] MSR:  8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 28022422  XER: 20040000
[ 1917.926222] CFAR: c00000000000dec4 DAR: 0000000000000000 DSISR: 40000000 IRQMASK: 0 
[ 1917.926222] GPR00: c00000000043f6f0 c00000082bc1f9b0 c00000000155e800 0000000000000000 
[ 1917.926222] GPR04: 0000000000000002 0000000000000000 0000000000000002 c0000008b3400000 
[ 1917.926222] GPR08: 000000000000febb 0000000000000000 0000000000000000 ffffffffffff0000 
[ 1917.926222] GPR12: 0000000000002000 c00000001ec5d600 0000000135ed11e0 c0000008879046d0 
[ 1917.926222] GPR16: 7fffffffffffffff fffffffffffffe00 fffffffffffffffc fffffffffffffff5 
[ 1917.926222] GPR20: c00000089fa27200 0000000000000000 c00000008609bf33 2f2f2f2f2f2f2f2f 
[ 1917.926222] GPR24: 0000000000000003 0000000000000000 c00000082bc1fab8 fffffffffffff000 
[ 1917.926222] GPR28: 0000000000200000 ffffffffffffffff 61c8864680b583eb 0000000000000001 
[ 1917.926270] NIP [c00000000043f638] link_path_walk.part.49+0x228/0x400
[ 1917.926275] LR [c00000000043f6f0] link_path_walk.part.49+0x2e0/0x400
[ 1917.926280] Call Trace:
[ 1917.926284] [c00000082bc1f9b0] [c00000000043f6f0] link_path_walk.part.49+0x2e0/0x400 (unreliable)
[ 1917.926291] [c00000082bc1fa40] [c00000000043f964] path_lookupat.isra.51+0x64/0x1f0
[ 1917.926296] [c00000082bc1fa90] [c000000000441fc0] filename_lookup.part.69+0xa0/0x1b0
[ 1917.926303] [c00000082bc1fbd0] [c000000000ae0784] unix_find_other+0x64/0x3f0
[ 1917.926309] [c00000082bc1fc60] [c000000000ae1b78] unix_stream_connect+0x148/0x930
[ 1917.926316] [c00000082bc1fd30] [c000000000933010] __sys_connect+0x140/0x170
[ 1917.926322] [c00000082bc1fe00] [c000000000933068] sys_connect+0x28/0x40
[ 1917.926328] [c00000082bc1fe20] [c00000000000b278] system_call+0x5c/0x68
[ 1917.926332] Instruction dump:
[ 1917.926336] 3bffffff e93a0058 38800000 7f43d378 7fff07b4 1d5f0030 7d295214 eac90020 
[ 1917.926343] 4bfffb21 2fa30000 409e00c8 e93a0008 <81290000> 55290256 7f89e000 419efecc 
[ 1917.926351] ---[ end trace 1b673f0e7295c08b ]---

Objdump o/p
===========

382 more quoted lines

Al Viro replied to Michael Ellerman

Message details and related content
Open message details and related content
On Thu, Mar 26, 2020 at 10:40:06PM +1100, Michael Ellerman wrote:
quoted
The code in question (link_path_walk() in fs/namei.c ) was recently changed by
following commit:

commit 881386f7e46a: 
  link_path_walk(): sample parent's i_uid and i_mode for the last component
That and about 10 other commits.

Unless Al can give us a clue we'll need to bisect.
	Already fixed yesterday.  It's not link_path_walk(), it's handle_dots()
ignoring an error returned by step_into().

commit 5e3c3570ec97 is the broken one; commit 20971012f63e is its variant with the
fix folded in.  So next-20200325 has the bug and next-20200326 should have it
fixed.  Could you check the current -next and see if you still observe that crap?

Sachin Sant replied to Al Viro

Message details and related content
Open message details and related content
On 26-Mar-2020, at 7:19 PM, Al Viro [off-list ref] wrote:

On Thu, Mar 26, 2020 at 10:40:06PM +1100, Michael Ellerman wrote:

quoted
quoted
The code in question (link_path_walk() in fs/namei.c ) was recently changed by
following commit:

commit 881386f7e46a: 
 link_path_walk(): sample parent's i_uid and i_mode for the last component
That and about 10 other commits.

Unless Al can give us a clue we'll need to bisect.
	Already fixed yesterday.  It's not link_path_walk(), it's handle_dots()
ignoring an error returned by step_into().

commit 5e3c3570ec97 is the broken one; commit 20971012f63e is its variant with the
fix folded in.  So next-20200325 has the bug and next-20200326 should have it
fixed.  Could you check the current -next and see if you still observe that crap?
Thanks Al for the information. 

I confirm that today’s next tree (20200326) work for me. I can no longer recreate this
problem.

Thanks
-Sachin








    
  

  
    

      

        Keyboard shortcuts
      

      

        
          
hback out one level

          
jnext message in thread / next series-diff position

          
kprevious message in thread / previous series-diff position

          
ldrill in

          
Escclose help / fold thread tree

          
?toggle this help