Re: [BUG] random kernel crashes after THP rework on s390 (maybe also on PowerPC and ARM)

On Mon, 15 Feb 2016 13:31:59 +0200
"Kirill A. Shutemov" [off-list ref] wrote:

On Sat, Feb 13, 2016 at 12:58:31PM +0100, Sebastian Ott wrote:

quoted

=20
On Sat, 13 Feb 2016, Kirill A. Shutemov wrote:

quoted

Could you check if revert of fecffad25458 helps?

=20
I reverted fecffad25458 on top of 721675fcf277cf - it oopsed with:
=20
=C2=A2 1851.721062! Unable to handle kernel pointer dereference in virt=

ual kernel address space

quoted

=C2=A2 1851.721075! failing address: 0000000000000000 TEID: 00000000000=

00483

quoted

=C2=A2 1851.721078! Fault in home space mode while using kernel ASCE.
=C2=A2 1851.721085! AS:0000000000d5c007 R3:00000000ffff0007 S:00000000f=

fffa800 P:000000000000003d

quoted

=C2=A2 1851.721128! Oops: 0004 ilc:3 =C2=A2#1! PREEMPT SMP DEBUG_PAGEAL=

LOC

quoted

=C2=A2 1851.721135! Modules linked in: bridge stp llc btrfs mlx4_ib mlx=

4_en ib_sa ib_mad vxlan xor ip6_udp_tunnel ib_core udp_tunnel ptp pps_core =
ib_addr ghash_s390raid6_pq prng ecb aes_s390 mlx4_core des_s390 des_generic=
 genwqe_card sha512_s390 sha256_s390 sha1_s390 sha_common crc_itu_t dm_mod =
scm_block vhost_net tun vhost eadm_sch macvtap macvlan kvm autofs4

quoted

=C2=A2 1851.721183! CPU: 7 PID: 256422 Comm: bash Not tainted 4.5.0-rc3=

-00058-g07923d7-dirty #178

quoted

=C2=A2 1851.721186! task: 000000007fbfd290 ti: 000000008c604000 task.ti=

: 000000008c604000

quoted

=C2=A2 1851.721189! Krnl PSW : 0704d00180000000 000000000045d3b8 (__rb_=

erase_color+0x280/0x308)

quoted

=C2=A2 1851.721200!            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3=

 CC:1 PM:0 EA:3

quoted

               Krnl GPRS: 0000000000000001 0000000000000020 00000000000=

00000 00000000bd07eff1

quoted

=C2=A2 1851.721205!            000000000027ca10 0000000000000000 000000=

0083e45898 0000000077b61198

quoted

=C2=A2 1851.721207!            000000007ce1a490 00000000bd07eff0 000000=

007ce1a548 000000000027ca10

quoted

=C2=A2 1851.721210!            00000000bd07c350 00000000bd07eff0 000000=

008c607aa8 000000008c607a68

quoted

=C2=A2 1851.721221! Krnl Code: 000000000045d3aa: e3c0d0080024       stg=

     %%r12,8(%%r13)

quoted

                          000000000045d3b0: b9040039           lgr     =

%%r3,%%r9

quoted

                         #000000000045d3b4: a53b0001           oill    =

%%r3,1

quoted

                         >000000000045d3b8: e33010000024       stg     =

%%r3,0(%%r1)

quoted

                          000000000045d3be: ec28000e007c       cgij    =

%%r2,0,8,45d3da

quoted

                          000000000045d3c4: e34020000004       lg      =

%%r4,0(%%r2)

quoted

                          000000000045d3ca: b904001c           lgr     =

%%r1,%%r12

quoted

                          000000000045d3ce: ec143f3f0056       rosbg   =

%%r1,%%r4,63,63,0

quoted

=C2=A2 1851.721269! Call Trace:
=C2=A2 1851.721273! (=C2=A2<0000000083e45898>! 0x83e45898)
=C2=A2 1851.721279!  =C2=A2<000000000029342a>! unlink_anon_vmas+0x9a/0x=

1d8

quoted

=C2=A2 1851.721282!  =C2=A2<0000000000283f34>! free_pgtables+0xcc/0x148
=C2=A2 1851.721285!  =C2=A2<000000000028c376>! exit_mmap+0xd6/0x300
=C2=A2 1851.721289!  =C2=A2<0000000000134db8>! mmput+0x90/0x118
=C2=A2 1851.721294!  =C2=A2<00000000002d76bc>! flush_old_exec+0x5d4/0x7=

00

quoted

=C2=A2 1851.721298!  =C2=A2<00000000003369f4>! load_elf_binary+0x2f4/0x=

13e8

quoted

=C2=A2 1851.721301!  =C2=A2<00000000002d6e4a>! search_binary_handler+0x=

9a/0x1f8

quoted

=C2=A2 1851.721304!  =C2=A2<00000000002d8970>! do_execveat_common.isra.=

32+0x668/0x9a0

quoted

=C2=A2 1851.721307!  =C2=A2<00000000002d8cec>! do_execve+0x44/0x58
=C2=A2 1851.721310!  =C2=A2<00000000002d8f92>! SyS_execve+0x3a/0x48
=C2=A2 1851.721315!  =C2=A2<00000000006fb096>! system_call+0xd6/0x258
=C2=A2 1851.721317!  =C2=A2<000003ff997436d6>! 0x3ff997436d6
=C2=A2 1851.721319! INFO: lockdep is turned off.
=C2=A2 1851.721321! Last Breaking-Event-Address:
=C2=A2 1851.721323!  =C2=A2<000000000045d31a>! __rb_erase_color+0x1e2/0=

x308

quoted

=C2=A2 1851.721327!
=C2=A2 1851.721329! ---=C2=A2 end trace 0d80041ac00cfae2 !---
=20
=20

quoted

=20
And could you share how crashes looks like? I haven't seen backtraces=

 yet.

quoted

quoted

=20

=20
Sure. I didn't because they really looked random to me. Most of the time
in rcu or list debugging but I thought these have just been the messeng=

er

quoted

observing a corruption first. Anyhow, here is an older one that might l=

ook

quoted

interesting:
=20
[   59.851421] list_del corruption. next->prev should be 000000006e1eb0=

00, but was 0000000000000400

=20
This kinda interesting: 0x400 is TAIL_MAPPING.. Hm..
=20
Could you check if you see the problem on commit 1c290f642101 and its
immediate parent?
=20

How should the page->mapping poison end up as next->prev in the list of
pre-allocated THP splitting page tables? Also, commit 1c290f642101
is before the THP rework, at least the non-bisectable part, so we should
expect not to see the problem there.

0x400 is also the value of an empty pte on s390, and the thp_deposit/withdr=
aw
listheads are placed inside the pre-allocated pagetables instead of page->l=
ru,
because we have 2K pagetables on s390 and cannot use struct page =3D=3D pgt=
able_t.

So, for example, two concurrent withdraws could produce such a list
corruption, because the first withdraw will overwrite the listhead at the
beginning of the pagetable with 2 empty ptes.

Has anything changed regarding the general THP deposit/withdraw logic?