Crash in netlink/sk_filter_trim_cap on ARMv7 on 4.18rc1

Hi All,

I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter

[    9.955543] Modules linked in:
[    9.955562] CPU: 1 PID: 213 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl #1
[    9.955566] Hardware name: BCM2835
[    9.955584] PC is at sk_filter_trim_cap+0x15c/0x1b8
[    9.955590] LR is at   (null)
[    9.955597] pc : [<c09d4d58>]    lr : [<00000000>]    psr: 60000013
[    9.955602] sp : c2cf9d58  ip : 00000000  fp : 00000000
[    9.955608] r10: ef2c3c00  r9 : c13093c0  r8 : 00000000
[    9.955615] r7 : 00000000  r6 : 00000001  r5 : f0f6a000  r4 : 00000000
[    9.955621] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    9.955629] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[    9.955640] Control: 10c5387d  Table: 02e6406a  DAC: 00000051
[    9.963334] Unable to handle kernel NULL pointer dereference at
virtual address 0000000c
[    9.964631] Process systemd-udevd (pid: 213, stack limit = 0x(ptrval))
[    9.964640] Stack: (0xc2cf9d58 to 0xc2cfa000)
[    9.964649] 9d40:
    00000000 c2c90540
[    9.964663] 9d60: 006000c0 00000000 00000000 c09a233c c2c90b40
c2c90b40 c2c90540 00000000
[    9.964678] 9d80: 00000000 00000000 c13093c0 c09fa2bc 006000c0
00000001 ee7f1800 00000000
[    9.964691] 9da0: 00000002 00000000 00000001 ef2c3c64 c2cf9f70
00000002 c2c90540 00000000
[    9.964706] 9dc0: c2cf9f68 00000083 ee7f1800 00000008 00000000
c09fa3b8 006000c0 00000000
[    9.964724] 9de0: 00000000 00000002 00000002 c09fc704 006000c0
00000000 ee7c7c00 00000000
[    9.976159] pgd = (ptrval)
[    9.979536] 9e00: 000000d5 00000000 00000000 00000000 c126a314
c2cf9f68 eec77880 c2cf9e50
[    9.979550] 9e20: 00000040 00000000 eec77880 00000000 00000000
c099a624 c2cf9f68 00000000
[    9.979565] 9e40: c2cf9e50 c099ae48 00000100 00000000 00000080
c04ab918 ee78e8c0 7fff0000
[    9.979580] 9e60: c2cf9e90 c2cf9eec ffff0000 000000a0 bed817e4
00000028 01a040a8 0000005b
[    9.979594] 9e80: 00000000 00000000 00000000 01a0ef00 00000128
40000028 b6cd9548 00000000
[    9.979607] 9ea0: 0000000d 00000000 bed817b8 00000000 00000010
00000000 00000002 00000000
[    9.985866] [0000000c] *pgd=00000000
[    9.988810] 9ec0: 00000000 00000000 01a0ef00 00000000 c2cf9fb0
00000128 bed817b8 00000000
[    9.988825] 9ee0: 00000000 c0407f18 00000000 00000000 c120bbec
b6e2ba00 c2cf9fb0 10c5387d
[    9.988841] 9f00: 01a0efb8 bed81720 bed81728 c03165fc 00005010
00001000 3e600000 c04ced24
[    9.988855] 9f20: ee4b5010 00000ff0 ee4b5000 00000000 ee4b6000
eec77880 bed817b8 00000000
[    9.988875] 9f40: 00000128 c0301204 c2cf8000 00000128 00000000
c099bc5c 00000000 00000000
[   10.000948] 9f60: 00000000 fffffff7 c2cf9eb0 0000000c 00000001
00000000 00000000 c2cf9e80
[   10.000961] 9f80: 00000000 c030ac08 00000000 00000000 00000040
00000000 00000000 01a0ef00
[   10.000976] 9fa0: bed817b8 c03011d4 00000000 01a0ef00 0000000d
bed817b8 00000000 00000000
[   10.000995] 9fc0: 00000000 01a0ef00 bed817b8 00000128 0000005b
01a0af00 01a0f620 00000000
[   10.228876] 9fe0: b6f9fad4 bed81780 b6de4780 b6cd9548 60000010
0000000d 00000000 00000000
[   10.237081] [<c09d4d58>] (sk_filter_trim_cap) from [<c09fa2bc>]
(netlink_broadcast_filtered+0x304/0x3dc)
[   10.246575] [<c09fa2bc>] (netlink_broadcast_filtered) from
[<c09fa3b8>] (netlink_broadcast+0x24/0x2c)
[   10.255806] [<c09fa3b8>] (netlink_broadcast) from [<c09fc704>]
(netlink_sendmsg+0x30c/0x340)
[   10.264258] [<c09fc704>] (netlink_sendmsg) from [<c099a624>]
(sock_sendmsg+0x3c/0x4c)
[   10.272100] [<c099a624>] (sock_sendmsg) from [<c099ae48>]
(___sys_sendmsg+0x1d8/0x218)
[   10.280030] [<c099ae48>] (___sys_sendmsg) from [<c099bc5c>]
(__sys_sendmsg+0x48/0x6c)
[   10.287872] [<c099bc5c>] (__sys_sendmsg) from [<c03011d4>]
(__sys_trace_return+0x0/0x10)
[   10.295962] Exception stack(0xc2cf9fa8 to 0xc2cf9ff0)
[   10.301018] 9fa0:                   00000000 01a0ef00 0000000d
bed817b8 00000000 00000000
[   10.309202] 9fc0: 00000000 01a0ef00 bed817b8 00000128 0000005b
01a0af00 01a0f620 00000000
[   10.317381] 9fe0: b6f9fad4 bed81780 b6de4780 b6cd9548
[   10.322442] Code: 1afffff7 e59c0000 e5830000 e3520000 (e584800c)
[   10.328557] Internal error: Oops: 805 [#8] SMP ARM
[   10.328768] ---[ end trace 2cb865e83300a747 ]---
[   10.333357] Modules linked in:
[   10.333374] CPU: 2 PID: 212 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl #1
[   10.333378] Hardware name: BCM2835
[   10.333396] PC is at sk_filter_trim_cap+0x15c/0x1b8
[   10.333409] LR is at   (null)
[   10.341840] Unable to handle kernel NULL pointer dereference at
virtual address 0000000c
[   10.351172] pc : [<c09d4d58>]    lr : [<00000000>]    psr: 60000013
[   10.351179] sp : c2e5dd58  ip : 00000000  fp : 00000000
[   10.351185] r10: ef2c3c00  r9 : c13093c0  r8 : 00000000
[   10.351192] r7 : 00000000  r6 : 00000001  r5 : f0f6a000  r4 : 00000000
[   10.351198] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[   10.351207] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   10.351215] Control: 10c5387d  Table: 02e6006a  DAC: 00000051
[   10.351231] Process systemd-udevd (pid: 212, stack limit = 0x(ptrval))
[   10.354654] pgd = (ptrval)
[   10.359496] Stack: (0xc2e5dd58 to 0xc2e5e000)
[   10.359505] dd40:
    00000000 ef3c0540
[   10.359520] dd60: 006000c0 00000000 00000000 c09a233c ef3c0b40
ef3c0b40 ef3c0540 00000000
[   10.359534] dd80: 00000000 00000000 c13093c0 c09fa2bc 006000c0
00000001 ee7f2000 00000000
[   10.359548] dda0: 00000002 00000000 00000001 ef2c3c64 c2e5df70
00000002 ef3c0540 00000000
[   10.359563] ddc0: c2e5df68 00000084 ee7f2000 00000008 00000000
c09fa3b8 006000c0 00000000
[   10.359585] dde0: 00000000 00000002 00000002 c09fc704 006000c0
00000000 c2c68d00 00000000
[   10.362574] [0000000c] *pgd=00000000
[   10.382706] de00: 000000d4 00000000 00000000 00000000 c126a314
c2e5df68 eec76c40 c2e5de50
[   10.382721] de20: 00000040 00000000 eec76c40 00000000 00000000
c099a624 c2e5df68 00000000
[   10.382735] de40: c2e5de50 c099ae48 00000100 00000000 00000080
c04ab918 ee78e8c0 7fff0000
[   10.382750] de60: c2e5de90 c2e5deec ffff0000 000000a0 bed817e4
00000028 01a040a8 0000005c
[   10.382764] de80: 00000000 00000000 00000000 01a0e0f0 00000128
40000028 b6cd9548 00000000
[   10.382780] dea0: 0000000d 00000000 bed817b8 00000000 00000010
00000000 00000002 00000000
[   10.397129] dec0: 00000000 00000000 01a0e0f0 00000000 c2e5dfb0
00000128 bed817b8 00000000
[   10.397144] dee0: 00000000 c0407f18 00000000 00000000 c120bbec
b6e2ba00 c2e5dfb0 10c5387d
[   10.397159] df00: 01a0e1a8 bed81720 bed81728 c03165fc 00006010
00001000 3e600000 c04ced24
[   10.397174] df20: ef216010 00000ff0 ef216000 00000000 ef217000
eec76c40 bed817b8 00000000
[   10.397189] df40: 00000128 c0301204 c2e5c000 00000128 00000000
c099bc5c 00000000 00000000
[   10.589571] df60: 00000000 fffffff7 c2e5deb0 0000000c 00000001
00000000 00000000 c2e5de80
[   10.589596] df80: 00000000 c030ac08 00000000 00000000 00000040
00000000 00000000 01a0e0f0
[   10.605946] dfa0: bed817b8 c03011d4 00000000 01a0e0f0 0000000d
bed817b8 00000000 00000000
[   10.614131] dfc0: 00000000 01a0e0f0 bed817b8 00000128 0000005c
01a0af00 01a0e920 00000000
[   10.622316] dfe0: b6f9fad4 bed81780 b6de4780 b6cd9548 60000010
0000000d 00000000 00000000
[   10.630594] [<c09d4d58>] (sk_filter_trim_cap) from [<c09fa2bc>]
(netlink_broadcast_filtered+0x304/0x3dc)
[   10.640088] [<c09fa2bc>] (netlink_broadcast_filtered) from
[<c09fa3b8>] (netlink_broadcast+0x24/0x2c)
[   10.650447] [<c09fa3b8>] (netlink_broadcast) from [<c09fc704>]
(netlink_sendmsg+0x30c/0x340)
[   10.658899] [<c09fc704>] (netlink_sendmsg) from [<c099a624>]
(sock_sendmsg+0x3c/0x4c)
[   10.666742] [<c099a624>] (sock_sendmsg) from [<c099ae48>]
(___sys_sendmsg+0x1d8/0x218)
[   10.674673] [<c099ae48>] (___sys_sendmsg) from [<c099bc5c>]
(__sys_sendmsg+0x48/0x6c)
[   10.682515] [<c099bc5c>] (__sys_sendmsg) from [<c03011d4>]
(__sys_trace_return+0x0/0x10)
[   10.690604] Exception stack(0xc2e5dfa8 to 0xc2e5dff0)
[   10.695660] dfa0:                   00000000 01a0e0f0 0000000d
bed817b8 00000000 00000000
[   10.703845] dfc0: 00000000 01a0e0f0 bed817b8 00000128 0000005c
01a0af00 01a0e920 00000000
[   10.712025] dfe0: b6f9fad4 bed81780 b6de4780 b6cd9548
[   10.717086] Code: 1afffff7 e59c0000 e5830000 e3520000 (e584800c)
[   10.723199] Internal error: Oops: 805 [#9] SMP ARM
[   10.723343] ---[ end trace 2cb865e83300a748 ]---

On 06/22/2018 04:19 AM, Peter Robinson wrote:

Hi All,

I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter

Hi Peter

Could you provide symbolic information ?

Thanks !

quoted

I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter

Hi Peter

Could you provide symbolic information ?

I passed in through scripts/decode_stacktrace.sh is that what you were after:

[    8.673880] Internal error: Oops: a06 [#10] SMP ARM
[    8.673949] ---[ end trace 049df4786ea3140a ]---
[    8.678754] Modules linked in:
[    8.678766] CPU: 1 PID: 206 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl+lpae #1
[    8.678769] Hardware name: Allwinner sun8i Family
[    8.678781] PC is at sk_filter_trim_cap ()
[    8.678790] LR is at   (null)
[    8.709463] pc : lr : psr: 60000013 ()
[    8.715722] sp : c996bd60  ip : 00000000  fp : 00000000
[    8.720939] r10: ee79dc00  r9 : c12c9f80  r8 : 00000000
[    8.726157] r7 : 00000000  r6 : 00000001  r5 : f1648000  r4 : 00000000
[    8.732674] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    8.739193] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[    8.746318] Control: 30c5387d  Table: 6e7bc880  DAC: ffe75ece
[    8.752055] Process systemd-udevd (pid: 206, stack limit = 0x(ptrval))
[    8.758574] Stack: (0xc996bd60 to 0xc996c000)
[    8.762929] bd60: 00000000 ee7ad0c0 006000c0 00000000 00000000
c0a64ab8 ee7ad240 ee7ad240
[    8.771098] bd80: ee7ad0c0 00000000 00000000 00000000 c12c9f80
c0abbb8c ef001a00 00000001
[    8.779267] bda0: ee722400 00000000 00000002 00000000 00000001
ee79dc64 c996bf70 00000002
[    8.787435] bdc0: ee7ad0c0 00000000 c996bf68 0000008b ee722400
00000008 00000000 c0abbc88
[    8.795604] bde0: 006000c0 00000000 00000000 00000002 00000002
c0abdfb0 006000c0 00000000
[    8.803772] be00: c98ce580 00000000 000000ce 00000000 00000000
00000000 c124ebf4 c996bf68
[    8.811941] be20: eead4c40 c996be58 00000040 00000000 eead4c40
00000000 00000000 c0a5d198
[    8.820110] be40: c996bf68 00000000 c996be58 c0a5d958 00000000
00000000 ee78c2c0 7fff0000
[    8.828278] be60: c996be90 c996beec ffff0000 000000a0 00000000
c05103ac bef897e4 00000028
[    8.836447] be80: 004ee0a8 00000063 00000000 004f3820 00000128
40000028 b6c9a548 00000000
[    8.844615] bea0: 0000000d 00000000 bef897b8 00000000 00000000
00000000 00000010 00000000
[    8.852784] bec0: 00000002 00000000 004f3820 00000000 c996bfb0
00000128 bef897b8 00000000
[    8.860953] bee0: 00000000 c0510450 00000000 00000000 c120eaa4
b6deca00 c996bfb0 30c5387d
[    8.869122] bf00: 004f38d8 bef89720 bef89728 c0434e94 00000000
c05e0290 ee4e6010 00000ff0
[    8.877291] bf20: ee4e6010 00000ff0 ee4e6000 00000000 00000000
c0506354 eead4c40 bef897b8
[    8.885460] bf40: 00000000 00000128 c0401324 c996a000 00000128
c0a5e6d4 00000000 00000000
[    8.893628] bf60: 00000000 fffffff7 c996beb8 0000000c 00000001
00000000 00000000 c996be88
[    8.901796] bf80: 00000000 c0429ac0 00000000 00000000 00000040
00000000 00000000 004f3820
[    8.909965] bfa0: bef897b8 c04012e8 00000000 004f3820 0000000d
bef897b8 00000000 00000000
[    8.918134] bfc0: 00000000 004f3820 bef897b8 00000128 00000063
004eae70 004f4078 00000000
[    8.926302] bfe0: b6f60ad4 bef89780 b6da5780 b6c9a548 60000010
0000000d 00000000 00000000
[    8.934488] (sk_filter_trim_cap) from netlink_broadcast_filtered ()
[    8.943963] (netlink_broadcast_filtered) from netlink_broadcast ()
[    8.953174] (netlink_broadcast) from netlink_sendmsg ()
[    8.961608] (netlink_sendmsg) from sock_sendmsg ()
[    8.969432] (sock_sendmsg) from ___sys_sendmsg ()
[    8.977343] (___sys_sendmsg) from __sys_sendmsg ()
[    8.985170] (__sys_sendmsg) from __sys_trace_return ()
[    8.993247] Exception stack(0xc996bfa8 to 0xc996bff0)
[    8.998294] bfa0:                   00000000 004f3820 0000000d
bef897b8 00000000 00000000
[    9.006463] bfc0: 00000000 004f3820 bef897b8 00000128 00000063
004eae70 004f4078 00000000
[    9.014629] bfe0: b6f60ad4 bef89780 b6da5780 b6c9a548
[ 9.019680] Code: 1afffff7 e59c0000 e5830000 e3520000 (e584800c)
All code
========
   0:   1afffff7        .word   0x1afffff7
   4:   e59c0000        .word   0xe59c0000
   8:   e5830000        .word   0xe5830000
   c:   e3520000        .word   0xe3520000
  10:*  e584800c        .word   0xe584800c              <-- trapping instruction

Code starting with the faulting instruction
===========================================
   0:   e584800c        .word   0xe584800c
[    9.025823] ---[ end trace 049df4786ea3140b ]---

On 06/24/2018 11:24 AM, Peter Robinson wrote:

quoted

quoted

I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter

Hi Peter

Could you provide symbolic information ?

I passed in through scripts/decode_stacktrace.sh is that what you were after:

[    8.673880] Internal error: Oops: a06 [#10] SMP ARM
[    8.673949] ---[ end trace 049df4786ea3140a ]---
[    8.678754] Modules linked in:
[    8.678766] CPU: 1 PID: 206 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl+lpae #1
[    8.678769] Hardware name: Allwinner sun8i Family
[    8.678781] PC is at sk_filter_trim_cap ()
[    8.678790] LR is at   (null)
[    8.709463] pc : lr : psr: 60000013 ()
[    8.715722] sp : c996bd60  ip : 00000000  fp : 00000000
[    8.720939] r10: ee79dc00  r9 : c12c9f80  r8 : 00000000
[    8.726157] r7 : 00000000  r6 : 00000001  r5 : f1648000  r4 : 00000000
[    8.732674] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    8.739193] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[    8.746318] Control: 30c5387d  Table: 6e7bc880  DAC: ffe75ece
[    8.752055] Process systemd-udevd (pid: 206, stack limit = 0x(ptrval))
[    8.758574] Stack: (0xc996bd60 to 0xc996c000)

[...]

Should be fixed by (PR to Linus with fix is pending):

https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=9262478220eac908ae6e168c3df2c453c87e2da3

Sorry for that,
Daniel

On Mon, Jun 25, 2018 at 9:48 AM, Daniel Borkmann [off-list ref] wrote:

On 06/24/2018 11:24 AM, Peter Robinson wrote:

quoted

quoted

quoted

I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter

Hi Peter

Could you provide symbolic information ?

I passed in through scripts/decode_stacktrace.sh is that what you were after:

[    8.673880] Internal error: Oops: a06 [#10] SMP ARM
[    8.673949] ---[ end trace 049df4786ea3140a ]---
[    8.678754] Modules linked in:
[    8.678766] CPU: 1 PID: 206 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl+lpae #1
[    8.678769] Hardware name: Allwinner sun8i Family
[    8.678781] PC is at sk_filter_trim_cap ()
[    8.678790] LR is at   (null)
[    8.709463] pc : lr : psr: 60000013 ()
[    8.715722] sp : c996bd60  ip : 00000000  fp : 00000000
[    8.720939] r10: ee79dc00  r9 : c12c9f80  r8 : 00000000
[    8.726157] r7 : 00000000  r6 : 00000001  r5 : f1648000  r4 : 00000000
[    8.732674] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    8.739193] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[    8.746318] Control: 30c5387d  Table: 6e7bc880  DAC: ffe75ece
[    8.752055] Process systemd-udevd (pid: 206, stack limit = 0x(ptrval))
[    8.758574] Stack: (0xc996bd60 to 0xc996c000)

[...]

Should be fixed by (PR to Linus with fix is pending):

https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=9262478220eac908ae6e168c3df2c453c87e2da3

Unfortunately it's not, building against the git checkout of the first
failed Fedora kernel (see rc2 issue below) it has the same effect :-(

I thought it might have been [1] because it touches bits of that code
but if I trying with rc2 and that reverted I got no output at all,
checking the vanilla Fedora build from friday (so almost rc2) it
doesn't boot at all either so I've got a second thing to investigate.

Peter

[1] https://lkml.org/lkml/2018/4/29/30