[offlist] Re: Crash in netlink/sk_filter_trim_cap on ARMv7 on 4.18rc1

quoted

On Mon, Jun 25, 2018 at 2:39 PM, Peter Robinson [off-list ref] wrote:
Hi Daniel,

On 06/24/2018 11:24 AM, Peter Robinson wrote:
I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.
So this is arm32 right?
Correct.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter
Hi Peter

Could you provide symbolic information ?
I passed in through scripts/decode_stacktrace.sh is that what you were after:

[    8.673880] Internal error: Oops: a06 [#10] SMP ARM
[    8.673949] ---[ end trace 049df4786ea3140a ]---
[    8.678754] Modules linked in:
[    8.678766] CPU: 1 PID: 206 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl+lpae #1
[    8.678769] Hardware name: Allwinner sun8i Family
[    8.678781] PC is at sk_filter_trim_cap ()
[    8.678790] LR is at   (null)
[    8.709463] pc : lr : psr: 60000013 ()
[    8.715722] sp : c996bd60  ip : 00000000  fp : 00000000
[    8.720939] r10: ee79dc00  r9 : c12c9f80  r8 : 00000000
[    8.726157] r7 : 00000000  r6 : 00000001  r5 : f1648000  r4 : 00000000
[    8.732674] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    8.739193] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[    8.746318] Control: 30c5387d  Table: 6e7bc880  DAC: ffe75ece
[    8.752055] Process systemd-udevd (pid: 206, stack limit = 0x(ptrval))
[    8.758574] Stack: (0xc996bd60 to 0xc996c000)
Do you have BPF JIT enabled or disabled? Does it happen with disabled?
Enabled, I can test with it disabled, BPF configs bits are:
CONFIG_BPF_EVENTS=y
# CONFIG_BPFILTER is not set
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF=y
CONFIG_CGROUP_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_LWTUNNEL_BPF=y
# CONFIG_NBPFAXI_DMA is not set
CONFIG_NET_ACT_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_TEST_BPF is not set

I can see one bug, but your stack trace seems unrelated.

Anyway, could you try with this?
Build in process.

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index 6e8b716..f6a62ae 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -1844,7 +1844,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
                /* there are 2 passes here */
                bpf_jit_dump(prog->len, image_size, 2, ctx.target);

-       set_memory_ro((unsigned long)header, header->pages);
+       bpf_jit_binary_lock_ro(header);
        prog->bpf_func = (void *)ctx.target;
        prog->jited = 1;
        prog->jited_len = image_size;
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.

Will look at the other patches shortly, there's been some other issue
introduced between rc1 and rc2 which I have to work out before I can
test those though.

Peter
Hi Daniel,

On 06/24/2018 11:24 AM, Peter Robinson wrote:
I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.
So this is arm32 right?
Correct.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter
Hi Peter

Could you provide symbolic information ?
I passed in through scripts/decode_stacktrace.sh is that what you were after:

[    8.673880] Internal error: Oops: a06 [#10] SMP ARM
[    8.673949] ---[ end trace 049df4786ea3140a ]---
[    8.678754] Modules linked in:
[    8.678766] CPU: 1 PID: 206 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl+lpae #1
[    8.678769] Hardware name: Allwinner sun8i Family
[    8.678781] PC is at sk_filter_trim_cap ()
[    8.678790] LR is at   (null)
[    8.709463] pc : lr : psr: 60000013 ()
[    8.715722] sp : c996bd60  ip : 00000000  fp : 00000000
[    8.720939] r10: ee79dc00  r9 : c12c9f80  r8 : 00000000
[    8.726157] r7 : 00000000  r6 : 00000001  r5 : f1648000  r4 : 00000000
[    8.732674] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    8.739193] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[    8.746318] Control: 30c5387d  Table: 6e7bc880  DAC: ffe75ece
[    8.752055] Process systemd-udevd (pid: 206, stack limit = 0x(ptrval))
[    8.758574] Stack: (0xc996bd60 to 0xc996c000)
Do you have BPF JIT enabled or disabled? Does it happen with disabled?
Enabled, I can test with it disabled, BPF configs bits are:
CONFIG_BPF_EVENTS=y
# CONFIG_BPFILTER is not set
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF=y
CONFIG_CGROUP_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_LWTUNNEL_BPF=y
# CONFIG_NBPFAXI_DMA is not set
CONFIG_NET_ACT_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_TEST_BPF is not set

I can see one bug, but your stack trace seems unrelated.

Anyway, could you try with this?
Build in process.

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index 6e8b716..f6a62ae 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -1844,7 +1844,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
                /* there are 2 passes here */
                bpf_jit_dump(prog->len, image_size, 2, ctx.target);

-       set_memory_ro((unsigned long)header, header->pages);
+       bpf_jit_binary_lock_ro(header);
        prog->bpf_func = (void *)ctx.target;
        prog->jited = 1;
        prog->jited_len = image_size;
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.

Will look at the other patches shortly, there's been some other issue
introduced between rc1 and rc2 which I have to work out before I can
test those though.
Quick update, with linus's head as of yesterday, basically rc2 plus
davem's network fixes it works if the JIT is disabled IE:
# CONFIG_BPF_JIT_ALWAYS_ON is not set
# CONFIG_BPF_JIT is not set

If I enable it the boot breaks even worse than the errors above in
that I get no console output at all, even with earlycon, so we've gone
backwards since rc1 somehow.

I'll try the above two reverted unless you have any other suggestions.

Peter
On 06/26/2018 02:23 PM, Peter Robinson wrote:
On 06/24/2018 11:24 AM, Peter Robinson wrote:
I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.
So this is arm32 right?
Correct.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter
Hi Peter

Could you provide symbolic information ?
I passed in through scripts/decode_stacktrace.sh is that what you were after:

[    8.673880] Internal error: Oops: a06 [#10] SMP ARM
[    8.673949] ---[ end trace 049df4786ea3140a ]---
[    8.678754] Modules linked in:
[    8.678766] CPU: 1 PID: 206 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl+lpae #1
[    8.678769] Hardware name: Allwinner sun8i Family
[    8.678781] PC is at sk_filter_trim_cap ()
[    8.678790] LR is at   (null)
[    8.709463] pc : lr : psr: 60000013 ()
[    8.715722] sp : c996bd60  ip : 00000000  fp : 00000000
[    8.720939] r10: ee79dc00  r9 : c12c9f80  r8 : 00000000
[    8.726157] r7 : 00000000  r6 : 00000001  r5 : f1648000  r4 : 00000000
[    8.732674] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    8.739193] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[    8.746318] Control: 30c5387d  Table: 6e7bc880  DAC: ffe75ece
[    8.752055] Process systemd-udevd (pid: 206, stack limit = 0x(ptrval))
[    8.758574] Stack: (0xc996bd60 to 0xc996c000)
Do you have BPF JIT enabled or disabled? Does it happen with disabled?
Enabled, I can test with it disabled, BPF configs bits are:
CONFIG_BPF_EVENTS=y
# CONFIG_BPFILTER is not set
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF=y
CONFIG_CGROUP_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_LWTUNNEL_BPF=y
# CONFIG_NBPFAXI_DMA is not set
CONFIG_NET_ACT_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_TEST_BPF is not set

I can see one bug, but your stack trace seems unrelated.

Anyway, could you try with this?
Build in process.

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index 6e8b716..f6a62ae 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -1844,7 +1844,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
                /* there are 2 passes here */
                bpf_jit_dump(prog->len, image_size, 2, ctx.target);

-       set_memory_ro((unsigned long)header, header->pages);
+       bpf_jit_binary_lock_ro(header);
        prog->bpf_func = (void *)ctx.target;
        prog->jited = 1;
        prog->jited_len = image_size;
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.

Will look at the other patches shortly, there's been some other issue
introduced between rc1 and rc2 which I have to work out before I can
test those though.
Quick update, with linus's head as of yesterday, basically rc2 plus
davem's network fixes it works if the JIT is disabled IE:
# CONFIG_BPF_JIT_ALWAYS_ON is not set
# CONFIG_BPF_JIT is not set

If I enable it the boot breaks even worse than the errors above in
that I get no console output at all, even with earlycon, so we've gone
backwards since rc1 somehow.

I'll try the above two reverted unless you have any other suggestions.
Ok, thanks, lets do that!

I'm still working on fixes meanwhile, should have something by end of day.

Thanks,
Daniel
On Tue, Jun 26, 2018 at 1:52 PM, Daniel Borkmann [off-list ref] wrote:
On 06/26/2018 02:23 PM, Peter Robinson wrote:
On 06/24/2018 11:24 AM, Peter Robinson wrote:
I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.
So this is arm32 right?
Correct.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter
Hi Peter

Could you provide symbolic information ?
I passed in through scripts/decode_stacktrace.sh is that what you were after:

[    8.673880] Internal error: Oops: a06 [#10] SMP ARM
[    8.673949] ---[ end trace 049df4786ea3140a ]---
[    8.678754] Modules linked in:
[    8.678766] CPU: 1 PID: 206 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl+lpae #1
[    8.678769] Hardware name: Allwinner sun8i Family
[    8.678781] PC is at sk_filter_trim_cap ()
[    8.678790] LR is at   (null)
[    8.709463] pc : lr : psr: 60000013 ()
[    8.715722] sp : c996bd60  ip : 00000000  fp : 00000000
[    8.720939] r10: ee79dc00  r9 : c12c9f80  r8 : 00000000
[    8.726157] r7 : 00000000  r6 : 00000001  r5 : f1648000  r4 : 00000000
[    8.732674] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    8.739193] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[    8.746318] Control: 30c5387d  Table: 6e7bc880  DAC: ffe75ece
[    8.752055] Process systemd-udevd (pid: 206, stack limit = 0x(ptrval))
[    8.758574] Stack: (0xc996bd60 to 0xc996c000)
Do you have BPF JIT enabled or disabled? Does it happen with disabled?
Enabled, I can test with it disabled, BPF configs bits are:
CONFIG_BPF_EVENTS=y
# CONFIG_BPFILTER is not set
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF=y
CONFIG_CGROUP_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_LWTUNNEL_BPF=y
# CONFIG_NBPFAXI_DMA is not set
CONFIG_NET_ACT_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_TEST_BPF is not set

I can see one bug, but your stack trace seems unrelated.

Anyway, could you try with this?
Build in process.

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index 6e8b716..f6a62ae 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -1844,7 +1844,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
                /* there are 2 passes here */
                bpf_jit_dump(prog->len, image_size, 2, ctx.target);

-       set_memory_ro((unsigned long)header, header->pages);
+       bpf_jit_binary_lock_ro(header);
        prog->bpf_func = (void *)ctx.target;
        prog->jited = 1;
        prog->jited_len = image_size;
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.

Will look at the other patches shortly, there's been some other issue
introduced between rc1 and rc2 which I have to work out before I can
test those though.
Quick update, with linus's head as of yesterday, basically rc2 plus
davem's network fixes it works if the JIT is disabled IE:
# CONFIG_BPF_JIT_ALWAYS_ON is not set
# CONFIG_BPF_JIT is not set

If I enable it the boot breaks even worse than the errors above in
that I get no console output at all, even with earlycon, so we've gone
backwards since rc1 somehow.

I'll try the above two reverted unless you have any other suggestions.
Ok, thanks, lets do that!

I'm still working on fixes meanwhile, should have something by end of day.
Sorry for the delay on this from my end. I noticed there was some bpf
bits land in the last net fixes pull request landed Monday so I built
a kernel with the JIT reenabled. It seems it's improved in that the
completely dead no output boot has gone but the original problem that
arrived in the merge window still persists:

[   17.564142] note: systemd-udevd[194] exited with preempt_count 1
[   17.592739] Unable to handle kernel NULL pointer dereference at
virtual address 0000000c
[   17.601002] pgd = (ptrval)
[   17.603819] [0000000c] *pgd=00000000
[   17.607487] Internal error: Oops: 805 [#10] SMP ARM
[   17.612396] Modules linked in:
[   17.615484] CPU: 0 PID: 195 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc3.git1.1.bpf1.fc29.armv7hl #1
[   17.626056] Hardware name: Generic AM33XX (Flattened Device Tree)
[   17.632198] PC is at sk_filter_trim_cap+0x218/0x2fc
[   17.637102] LR is at   (null)
[   17.640086] pc : [<c0ab03b4>]    lr : [<00000000>]    psr: 60000013
[   17.646384] sp : cfe1dd48  ip : 00000000  fp : 00000000
[   17.651635] r10: d837e000  r9 : d833be00  r8 : 00000000
[   17.656887] r7 : 00000001  r6 : e003d000  r5 : 00000000  r4 : 00000000
[   17.663447] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[   17.670009] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   17.677180] Control: 10c5387d  Table: 8fe20019  DAC: 00000051
[   17.682956] Process systemd-udevd (pid: 195, stack limit = 0x(ptrval))
[   17.689518] Stack: (0xcfe1dd48 to 0xcfe1e000)
[   17.693901] dd40:                   00000000 00000000 c0ab0234
c1308f38 da610180 da610180
[   17.702123] dd60: 006000c0 00000000 00000000 c0a74524 da610600
da610600 da610180 00000000
[   17.710345] dd80: 00000000 00000000 c1424000 c0addd48 cfc72400
00000001 cfc72400 00000000
[   17.718567] dda0: 00000002 00000000 00000001 d837e064 cfe1de78
00000002 da610180 00000000
[   17.726790] ddc0: cfe1df68 00000085 cfc72400 00000008 00000000
c0adde48 006000c0 00000000
[   17.735012] dde0: 00000000 00000002 00000002 c0ae0c5c 006000c0
00000000 cfd5b580 00000000
[   17.743234] de00: 000000c3 00000000 00000000 00000000 c1379d6c
cfe1df68 cf979900 cfe1de50
[   17.751456] de20: 00000040 00000000 cf979900 00000000 00000000
c0a6ac80 cfe1df68 00000000
[   17.759678] de40: cfe1de50 c0a6b4c4 00000003 00000000 d83f2940
7fff0000 cfe1de88 cfe1dee4
[   17.767899] de60: ffff0000 000000a0 00000000 c044b6c0 beec47e4
00000028 012c6750 0000005d
[   17.776121] de80: 00000000 012d71c8 00000128 40000028 b6c35548
00000000 0000000d 00000000
[   17.784343] dea0: beec47b8 00000000 00000000 00000000 00000010
00000000 00000002 00000002
[   17.792565] dec0: 60000093 c13085ec 00000000 c03bf5f4 00000001
00000080 00000000 c0438d18
[   17.800787] dee0: 00000000 00000000 c1bcd804 00000001 c1bcd7c0
c1bcd804 cfe1df40 c0438d18
[   17.809009] df00: 00000000 60000013 00000001 c03f6064 00000001
00000000 c0438d18 00000000
[   17.817231] df20: cfc73800 cfe1df40 00000001 00000000 beec47b8
cf979900 beec47b8 00000000
[   17.825453] df40: 00000128 c03011c4 cfe1c000 00000128 00000000
c0a6c314 00000000 00000000
[   17.833674] df60: 00000000 fffffff7 cfe1deb0 0000000c 00000001
00000000 00000000 cfe1de80
[   17.841894] df80: 00000000 00000128 00000000 00000000 00000040
00000000 00000000 012d71c8
[   17.850117] dfa0: beec47b8 c03011a0 00000000 012d71c8 0000000d
beec47b8 00000000 00000000
[   17.858339] dfc0: 00000000 012d71c8 beec47b8 00000128 0000005d
012bb998 012d78e8 00000000
[   17.866561] dfe0: b6efbad4 beec4780 b6d40780 b6c35548 60000010
0000000d 00000000 00000000
[   17.874805] [<c0ab03b4>] (sk_filter_trim_cap) from [<c0addd48>]
(netlink_broadcast_filtered+0x2e0/0x3bc)
[   17.884341] [<c0addd48>] (netlink_broadcast_filtered) from
[<c0adde48>] (netlink_broadcast+0x24/0x2c)
[   17.893615] [<c0adde48>] (netlink_broadcast) from [<c0ae0c5c>]
(netlink_sendmsg+0x338/0x370)
[   17.902107] [<c0ae0c5c>] (netlink_sendmsg) from [<c0a6ac80>]
(sock_sendmsg+0x3c/0x4c)
[   17.909986] [<c0a6ac80>] (sock_sendmsg) from [<c0a6b4c4>]
(___sys_sendmsg+0x1e4/0x228)
[   17.917949] [<c0a6b4c4>] (___sys_sendmsg) from [<c0a6c314>]
(__sys_sendmsg+0x48/0x6c)
[   17.925828] [<c0a6c314>] (__sys_sendmsg) from [<c03011a0>]
(__sys_trace_return+0x0/0x10)
[   17.933957] Exception stack(0xcfe1dfa8 to 0xcfe1dff0)
[   17.939037] dfa0:                   00000000 012d71c8 0000000d
beec47b8 00000000 00000000
[   17.947259] dfc0: 00000000 012d71c8 beec47b8 00000128 0000005d
012bb998 012d78e8 00000000
[   17.955478] dfe0: b6efbad4 beec4780 b6d40780 b6c35548
[   17.960563] Code: 1afffff7 e59c0000 e5830000 e3520000 (e584800c)
[   17.966827] ---[ end trace 27a2820a2162a4fd ]---
On 07/04/2018 09:33 AM, Peter Robinson wrote:
On Tue, Jun 26, 2018 at 1:52 PM, Daniel Borkmann [off-list ref] wrote:
On 06/26/2018 02:23 PM, Peter Robinson wrote:
On 06/24/2018 11:24 AM, Peter Robinson wrote:
I'm seeing this netlink/sk_filter_trim_cap crash on ARMv7 across quite
a few ARMv7 platforms on Fedora with 4.18rc1. I've tested RPi2/RPi3
(doesn't happen on aarch64), AllWinner H3, BeagleBone and a few
others, both LPAE/normal kernels.
So this is arm32 right?
Correct.

I'm a bit out of my depth in this part of the kernel but I'm wondering
if it's known, I couldn't find anything that looked obvious on a few
mailing lists.

Peter
Hi Peter

Could you provide symbolic information ?
I passed in through scripts/decode_stacktrace.sh is that what you were after:

[    8.673880] Internal error: Oops: a06 [#10] SMP ARM
[    8.673949] ---[ end trace 049df4786ea3140a ]---
[    8.678754] Modules linked in:
[    8.678766] CPU: 1 PID: 206 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc1.git0.1.fc29.armv7hl+lpae #1
[    8.678769] Hardware name: Allwinner sun8i Family
[    8.678781] PC is at sk_filter_trim_cap ()
[    8.678790] LR is at   (null)
[    8.709463] pc : lr : psr: 60000013 ()
[    8.715722] sp : c996bd60  ip : 00000000  fp : 00000000
[    8.720939] r10: ee79dc00  r9 : c12c9f80  r8 : 00000000
[    8.726157] r7 : 00000000  r6 : 00000001  r5 : f1648000  r4 : 00000000
[    8.732674] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    8.739193] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[    8.746318] Control: 30c5387d  Table: 6e7bc880  DAC: ffe75ece
[    8.752055] Process systemd-udevd (pid: 206, stack limit = 0x(ptrval))
[    8.758574] Stack: (0xc996bd60 to 0xc996c000)
Do you have BPF JIT enabled or disabled? Does it happen with disabled?
Enabled, I can test with it disabled, BPF configs bits are:
CONFIG_BPF_EVENTS=y
# CONFIG_BPFILTER is not set
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF=y
CONFIG_CGROUP_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_LWTUNNEL_BPF=y
# CONFIG_NBPFAXI_DMA is not set
CONFIG_NET_ACT_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_TEST_BPF is not set

I can see one bug, but your stack trace seems unrelated.

Anyway, could you try with this?
Build in process.

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index 6e8b716..f6a62ae 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -1844,7 +1844,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
                /* there are 2 passes here */
                bpf_jit_dump(prog->len, image_size, 2, ctx.target);

-       set_memory_ro((unsigned long)header, header->pages);
+       bpf_jit_binary_lock_ro(header);
        prog->bpf_func = (void *)ctx.target;
        prog->jited = 1;
        prog->jited_len = image_size;
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.

Will look at the other patches shortly, there's been some other issue
introduced between rc1 and rc2 which I have to work out before I can
test those though.
Quick update, with linus's head as of yesterday, basically rc2 plus
davem's network fixes it works if the JIT is disabled IE:
# CONFIG_BPF_JIT_ALWAYS_ON is not set
# CONFIG_BPF_JIT is not set

If I enable it the boot breaks even worse than the errors above in
that I get no console output at all, even with earlycon, so we've gone
backwards since rc1 somehow.

I'll try the above two reverted unless you have any other suggestions.
Ok, thanks, lets do that!

I'm still working on fixes meanwhile, should have something by end of day.
Sorry for the delay on this from my end. I noticed there was some bpf
bits land in the last net fixes pull request landed Monday so I built
a kernel with the JIT reenabled. It seems it's improved in that the
completely dead no output boot has gone but the original problem that
arrived in the merge window still persists:
Okay, thanks a lot! And on top of that tree could you try with the below
applied to check whether it fixes the issue?

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index f6a62ae..45e6b49 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -234,11 +234,11 @@ static void jit_fill_hole(void *area, unsigned int size)
 #define SCRATCH_SIZE 80

 /* total stack size used in JITed code */
-#define _STACK_SIZE	(ctx->prog->aux->stack_depth + SCRATCH_SIZE)
+#define _STACK_SIZE	(ctx->prog->aux->stack_depth + SCRATCH_SIZE + 4)
 #define STACK_SIZE	ALIGN(_STACK_SIZE, STACK_ALIGNMENT)

 /* Get the offset of eBPF REGISTERs stored on scratch space. */
-#define STACK_VAR(off) (STACK_SIZE - off)
+#define STACK_VAR(off) (STACK_SIZE - 4 - off)

 #if __LINUX_ARM_ARCH__ < 7
Subject says offlist, but this isn't...

On Wed, Jul 04, 2018 at 08:33:20AM +0100, Peter Robinson wrote:
Sorry for the delay on this from my end. I noticed there was some bpf
bits land in the last net fixes pull request landed Monday so I built
a kernel with the JIT reenabled. It seems it's improved in that the
completely dead no output boot has gone but the original problem that
arrived in the merge window still persists:

[   17.564142] note: systemd-udevd[194] exited with preempt_count 1
[   17.592739] Unable to handle kernel NULL pointer dereference at
virtual address 0000000c
[   17.601002] pgd = (ptrval)
[   17.603819] [0000000c] *pgd=00000000
[   17.607487] Internal error: Oops: 805 [#10] SMP ARM
[   17.612396] Modules linked in:
[   17.615484] CPU: 0 PID: 195 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc3.git1.1.bpf1.fc29.armv7hl #1
[   17.626056] Hardware name: Generic AM33XX (Flattened Device Tree)
[   17.632198] PC is at sk_filter_trim_cap+0x218/0x2fc
[   17.637102] LR is at   (null)
[   17.640086] pc : [<c0ab03b4>]    lr : [<00000000>]    psr: 60000013
[   17.646384] sp : cfe1dd48  ip : 00000000  fp : 00000000
[   17.651635] r10: d837e000  r9 : d833be00  r8 : 00000000
[   17.656887] r7 : 00000001  r6 : e003d000  r5 : 00000000  r4 : 00000000
[   17.663447] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[   17.670009] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   17.677180] Control: 10c5387d  Table: 8fe20019  DAC: 00000051
[   17.682956] Process systemd-udevd (pid: 195, stack limit = 0x(ptrval))
[   17.689518] Stack: (0xcfe1dd48 to 0xcfe1e000)
Can you provide a full disassembly of sk_filter_trim_cap from vmlinux
(iow, annotated with its linked address) for the above dump please -
alternatively a new dump with matching disassembly.  Thanks.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 13.8Mbps down 630kbps up
According to speedtest.net: 13Mbps down 490kbps up
On Thu, Jul 05, 2018 at 12:41:54AM +0100, Russell King - ARM Linux wrote:
Subject says offlist, but this isn't...

On Wed, Jul 04, 2018 at 08:33:20AM +0100, Peter Robinson wrote:
Sorry for the delay on this from my end. I noticed there was some bpf
bits land in the last net fixes pull request landed Monday so I built
a kernel with the JIT reenabled. It seems it's improved in that the
completely dead no output boot has gone but the original problem that
arrived in the merge window still persists:

[   17.564142] note: systemd-udevd[194] exited with preempt_count 1
[   17.592739] Unable to handle kernel NULL pointer dereference at
virtual address 0000000c
[   17.601002] pgd = (ptrval)
[   17.603819] [0000000c] *pgd=00000000
[   17.607487] Internal error: Oops: 805 [#10] SMP ARM
[   17.612396] Modules linked in:
[   17.615484] CPU: 0 PID: 195 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc3.git1.1.bpf1.fc29.armv7hl #1
[   17.626056] Hardware name: Generic AM33XX (Flattened Device Tree)
[   17.632198] PC is at sk_filter_trim_cap+0x218/0x2fc
[   17.637102] LR is at   (null)
[   17.640086] pc : [<c0ab03b4>]    lr : [<00000000>]    psr: 60000013
[   17.646384] sp : cfe1dd48  ip : 00000000  fp : 00000000
[   17.651635] r10: d837e000  r9 : d833be00  r8 : 00000000
[   17.656887] r7 : 00000001  r6 : e003d000  r5 : 00000000  r4 : 00000000
[   17.663447] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[   17.670009] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   17.677180] Control: 10c5387d  Table: 8fe20019  DAC: 00000051
[   17.682956] Process systemd-udevd (pid: 195, stack limit = 0x(ptrval))
[   17.689518] Stack: (0xcfe1dd48 to 0xcfe1e000)
Can you provide a full disassembly of sk_filter_trim_cap from vmlinux
(iow, annotated with its linked address) for the above dump please -
alternatively a new dump with matching disassembly.  Thanks.
Also probably a good idea to have bpf_jit_enable set to 2 to get a
dump of the bpf program being run, which I think for your problem,
you'll have to hack the kernel source to do that.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 13.8Mbps down 630kbps up
According to speedtest.net: 13Mbps down 490kbps up
On 07/05/2018 09:31 AM, Russell King - ARM Linux wrote:
On Thu, Jul 05, 2018 at 12:41:54AM +0100, Russell King - ARM Linux wrote:
Subject says offlist, but this isn't...

On Wed, Jul 04, 2018 at 08:33:20AM +0100, Peter Robinson wrote:
Sorry for the delay on this from my end. I noticed there was some bpf
bits land in the last net fixes pull request landed Monday so I built
a kernel with the JIT reenabled. It seems it's improved in that the
completely dead no output boot has gone but the original problem that
arrived in the merge window still persists:

[   17.564142] note: systemd-udevd[194] exited with preempt_count 1
[   17.592739] Unable to handle kernel NULL pointer dereference at
virtual address 0000000c
[   17.601002] pgd = (ptrval)
[   17.603819] [0000000c] *pgd=00000000
[   17.607487] Internal error: Oops: 805 [#10] SMP ARM
[   17.612396] Modules linked in:
[   17.615484] CPU: 0 PID: 195 Comm: systemd-udevd Tainted: G      D
        4.18.0-0.rc3.git1.1.bpf1.fc29.armv7hl #1
[   17.626056] Hardware name: Generic AM33XX (Flattened Device Tree)
[   17.632198] PC is at sk_filter_trim_cap+0x218/0x2fc
[   17.637102] LR is at   (null)
[   17.640086] pc : [<c0ab03b4>]    lr : [<00000000>]    psr: 60000013
[   17.646384] sp : cfe1dd48  ip : 00000000  fp : 00000000
[   17.651635] r10: d837e000  r9 : d833be00  r8 : 00000000
[   17.656887] r7 : 00000001  r6 : e003d000  r5 : 00000000  r4 : 00000000
[   17.663447] r3 : 00000007  r2 : 00000000  r1 : 00000000  r0 : 00000000
[   17.670009] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   17.677180] Control: 10c5387d  Table: 8fe20019  DAC: 00000051
[   17.682956] Process systemd-udevd (pid: 195, stack limit = 0x(ptrval))
[   17.689518] Stack: (0xcfe1dd48 to 0xcfe1e000)
Can you provide a full disassembly of sk_filter_trim_cap from vmlinux
(iow, annotated with its linked address) for the above dump please -
alternatively a new dump with matching disassembly.  Thanks.
Also probably a good idea to have bpf_jit_enable set to 2 to get a
dump of the bpf program being run, which I think for your problem,
you'll have to hack the kernel source to do that.
Agree, that would be good as well. You could use something like the below
to bail out to interpreter after JIT did the dump.

Dump will then land in kernel log which you could paste here.

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index f6a62ae..d6a7dfd 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -1844,6 +1844,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		/* there are 2 passes here */
 		bpf_jit_dump(prog->len, image_size, 2, ctx.target);

+	/* Defer to interpreter after dump. */
+	if (1) {
+		bpf_jit_binary_free(header);
+		prog = orig_prog;
+		goto out_imms;
+	}
+
 	bpf_jit_binary_lock_ro(header);
 	prog->bpf_func = (void *)ctx.target;
 	prog->jited = 1;
On Mon, Jun 25, 2018 at 05:41:27PM +0100, Peter Robinson wrote:
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.
I can confirm the crash with the released 4.18.1 on Banana Pi, and I can
also confirm that disabling BPF JIT makes the Banana Pi work again.,

Greetings
Marc

[    0.004930] /cpus/cpu@0 missing clock-frequency property
[    0.004965] /cpus/cpu@1 missing clock-frequency property
[    4.959858] zswap: default zpool zbud not available
[    4.964820] zswap: pool creation failed
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
[   10.721077] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
[   10.722949] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
[   10.729288] pgd = (ptrval)
[   10.729299] [0000000c] *pgd=6dc65003, *pmd=00000000
[   10.737464] pgd = (ptrval)
[   10.740176] Internal error: Oops: a06 [#1] SMP ARM
[   10.745056] [0000000c] *pgd=6e72a003
[   10.747742] Modules linked in: ip_tables x_tables autofs4 btrfs
[   10.752561] , *pmd=00000000
[   10.756113]  libcrc32c crc32c_generic xor zstd_decompress zstd_compress xxhash
[   10.764833]  zlib_deflate raid6_pq dm_mod dax axp20x_regulator realtek ahci_sunxi dwmac_sunxi stmmac_platform libahci_platform stmmac i2c_mv64xxx libahci libata scsi_mod ohci_platform ohci_hcd ehci_platform ehci_hcd phy_sun4i_usb sunxi_mmc
[   10.793306] CPU: 1 PID: 238 Comm: systemd-udevd Not tainted 4.18.1-zgbpi-armmp-lpae #3
[   10.801212] Hardware name: Allwinner sun7i (A20) Family
[   10.806448] PC is at sk_filter_trim_cap+0xa0/0x1d4
[   10.811238] LR is at   (null)
[   10.814205] pc : [<c06de388>]    lr : [<00000000>]    psr: 600f0013
[   10.820466] sp : edc7dcf8  ip : 00000000  fp : edc7dd34
[   10.825686] r10: 00000000  r9 : 00000000  r8 : 00000000
[   10.830907] r7 : 00000001  r6 : f0e96000  r5 : c0e04cc8  r4 : 00000000
[   10.837428] r3 : 00000007  r2 : fb5e2d70  r1 : 00000000  r0 : 00000000
[   10.843952] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   10.851081] Control: 30c5387d  Table: 6e6c7580  DAC: 2c983336
[   10.856822] Process systemd-udevd (pid: 238, stack limit = 0x(ptrval))
[   10.863344] Stack: (0xedc7dcf8 to 0xedc7e000)
[   10.867700] dce0:                                                       edc7dd1c edc7dd08
[   10.875873] dd00: c06a41dc c06a4048 ee7d39c0 fb5e2d70 ee479800 ee6c2400 edc33840 c0e6aac0
[   10.884046] dd20: 00000000 00000001 edc7dd8c edc7dd38 c0705884 c06de2f4 edc7de24 00000001
[   10.892219] dd40: c0ec649c ee479864 00000000 00000000 ee7d39c0 00000000 00000000 00000002
[   10.900391] dd60: 00000000 edc7df44 c0e04cc8 ee7d39c0 ee6c2400 00000000 0000008c 00000002
[   10.908565] dd80: edc7ddf4 edc7dd90 c0705ee0 c0705610 006000c0 00000000 00000000 fb5e2d70
[   10.916737] dda0: 00000008 00000000 00000000 ef357c80 00000000 000000ee 00000000 00000000
[   10.924910] ddc0: 00000000 fb5e2d70 0000008c edc7df44 eef08700 00000040 00000000 eef08700
[   10.933083] dde0: 00000000 edc7dedc edc7de0c edc7ddf8 c069b948 c0705b78 edc7df44 c0e04cc8
[   10.941256] de00: edc7df2c edc7de10 c069c2f8 c069b910 c0e04cc8 edc7dec0 00000000 be8dcfac
[   10.949428] de20: 00000028 0186a660 00000064 bf387954 edc7df48 be8dcf80 00000000 00000000
[   10.957602] de40: be8dcf80 b6f19ce8 00000128 40000028 b6e01346 00000000 0000000e 00000010
[   10.965774] de60: 00000000 00000002 00000000 00000000 00000000 00000000 be8dcf80 00000000
[   10.973948] de80: b6f19ce8 00000000 00000000 fb5e2d70 edc7deb4 ffffe000 00000000 c0e04cc8
[   10.982120] dea0: 00000128 c0201204 00000000 00000080 edc7df6c edc7dec0 c02f5e2c c02f5c18
[   10.990293] dec0: 00000000 fb5e2d70 edc7def4 a0010013 c9f1e000 c03f986c edc7df50 00000000
[   10.998466] dee0: 0000000e 00004000 edc7df3c fb5e2d70 c0409c98 c0409d34 edc7df14 fb5e2d70
[   11.006639] df00: c0409d34 c0e04cc8 be8dcf80 00000000 eef08700 c0201204 edc7c000 00000128
[   11.014812] df20: edc7df94 edc7df30 c069d818 c069c0a0 00000000 00000000 c0e04cc8 00000000
[   11.022984] df40: fffffff7 edc7de5c 0000000c 00000001 00000000 00000000 edc7de2c 00000000
[   11.031156] df60: edc7df7c 00000000 00000000 00000040 00000000 fb5e2d70 be8dcf80 b6f19ce8
[   11.039329] df80: 01878670 00000128 edc7dfa4 edc7df98 c069d870 c069d7c4 00000000 edc7dfa8
[   11.047502] dfa0: c02011cc c069d860 be8dcf80 b6f19ce8 0000000e be8dcf80 00000000 00000000
[   11.055675] dfc0: be8dcf80 b6f19ce8 01878670 00000128 00000000 00000064 01878e80 00000000
[   11.063848] dfe0: 00000128 be8dcf50 b6e003e3 b6e01346 200f0030 0000000e 00000000 00000000
[   11.072038] [<c06de388>] (sk_filter_trim_cap) from [<c0705884>] (netlink_broadcast_filtered+0x280/0x460)
[   11.081517] [<c0705884>] (netlink_broadcast_filtered) from [<c0705ee0>] (netlink_sendmsg+0x374/0x3b0)
[   11.090734] [<c0705ee0>] (netlink_sendmsg) from [<c069b948>] (sock_sendmsg+0x44/0x54)
[   11.098567] [<c069b948>] (sock_sendmsg) from [<c069c2f8>] (___sys_sendmsg+0x264/0x278)
[   11.106485] [<c069c2f8>] (___sys_sendmsg) from [<c069d818>] (__sys_sendmsg+0x60/0x9c)
[   11.114315] [<c069d818>] (__sys_sendmsg) from [<c069d870>] (sys_sendmsg+0x1c/0x20)
[   11.121886] [<c069d870>] (sys_sendmsg) from [<c02011cc>] (__sys_trace_return+0x0/0x10)
[   11.129793] Exception stack(0xedc7dfa8 to 0xedc7dff0)
[   11.134845] dfa0:                   be8dcf80 b6f19ce8 0000000e be8dcf80 00000000 00000000
[   11.143019] dfc0: be8dcf80 b6f19ce8 01878670 00000128 00000000 00000064 01878e80 00000000
[   11.151188] dfe0: 00000128 be8dcf50 b6e003e3 b6e01346
[   11.156243] Code: e3130010 e1a0c000 1a000030 e35c0000 (e584900c) 
[   11.162340] Internal error: Oops: a06 [#2] SMP ARM
[   11.162559] ---[ end trace 1b60255ae59ac006 ]---
[   11.167129] Modules linked in: ip_tables x_tables autofs4 btrfs libcrc32c crc32c_generic xor zstd_decompress zstd_compress xxhash zlib_deflate raid6_pq dm_mod dax axp20x_regulator realtek ahci_sunxi dwmac_sunxi stmmac_platform libahci_platform stmmac i2c_mv64xxx libahci libata scsi_mod ohci_platform ohci_hcd ehci_platform ehci_hcd phy_sun4i_usb sunxi_mmc
[   11.185005] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
[   11.203186] CPU: 0 PID: 237 Comm: systemd-udevd Tainted: G      D           4.18.1-zgbpi-armmp-lpae #3
[   11.203191] Hardware name: Allwinner sun7i (A20) Family
[   11.203216] PC is at sk_filter_trim_cap+0xa0/0x1d4
[   11.203223] LR is at   (null)
[   11.203229] pc : [<c06de388>]    lr : [<00000000>]    psr: 600f0013
[   11.203234] sp : edc41cf8  ip : 00000000  fp : edc41d34
[   11.203239] r10: 00000000  r9 : 00000000  r8 : 00000000
[   11.203245] r7 : 00000001  r6 : f0e96000  r5 : c0e04cc8  r4 : 00000000
[   11.203250] r3 : 00000007  r2 : fb5e2d70  r1 : 00000000  r0 : 00000000
[   11.203258] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   11.203264] Control: 30c5387d  Table: 6e6c84c0  DAC: fffffffd
[   11.203270] Process systemd-udevd (pid: 237, stack limit = 0x(ptrval))
[   11.203276] Stack: (0xedc41cf8 to 0xedc42000)
[   11.203288] 1ce0:                                                       edc41d1c edc41d08
[   11.211398] pgd = (ptrval)
[   11.220660] 1d00: c06a41dc c06a4048 c9c16cc0 fb5e2d70 ee479800 ee6c6400 c9c16240 c0e6aac0
[   11.220670] 1d20: 00000000 00000001 edc41d8c edc41d38 c0705884 c06de2f4 edc41e24 00000001
[   11.220680] 1d40: c0ec649c ee479864 00000000 00000000 c9c16cc0 00000000 00000000 00000002
[   11.220693] 1d60: 00000000 edc41f44 c0e04cc8 c9c16cc0 ee6c6400 00000000 00000085 00000002
[   11.226034] [0000000c] *pgd=6dc79003
[   11.230697] 1d80: edc41df4 edc41d90 c0705ee0 c0705610 006000c0 00000000 00000000 fb5e2d70
[   11.230707] 1da0: 00000008 00000000 00000000 ef357300 00000000 000000ed 00000000 00000000
[   11.230717] 1dc0: 00000000 fb5e2d70 00000085 edc41f44 ee0591c0 00000040 00000000 ee0591c0
[   11.230730] 1de0: 00000000 edc41edc edc41e0c edc41df8 c069b948 c0705b78 edc41f44 c0e04cc8
[   11.233692] , *pmd=00000000
[   11.239953] 1e00: edc41f2c edc41e10 c069c2f8 c069b910 c0e04cc8 edc41ec0 00000000 be8dcfac
[   11.239963] 1e20: 00000028 0186a660 0000005d bf387954 edc41f48 be8dcf80 00000000 00000000
[   11.239973] 1e40: be8dcf80 b6f19ce8 00000128 40000028 b6e01346 00000000 0000000d 00000010
[   11.239982] 1e60: 00000000 00000002 00000000 00000000 00000000 00000000 be8dcf80 00000000
[   11.239992] 1e80: b6f19ce8 00000000 00000000 fb5e2d70 edc41eb4 ffffe000 00000000 c0e04cc8
[   11.240002] 1ea0: 00000128 c0201204 00000000 00000080 edc41f6c edc41ec0 c02f5e2c c02f5c18
[   11.250433] 1ec0: 00000000 fb5e2d70 edc41ef4 a0010013 c9def000 c03f986c edc41f50 00000000
[   11.250443] 1ee0: 0000000d 00004000 edc41f3c fb5e2d70 c0409c98 c0409d34 edc41f14 fb5e2d70
[   11.250454] 1f00: c0409d34 c0e04cc8 be8dcf80 00000000 ee0591c0 c0201204 edc40000 00000128
[   11.250463] 1f20: edc41f94 edc41f30 c069d818 c069c0a0 00000000 00000000 c0e04cc8 00000000
[   11.451342] 1f40: fffffff7 edc41e5c 0000000c 00000001 00000000 00000000 edc41e2c 00000000
[   11.459515] 1f60: edc41f7c 00000000 00000000 00000040 00000000 fb5e2d70 be8dcf80 b6f19ce8
[   11.467688] 1f80: 0186d740 00000128 edc41fa4 edc41f98 c069d870 c069d7c4 00000000 edc41fa8
[   11.475861] 1fa0: c02011cc c069d860 be8dcf80 b6f19ce8 0000000d be8dcf80 00000000 00000000
[   11.484034] 1fc0: be8dcf80 b6f19ce8 0186d740 00000128 00000000 0000005d 018776c0 00000000
[   11.492207] 1fe0: 00000128 be8dcf50 b6e003e3 b6e01346 200f0030 0000000d 00000000 00000000
[   11.500397] [<c06de388>] (sk_filter_trim_cap) from [<c0705884>] (netlink_broadcast_filtered+0x280/0x460)
[   11.509876] [<c0705884>] (netlink_broadcast_filtered) from [<c0705ee0>] (netlink_sendmsg+0x374/0x3b0)
[   11.519093] [<c0705ee0>] (netlink_sendmsg) from [<c069b948>] (sock_sendmsg+0x44/0x54)
[   11.526925] [<c069b948>] (sock_sendmsg) from [<c069c2f8>] (___sys_sendmsg+0x264/0x278)
[   11.534842] [<c069c2f8>] (___sys_sendmsg) from [<c069d818>] (__sys_sendmsg+0x60/0x9c)
[   11.542673] [<c069d818>] (__sys_sendmsg) from [<c069d870>] (sys_sendmsg+0x1c/0x20)
[   11.550244] [<c069d870>] (sys_sendmsg) from [<c02011cc>] (__sys_trace_return+0x0/0x10)
[   11.558151] Exception stack(0xedc41fa8 to 0xedc41ff0)
[   11.563202] 1fa0:                   be8dcf80 b6f19ce8 0000000d be8dcf80 00000000 00000000
[   11.571375] 1fc0: be8dcf80 b6f19ce8 0186d740 00000128 00000000 0000005d 018776c0 00000000
[   11.579544] 1fe0: 00000128 be8dcf50 b6e003e3 b6e01346
[   11.584600] Code: e3130010 e1a0c000 1a000030 e35c0000 (e584900c) 
[   11.590702] Internal error: Oops: a06 [#3] SMP ARM
[   11.590859] ---[ end trace 1b60255ae59ac007 ]---
[   11.595493] Modules linked in: ip_tables x_tables autofs4 btrfs libcrc32c crc32c_generic xor zstd_decompress zstd_compress xxhash zlib_deflate raid6_pq dm_mod dax axp20x_regulator realtek ahci_sunxi dwmac_sunxi stmmac_platform libahci_platform stmmac i2c_mv64xxx libahci libata scsi_mod ohci_platform ohci_hcd ehci_platform ehci_hcd phy_sun4i_usb sunxi_mmc
[   11.602116] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
[   11.631550] CPU: 1 PID: 240 Comm: systemd-udevd Tainted: G      D           4.18.1-zgbpi-armmp-lpae #3
[   11.631555] Hardware name: Allwinner sun7i (A20) Family
[   11.631576] PC is at sk_filter_trim_cap+0xa0/0x1d4
[   11.631582] LR is at   (null)
[   11.631593] pc : [<c06de388>]    lr : [<00000000>]    psr: 600f0013
[   11.639693] pgd = (ptrval)
[   11.648959] sp : edc81cf8  ip : 00000000  fp : edc81d34
[   11.648964] r10: 00000000  r9 : 00000000  r8 : 00000000
[   11.648970] r7 : 00000001  r6 : f0e96000  r5 : c0e04cc8  r4 : 00000000
[   11.648976] r3 : 00000007  r2 : fb5e2d70  r1 : 00000000  r0 : 00000000
[   11.648983] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   11.648990] Control: 30c5387d  Table: 6e71a180  DAC: 2c983336
[   11.654224] [0000000c] *pgd=6dc6e003
[   11.658989] Process systemd-udevd (pid: 240, stack limit = 0x(ptrval))
[   11.658995] Stack: (0xedc81cf8 to 0xedc82000)
[   11.659002] 1ce0:                                                       edc81d1c edc81d08
[   11.659013] 1d00: c06a41dc c06a4048 ee7d36c0 fb5e2d70 ee479800 edc77800 ee7d3d80 c0e6aac0
[   11.661987] , *pmd=00000000
[   11.668231] 1d20: 00000000 00000001 edc81d8c edc81d38 c0705884 c06de2f4 edc81e24 00000001
[   11.668241] 1d40: c0ec649c ee479864 00000000 00000000 ee7d36c0 00000000 00000000 00000002
[   11.668251] 1d60: 00000000 edc81f44 c0e04cc8 ee7d36c0 edc77800 00000000 0000008a 00000002
[   11.676168] 1d80: edc81df4 edc81d90 c0705ee0 c0705610 006000c0 00000000 00000000 fb5e2d70
[   11.676178] 1da0: 00000008 00000000 00000000 ef0ea980 00000000 000000f0 00000000 00000000
[   11.676188] 1dc0: 00000000 fb5e2d70 0000008a edc81f44 ee059a80 00000040 00000000 ee059a80
[   11.789803] 1de0: 00000000 edc81edc edc81e0c edc81df8 c069b948 c0705b78 edc81f44 c0e04cc8
[   11.797977] 1e00: edc81f2c edc81e10 c069c2f8 c069b910 c0e04cc8 edc81ec0 00000000 be8dcfac
[   11.806149] 1e20: 00000028 0186ade8 00000062 bf387954 edc81f48 be8dcf80 00000000 00000000
[   11.814322] 1e40: be8dcf80 b6f19ce8 00000128 40000028 b6e01346 00000000 0000000e 00000010
[   11.822494] 1e60: 00000000 00000002 00000000 00000000 00000000 00000000 be8dcf80 00000000
[   11.830667] 1e80: b6f19ce8 00000000 00000000 fb5e2d70 edc81eb4 ffffe000 00000000 c0e04cc8
[   11.838840] 1ea0: 00000128 c0201204 00000000 00000080 edc81f6c edc81ec0 c02f5e2c c02f5c18
[   11.847013] 1ec0: 00000000 fb5e2d70 edc81ef4 a00b0013 ef3c3000 c03f986c edc81f50 00000000
[   11.855186] 1ee0: 0000000e 00004000 edc81f3c fb5e2d70 c0409c98 c0409d34 edc81f14 fb5e2d70
[   11.863359] 1f00: c0409d34 c0e04cc8 be8dcf80 00000000 ee059a80 c0201204 edc80000 00000128
[   11.871532] 1f20: edc81f94 edc81f30 c069d818 c069c0a0 00000000 00000000 c0e04cc8 00000000
[   11.879705] 1f40: fffffff7 edc81e5c 0000000c 00000001 00000000 00000000 edc81e2c 00000000
[   11.887877] 1f60: edc81f7c 00000000 00000000 00000040 00000000 fb5e2d70 be8dcf80 b6f19ce8
[   11.896051] 1f80: 0186aea0 00000128 edc81fa4 edc81f98 c069d870 c069d7c4 00000000 edc81fa8
[   11.904223] 1fa0: c02011cc c069d860 be8dcf80 b6f19ce8 0000000e be8dcf80 00000000 00000000
[   11.912397] 1fc0: be8dcf80 b6f19ce8 0186aea0 00000128 00000000 00000062 0186b6e8 00000000
[   11.920569] 1fe0: 00000128 be8dcf50 b6e003e3 b6e01346 200f0030 0000000e 00000000 00000000
[   11.928757] [<c06de388>] (sk_filter_trim_cap) from [<c0705884>] (netlink_broadcast_filtered+0x280/0x460)
[   11.938235] [<c0705884>] (netlink_broadcast_filtered) from [<c0705ee0>] (netlink_sendmsg+0x374/0x3b0)
[   11.947452] [<c0705ee0>] (netlink_sendmsg) from [<c069b948>] (sock_sendmsg+0x44/0x54)
[   11.955284] [<c069b948>] (sock_sendmsg) from [<c069c2f8>] (___sys_sendmsg+0x264/0x278)
[   11.963201] [<c069c2f8>] (___sys_sendmsg) from [<c069d818>] (__sys_sendmsg+0x60/0x9c)
[   11.971031] [<c069d818>] (__sys_sendmsg) from [<c069d870>] (sys_sendmsg+0x1c/0x20)
[   11.978602] [<c069d870>] (sys_sendmsg) from [<c02011cc>] (__sys_trace_return+0x0/0x10)
[   11.986509] Exception stack(0xedc81fa8 to 0xedc81ff0)
[   11.991560] 1fa0:                   be8dcf80 b6f19ce8 0000000e be8dcf80 00000000 00000000
[   11.999732] 1fc0: be8dcf80 b6f19ce8 0186aea0 00000128 00000000 00000062 0186b6e8 00000000
[   12.007902] 1fe0: 00000128 be8dcf50 b6e003e3 b6e01346
[   12.012957] Code: e3130010 e1a0c000 1a000030 e35c0000 (e584900c) 
[   12.019056] Internal error: Oops: a06 [#4] SMP ARM
[   12.019171] ---[ end trace 1b60255ae59ac008 ]---


-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
On Thu, Aug 16, 2018 at 10:35:16PM +0200, Marc Haber wrote:
On Mon, Jun 25, 2018 at 05:41:27PM +0100, Peter Robinson wrote:
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.
I can confirm the crash with the released 4.18.1 on Banana Pi, and I can
also confirm that disabling BPF JIT makes the Banana Pi work again.,
Hi,

I'm afraid that the information in the crash dumps is insufficient
to be able to work very much out about these crashes.

We need a recipe (kernel configuration and what userspace is doing)
so that it's possible to recreate the crash, or we need responses
to requests for information - I requested the disassembly of
sk_filter_trim_cap and the BPF code dump via setting a sysctl back
in early July.  Without this, as I say, I don't see how this problem
can be progressed.

If the problem is at boot, one way to set the sysctl would be to
hack the kernel and explicitly initialise the sysctl to '2', or
boot with init=/bin/sh, then manually mount /proc, set the sysctl,
and then "exec /sbin/init" from that shell.  (Remember there's no
job control in that shell, so ^z, ^c, etc do not work.)

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 13.8Mbps down 630kbps up
According to speedtest.net: 13Mbps down 490kbps up
On Thu, Aug 16, 2018 at 11:58 PM, Russell King - ARM Linux
[off-list ref] wrote:
On Thu, Aug 16, 2018 at 10:35:16PM +0200, Marc Haber wrote:
On Mon, Jun 25, 2018 at 05:41:27PM +0100, Peter Robinson wrote:
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.
I can confirm the crash with the released 4.18.1 on Banana Pi, and I can
also confirm that disabling BPF JIT makes the Banana Pi work again.,
Hi,

I'm afraid that the information in the crash dumps is insufficient
to be able to work very much out about these crashes.

We need a recipe (kernel configuration and what userspace is doing)
so that it's possible to recreate the crash, or we need responses
to requests for information - I requested the disassembly of
sk_filter_trim_cap and the BPF code dump via setting a sysctl back
in early July.  Without this, as I say, I don't see how this problem
can be progressed.
I can provide a kernel config [1] but I've not had enough time to sit
down and get the rest of the stuff and debug it due to a combination
of travel and other priorities.

If the problem is at boot, one way to set the sysctl would be to
hack the kernel and explicitly initialise the sysctl to '2', or
boot with init=/bin/sh, then manually mount /proc, set the sysctl,
and then "exec /sbin/init" from that shell.  (Remember there's no
job control in that shell, so ^z, ^c, etc do not work.)
It starts to happen in the early kernel boot long before we get to any
userspace across a number of ARMv7 devices (RPi2/3, BeagleBone and
AllWinner H3 based devices at least).

[1] https://pbrobinson.fedorapeople.org/kernel-armv7hl.config
On 08/17/2018 02:25 PM, Peter Robinson wrote:
On Thu, Aug 16, 2018 at 11:58 PM, Russell King - ARM Linux
[off-list ref] wrote:
On Thu, Aug 16, 2018 at 10:35:16PM +0200, Marc Haber wrote:
On Mon, Jun 25, 2018 at 05:41:27PM +0100, Peter Robinson wrote:
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.
I can confirm the crash with the released 4.18.1 on Banana Pi, and I can
also confirm that disabling BPF JIT makes the Banana Pi work again.,
I'm afraid that the information in the crash dumps is insufficient
to be able to work very much out about these crashes.

We need a recipe (kernel configuration and what userspace is doing)
so that it's possible to recreate the crash, or we need responses
to requests for information - I requested the disassembly of
sk_filter_trim_cap and the BPF code dump via setting a sysctl back
in early July.  Without this, as I say, I don't see how this problem
can be progressed.
I can provide a kernel config [1] but I've not had enough time to sit
down and get the rest of the stuff and debug it due to a combination
of travel and other priorities.
Did you get a chance to try latest kernel from Linus' tree [1] from last
few days to see whether the issue is still persistent? There have been
a number of improvements, bit strange why e.g. Russell didn't run into
it while others have, hmm. Perhaps due to EABI vs non EABI.

[1] git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

If the problem is at boot, one way to set the sysctl would be to
hack the kernel and explicitly initialise the sysctl to '2', or
boot with init=/bin/sh, then manually mount /proc, set the sysctl,
and then "exec /sbin/init" from that shell.  (Remember there's no
job control in that shell, so ^z, ^c, etc do not work.)
It starts to happen in the early kernel boot long before we get to any
userspace across a number of ARMv7 devices (RPi2/3, BeagleBone and
AllWinner H3 based devices at least).

[1] https://pbrobinson.fedorapeople.org/kernel-armv7hl.config
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?

Thanks,
Daniel

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index f6a62ae..c864f6b 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -238,7 +238,7 @@ static void jit_fill_hole(void *area, unsigned int size)
 #define STACK_SIZE	ALIGN(_STACK_SIZE, STACK_ALIGNMENT)

 /* Get the offset of eBPF REGISTERs stored on scratch space. */
-#define STACK_VAR(off) (STACK_SIZE - off)
+#define STACK_VAR(off) (STACK_SIZE - off - 4)

 #if __LINUX_ARM_ARCH__ < 7
On Fri, Aug 17, 2018 at 1:40 PM, Daniel Borkmann [off-list ref] wrote:
On 08/17/2018 02:25 PM, Peter Robinson wrote:
On Thu, Aug 16, 2018 at 11:58 PM, Russell King - ARM Linux
[off-list ref] wrote:
On Thu, Aug 16, 2018 at 10:35:16PM +0200, Marc Haber wrote:
On Mon, Jun 25, 2018 at 05:41:27PM +0100, Peter Robinson wrote:
So with that and the other fix there was no improvement, with those
and the BPF JIT disabled it works, I'm not sure if the two patches
have any effect with the JIT disabled though.
I can confirm the crash with the released 4.18.1 on Banana Pi, and I can
also confirm that disabling BPF JIT makes the Banana Pi work again.,
I'm afraid that the information in the crash dumps is insufficient
to be able to work very much out about these crashes.

We need a recipe (kernel configuration and what userspace is doing)
so that it's possible to recreate the crash, or we need responses
to requests for information - I requested the disassembly of
sk_filter_trim_cap and the BPF code dump via setting a sysctl back
in early July.  Without this, as I say, I don't see how this problem
can be progressed.
I can provide a kernel config [1] but I've not had enough time to sit
down and get the rest of the stuff and debug it due to a combination
of travel and other priorities.
Did you get a chance to try latest kernel from Linus' tree [1] from last
few days to see whether the issue is still persistent? There have been
a number of improvements, bit strange why e.g. Russell didn't run into
it while others have, hmm. Perhaps due to EABI vs non EABI.
I haven't had a chance to try anything from the 4.19 merge window as
yet, I'm traveling this week so it was on the list for next week to
try.

[1] git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

If the problem is at boot, one way to set the sysctl would be to
hack the kernel and explicitly initialise the sysctl to '2', or
boot with init=/bin/sh, then manually mount /proc, set the sysctl,
and then "exec /sbin/init" from that shell.  (Remember there's no
job control in that shell, so ^z, ^c, etc do not work.)
It starts to happen in the early kernel boot long before we get to any
userspace across a number of ARMv7 devices (RPi2/3, BeagleBone and
AllWinner H3 based devices at least).

[1] https://pbrobinson.fedorapeople.org/kernel-armv7hl.config
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?
I will try and get someone to test that today, thanks

Thanks,
Daniel

diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index f6a62ae..c864f6b 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -238,7 +238,7 @@ static void jit_fill_hole(void *area, unsigned int size)
 #define STACK_SIZE     ALIGN(_STACK_SIZE, STACK_ALIGNMENT)

 /* Get the offset of eBPF REGISTERs stored on scratch space. */
-#define STACK_VAR(off) (STACK_SIZE - off)
+#define STACK_VAR(off) (STACK_SIZE - off - 4)

 #if __LINUX_ARM_ARCH__ < 7
On Fri, Aug 17, 2018 at 02:40:19PM +0200, Daniel Borkmann wrote:
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?
I think this is almost certainly the problem - looking at the history,
it seems that the "-4" was assumed to be part of the scratch stuff in
commit 38ca93060163 ("bpf, arm32: save 4 bytes of unneeded stack space")
but it isn't - it's because "off" of zero refers to the top word in the
stack (iow at STACK_SIZE-4).

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 13.8Mbps down 630kbps up
According to speedtest.net: 13Mbps down 490kbps up
On 08/17/2018 06:17 PM, Russell King - ARM Linux wrote:
On Fri, Aug 17, 2018 at 02:40:19PM +0200, Daniel Borkmann wrote:
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?
I think this is almost certainly the problem - looking at the history,
it seems that the "-4" was assumed to be part of the scratch stuff in
commit 38ca93060163 ("bpf, arm32: save 4 bytes of unneeded stack space")
but it isn't - it's because "off" of zero refers to the top word in the
stack (iow at STACK_SIZE-4).
Yeah agree, my thinking as well (albeit bit late, sigh, sorry about that).
Waiting for Peter to get back with results for definite confirmation. Your
rework in 1c35ba122d4a ("ARM: net: bpf: use negative numbers for stacked
registers") and 96cced4e774a ("ARM: net: bpf: access eBPF scratch space using
ARM FP register") fixes this in mainline, so unless I'm missing something this
would only need a stand-alone fix for 4.18/stable which I can cook up and
submit then.

Thanks,
Daniel
Hi Daniel,

Daniel Borkmann [off-list ref] hat am 17. August 2018 um 20:30 geschrieben:


On 08/17/2018 06:17 PM, Russell King - ARM Linux wrote:
On Fri, Aug 17, 2018 at 02:40:19PM +0200, Daniel Borkmann wrote:
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?
I think this is almost certainly the problem - looking at the history,
it seems that the "-4" was assumed to be part of the scratch stuff in
commit 38ca93060163 ("bpf, arm32: save 4 bytes of unneeded stack space")
but it isn't - it's because "off" of zero refers to the top word in the
stack (iow at STACK_SIZE-4).
Yeah agree, my thinking as well (albeit bit late, sigh, sorry about that).
Waiting for Peter to get back with results for definite confirmation. Your
rework in 1c35ba122d4a ("ARM: net: bpf: use negative numbers for stacked
registers") and 96cced4e774a ("ARM: net: bpf: access eBPF scratch space using
ARM FP register") fixes this in mainline, so unless I'm missing something this
would only need a stand-alone fix for 4.18/stable which I can cook up and
submit then.
i was able to reproduce this issue on RPi 3 with Linux 4.18.1 + multi_v7_defconfig and the following  config changes:

 --- a/arch/arm/configs/multi_v7_defconfig
+++ b/arch/arm/configs/multi_v7_defconfig
@@ -2,7 +2,10 @@ CONFIG_SYSVIPC=y
 CONFIG_NO_HZ=y
 CONFIG_HIGH_RES_TIMERS=y
 CONFIG_CGROUPS=y
+CONFIG_CGROUP_BPF=y
 CONFIG_BLK_DEV_INITRD=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_EMBEDDED=y
 CONFIG_PERF_EVENTS=y
 CONFIG_MODULES=y
@@ -153,6 +156,8 @@ CONFIG_IPV6_MIP6=m
 CONFIG_IPV6_TUNNEL=m
 CONFIG_IPV6_MULTIPLE_TABLES=y
 CONFIG_NET_DSA=m
+CONFIG_BPF_JIT=y
+CONFIG_BPF_STREAM_PARSER=y
 CONFIG_CAN=y
 CONFIG_CAN_AT91=m
 CONFIG_CAN_FLEXCAN=m

After applying the "-4" patch the oopses doesn't appear during boot anymore.

Stefan

Thanks,
Daniel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Stefan,

On 08/17/2018 06:17 PM, Russell King - ARM Linux wrote:
On Fri, Aug 17, 2018 at 02:40:19PM +0200, Daniel Borkmann wrote:
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?
I think this is almost certainly the problem - looking at the history,
it seems that the "-4" was assumed to be part of the scratch stuff in
commit 38ca93060163 ("bpf, arm32: save 4 bytes of unneeded stack space")
but it isn't - it's because "off" of zero refers to the top word in the
stack (iow at STACK_SIZE-4).
Yeah agree, my thinking as well (albeit bit late, sigh, sorry about that).
Waiting for Peter to get back with results for definite confirmation. Your
rework in 1c35ba122d4a ("ARM: net: bpf: use negative numbers for stacked
registers") and 96cced4e774a ("ARM: net: bpf: access eBPF scratch space using
ARM FP register") fixes this in mainline, so unless I'm missing something this
would only need a stand-alone fix for 4.18/stable which I can cook up and
submit then.
i was able to reproduce this issue on RPi 3 with Linux 4.18.1 + multi_v7_defconfig and the following  config changes:

 --- a/arch/arm/configs/multi_v7_defconfig
+++ b/arch/arm/configs/multi_v7_defconfig
@@ -2,7 +2,10 @@ CONFIG_SYSVIPC=y
 CONFIG_NO_HZ=y
 CONFIG_HIGH_RES_TIMERS=y
 CONFIG_CGROUPS=y
+CONFIG_CGROUP_BPF=y
 CONFIG_BLK_DEV_INITRD=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_EMBEDDED=y
 CONFIG_PERF_EVENTS=y
 CONFIG_MODULES=y
@@ -153,6 +156,8 @@ CONFIG_IPV6_MIP6=m
 CONFIG_IPV6_TUNNEL=m
 CONFIG_IPV6_MULTIPLE_TABLES=y
 CONFIG_NET_DSA=m
+CONFIG_BPF_JIT=y
+CONFIG_BPF_STREAM_PARSER=y
 CONFIG_CAN=y
 CONFIG_CAN_AT91=m
 CONFIG_CAN_FLEXCAN=m

After applying the "-4" patch the oopses doesn't appear during boot anymore.
Would be fab to get that into the kernel so this is widely tested
moving forward.

Peter
On Fri, Aug 17, 2018 at 7:30 PM, Daniel Borkmann [off-list ref] wrote:
On 08/17/2018 06:17 PM, Russell King - ARM Linux wrote:
On Fri, Aug 17, 2018 at 02:40:19PM +0200, Daniel Borkmann wrote:
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?
I think this is almost certainly the problem - looking at the history,
it seems that the "-4" was assumed to be part of the scratch stuff in
commit 38ca93060163 ("bpf, arm32: save 4 bytes of unneeded stack space")
but it isn't - it's because "off" of zero refers to the top word in the
stack (iow at STACK_SIZE-4).
Yeah agree, my thinking as well (albeit bit late, sigh, sorry about that).
Waiting for Peter to get back with results for definite confirmation. Your
rework in 1c35ba122d4a ("ARM: net: bpf: use negative numbers for stacked
registers") and 96cced4e774a ("ARM: net: bpf: access eBPF scratch space using
ARM FP register") fixes this in mainline, so unless I'm missing something this
would only need a stand-alone fix for 4.18/stable which I can cook up and
submit then.
I can confirm that fixes the problems I was seeing on Fedora 29.

Feel free to add a tested by from me:

Tested-by: Peter Robinson <redacted>
On 08/17/2018 11:13 PM, Peter Robinson wrote:
On Fri, Aug 17, 2018 at 7:30 PM, Daniel Borkmann [off-list ref] wrote:
On 08/17/2018 06:17 PM, Russell King - ARM Linux wrote:
On Fri, Aug 17, 2018 at 02:40:19PM +0200, Daniel Borkmann wrote:
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?
I think this is almost certainly the problem - looking at the history,
it seems that the "-4" was assumed to be part of the scratch stuff in
commit 38ca93060163 ("bpf, arm32: save 4 bytes of unneeded stack space")
but it isn't - it's because "off" of zero refers to the top word in the
stack (iow at STACK_SIZE-4).
Yeah agree, my thinking as well (albeit bit late, sigh, sorry about that).
Waiting for Peter to get back with results for definite confirmation. Your
rework in 1c35ba122d4a ("ARM: net: bpf: use negative numbers for stacked
registers") and 96cced4e774a ("ARM: net: bpf: access eBPF scratch space using
ARM FP register") fixes this in mainline, so unless I'm missing something this
would only need a stand-alone fix for 4.18/stable which I can cook up and
submit then.
I can confirm that fixes the problems I was seeing on Fedora 29.

Feel free to add a tested by from me:

Tested-by: Peter Robinson <redacted>
Great, thanks everyone! Will get it out asap.
On Fri, Aug 17, 2018 at 5:17 PM, Russell King - ARM Linux
[off-list ref] wrote:
On Fri, Aug 17, 2018 at 02:40:19PM +0200, Daniel Borkmann wrote:
I'd have one potential bug suspicion, for the 4.18 one you were trying,
could you run with the below patch to see whether it would help?
I think this is almost certainly the problem - looking at the history,
it seems that the "-4" was assumed to be part of the scratch stuff in
commit 38ca93060163 ("bpf, arm32: save 4 bytes of unneeded stack space")
but it isn't - it's because "off" of zero refers to the top word in the
stack (iow at STACK_SIZE-4).
I can confirm that patch fixes the problem I was seeing.

Peter

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help