Fw: [Bug 201423] New: eth0: hw csum failure
22 messages,
6 authors,
2018-11-04 · open the first message on its own page
Begin forwarded message:
Date: Sun, 14 Oct 2018 10:42:48 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 201423] New: eth0: hw csum failure
https://bugzilla.kernel.org/show_bug.cgi?id=201423
Bug ID: 201423
Summary: eth0: hw csum failure
Product: Networking
Version: 2.5
Kernel Version: 4.19.0-rc7
Hardware: Intel
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: stephen@networkplumber.org
Reporter: rossi.f@inwind.it
Regression: No
I have a P6T DELUXE V2 motherboard and using the sky2 driver for the ethernet
ports. I get the following error message:
[ 433.727397] eth0: hw csum failure
[ 433.727406] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19
[ 433.727406] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 433.727407] Call Trace:
[ 433.727409] <IRQ>
[ 433.727415] dump_stack+0x46/0x5b
[ 433.727419] __skb_checksum_complete+0xb0/0xc0
[ 433.727423] tcp_v4_rcv+0x528/0xb60
[ 433.727426] ? ipt_do_table+0x2d0/0x400
[ 433.727429] ip_local_deliver_finish+0x5a/0x110
[ 433.727430] ip_local_deliver+0xe1/0xf0
[ 433.727431] ? ip_sublist_rcv_finish+0x60/0x60
[ 433.727432] ip_rcv+0xca/0xe0
[ 433.727434] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 433.727436] __netif_receive_skb_one_core+0x4b/0x70
[ 433.727438] netif_receive_skb_internal+0x4e/0x130
[ 433.727439] napi_gro_receive+0x6a/0x80
[ 433.727442] sky2_poll+0x707/0xd20
[ 433.727446] ? rcu_check_callbacks+0x1b4/0x900
[ 433.727447] net_rx_action+0x237/0x380
[ 433.727449] __do_softirq+0xdc/0x1e0
[ 433.727452] irq_exit+0xa9/0xb0
[ 433.727453] do_IRQ+0x45/0xc0
[ 433.727455] common_interrupt+0xf/0xf
[ 433.727456] </IRQ>
[ 433.727459] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 433.727461] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f
ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1
4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 433.727462] RSP: 0000:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 433.727463] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 433.727464] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 433.727465] RBP: ffff880237b263a0 R08: 0000000000000714 R09:
000000650512105d
[ 433.727465] R10: 00000000ffffffff R11: 0000000000000342 R12:
00000064fc2a8b1c
[ 433.727466] R13: 00000064fc25b35f R14: 0000000000000004 R15:
ffffffff8204af20
[ 433.727468] ? cpuidle_enter_state+0x119/0x200
[ 433.727471] do_idle+0x1bf/0x200
[ 433.727473] cpu_startup_entry+0x6a/0x70
[ 433.727475] start_secondary+0x17f/0x1c0
[ 433.727476] secondary_startup_64+0xa4/0xb0
[ 441.662954] eth0: hw csum failure
[ 441.662959] CPU: 4 PID: 4347 Comm: radeon_cs:0 Not tainted 4.19.0-rc7 #19
[ 441.662960] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 441.662960] Call Trace:
[ 441.662963] <IRQ>
[ 441.662968] dump_stack+0x46/0x5b
[ 441.662972] __skb_checksum_complete+0xb0/0xc0
[ 441.662975] tcp_v4_rcv+0x528/0xb60
[ 441.662979] ? ipt_do_table+0x2d0/0x400
[ 441.662981] ip_local_deliver_finish+0x5a/0x110
[ 441.662983] ip_local_deliver+0xe1/0xf0
[ 441.662985] ? ip_sublist_rcv_finish+0x60/0x60
[ 441.662986] ip_rcv+0xca/0xe0
[ 441.662988] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 441.662990] __netif_receive_skb_one_core+0x4b/0x70
[ 441.662993] netif_receive_skb_internal+0x4e/0x130
[ 441.662994] napi_gro_receive+0x6a/0x80
[ 441.662998] sky2_poll+0x707/0xd20
[ 441.663000] net_rx_action+0x237/0x380
[ 441.663002] __do_softirq+0xdc/0x1e0
[ 441.663005] irq_exit+0xa9/0xb0
[ 441.663007] do_IRQ+0x45/0xc0
[ 441.663009] common_interrupt+0xf/0xf
[ 441.663010] </IRQ>
[ 441.663012] RIP: 0010:merge+0x22/0xb0
[ 441.663014] Code: c3 31 c0 c3 90 90 90 90 41 56 41 55 41 54 55 48 89 d5 53
48 89 cb 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 <48> 85 c9
74 70 48 85 d2 74 6b 49 89 fd 49 89 f6 49 89 e4 eb 14 48
[ 441.663015] RSP: 0018:ffffc9000090b988 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffffde
[ 441.663017] RAX: 0000000000000000 RBX: ffff88021ab2d408 RCX:
ffff88021ab2d408
[ 441.663018] RDX: ffff88021ab2d388 RSI: ffffffffa021c440 RDI:
0000000000000000
[ 441.663019] RBP: ffff88021ab2d388 R08: 0000000000005ecf R09:
0000000000008500
[ 441.663020] R10: ffffea000877ec00 R11: ffff880236803500 R12:
ffffffffa021c440
[ 441.663021] R13: ffff88021ab2d448 R14: 0000000000000004 R15:
ffffc9000090b9e0
[ 441.663048] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663063] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663065] ? merge+0x57/0xb0
[ 441.663080] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663082] list_sort+0x8b/0x230
[ 441.663094] radeon_cs_parser_fini+0xdf/0x110 [radeon]
[ 441.663110] radeon_cs_ioctl+0x2a4/0x710 [radeon]
[ 441.663113] ? __switch_to_asm+0x34/0x70
[ 441.663114] ? __switch_to_asm+0x40/0x70
[ 441.663130] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663141] drm_ioctl_kernel+0xa3/0xe0 [drm]
[ 441.663149] drm_ioctl+0x2e2/0x380 [drm]
[ 441.663164] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663168] ? page_add_new_anon_rmap+0x42/0x70
[ 441.663171] do_vfs_ioctl+0x9a/0x600
[ 441.663173] ksys_ioctl+0x35/0x60
[ 441.663175] __x64_sys_ioctl+0x11/0x20
[ 441.663177] do_syscall_64+0x3d/0xf0
[ 441.663179] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 441.663180] RIP: 0033:0x7f9377377f37
[ 441.663182] Code: 00 00 00 75 0c 48 c7 c0 ff ff ff ff 48 83 c4 18 c3 e8 ad
db 01 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 10 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 21 4f 2c 00 f7 d8 64 89 01 48
[ 441.663183] RSP: 002b:00007f92c3130d28 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 441.663185] RAX: ffffffffffffffda RBX: 0000564498327ec0 RCX:
00007f9377377f37
[ 441.663186] RDX: 0000564498337ec8 RSI: 00000000c0206466 RDI:
0000000000000010
[ 441.663186] RBP: 0000564498337ec8 R08: 0000000000000000 R09:
0000000000000000
[ 441.663187] R10: 0000000000000000 R11: 0000000000000246 R12:
00000000c0206466
[ 441.663188] R13: 0000000000000010 R14: 0000000000000000 R15:
0000564497a38120
[ 462.833418] eth0: hw csum failure
[ 462.833428] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19
[ 462.833429] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 462.833429] Call Trace:
[ 462.833432] <IRQ>
[ 462.833438] dump_stack+0x46/0x5b
[ 462.833442] __skb_checksum_complete+0xb0/0xc0
[ 462.833446] tcp_v4_rcv+0x528/0xb60
[ 462.833449] ? ipt_do_table+0x2d0/0x400
[ 462.833452] ip_local_deliver_finish+0x5a/0x110
[ 462.833454] ip_local_deliver+0xe1/0xf0
[ 462.833455] ? ip_sublist_rcv_finish+0x60/0x60
[ 462.833457] ip_rcv+0xca/0xe0
[ 462.833459] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 462.833461] __netif_receive_skb_one_core+0x4b/0x70
[ 462.833464] netif_receive_skb_internal+0x4e/0x130
[ 462.833466] napi_gro_receive+0x6a/0x80
[ 462.833469] sky2_poll+0x707/0xd20
[ 462.833471] net_rx_action+0x237/0x380
[ 462.833474] __do_softirq+0xdc/0x1e0
[ 462.833477] irq_exit+0xa9/0xb0
[ 462.833479] do_IRQ+0x45/0xc0
[ 462.833481] common_interrupt+0xf/0xf
[ 462.833482] </IRQ>
[ 462.833486] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 462.833488] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f
ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1
4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 462.833489] RSP: 0018:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 462.833491] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 462.833492] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 462.833493] RBP: ffff880237b263a0 R08: 0000000000000000 R09:
0000000000000000
[ 462.833494] R10: 00000000ffffffff R11: 0000000000000273 R12:
0000006bc3052131
[ 462.833495] R13: 0000006bc2f99f57 R14: 0000000000000004 R15:
ffffffff8204af20
[ 462.833498] ? cpuidle_enter_state+0x119/0x200
[ 462.833503] do_idle+0x1bf/0x200
[ 462.833506] cpu_startup_entry+0x6a/0x70
[ 462.833510] start_secondary+0x17f/0x1c0
[ 462.833513] secondary_startup_64+0xa4/0xb0
Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
--
You are receiving this mail because:
You are the assignee for the bug.
On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger
[off-list ref] wrote:
Begin forwarded message:
Date: Sun, 14 Oct 2018 10:42:48 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 201423] New: eth0: hw csum failure
https://bugzilla.kernel.org/show_bug.cgi?id=201423
Bug ID: 201423
Summary: eth0: hw csum failure
Product: Networking
Version: 2.5
Kernel Version: 4.19.0-rc7
Hardware: Intel
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: stephen@networkplumber.org
Reporter: rossi.f@inwind.it
Regression: No
I have a P6T DELUXE V2 motherboard and using the sky2 driver for the ethernet
ports. I get the following error message:
[ 433.727397] eth0: hw csum failure
[ 433.727406] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19
[ 433.727406] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 433.727407] Call Trace:
[ 433.727409] <IRQ>
[ 433.727415] dump_stack+0x46/0x5b
[ 433.727419] __skb_checksum_complete+0xb0/0xc0
[ 433.727423] tcp_v4_rcv+0x528/0xb60
[ 433.727426] ? ipt_do_table+0x2d0/0x400
[ 433.727429] ip_local_deliver_finish+0x5a/0x110
[ 433.727430] ip_local_deliver+0xe1/0xf0
[ 433.727431] ? ip_sublist_rcv_finish+0x60/0x60
[ 433.727432] ip_rcv+0xca/0xe0
[ 433.727434] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 433.727436] __netif_receive_skb_one_core+0x4b/0x70
[ 433.727438] netif_receive_skb_internal+0x4e/0x130
[ 433.727439] napi_gro_receive+0x6a/0x80
[ 433.727442] sky2_poll+0x707/0xd20
[ 433.727446] ? rcu_check_callbacks+0x1b4/0x900
[ 433.727447] net_rx_action+0x237/0x380
[ 433.727449] __do_softirq+0xdc/0x1e0
[ 433.727452] irq_exit+0xa9/0xb0
[ 433.727453] do_IRQ+0x45/0xc0
[ 433.727455] common_interrupt+0xf/0xf
[ 433.727456] </IRQ>
[ 433.727459] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 433.727461] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f
ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1
4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 433.727462] RSP: 0000:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 433.727463] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 433.727464] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 433.727465] RBP: ffff880237b263a0 R08: 0000000000000714 R09:
000000650512105d
[ 433.727465] R10: 00000000ffffffff R11: 0000000000000342 R12:
00000064fc2a8b1c
[ 433.727466] R13: 00000064fc25b35f R14: 0000000000000004 R15:
ffffffff8204af20
[ 433.727468] ? cpuidle_enter_state+0x119/0x200
[ 433.727471] do_idle+0x1bf/0x200
[ 433.727473] cpu_startup_entry+0x6a/0x70
[ 433.727475] start_secondary+0x17f/0x1c0
[ 433.727476] secondary_startup_64+0xa4/0xb0
[ 441.662954] eth0: hw csum failure
[ 441.662959] CPU: 4 PID: 4347 Comm: radeon_cs:0 Not tainted 4.19.0-rc7 #19
[ 441.662960] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 441.662960] Call Trace:
[ 441.662963] <IRQ>
[ 441.662968] dump_stack+0x46/0x5b
[ 441.662972] __skb_checksum_complete+0xb0/0xc0
[ 441.662975] tcp_v4_rcv+0x528/0xb60
[ 441.662979] ? ipt_do_table+0x2d0/0x400
[ 441.662981] ip_local_deliver_finish+0x5a/0x110
[ 441.662983] ip_local_deliver+0xe1/0xf0
[ 441.662985] ? ip_sublist_rcv_finish+0x60/0x60
[ 441.662986] ip_rcv+0xca/0xe0
[ 441.662988] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 441.662990] __netif_receive_skb_one_core+0x4b/0x70
[ 441.662993] netif_receive_skb_internal+0x4e/0x130
[ 441.662994] napi_gro_receive+0x6a/0x80
[ 441.662998] sky2_poll+0x707/0xd20
[ 441.663000] net_rx_action+0x237/0x380
[ 441.663002] __do_softirq+0xdc/0x1e0
[ 441.663005] irq_exit+0xa9/0xb0
[ 441.663007] do_IRQ+0x45/0xc0
[ 441.663009] common_interrupt+0xf/0xf
[ 441.663010] </IRQ>
[ 441.663012] RIP: 0010:merge+0x22/0xb0
[ 441.663014] Code: c3 31 c0 c3 90 90 90 90 41 56 41 55 41 54 55 48 89 d5 53
48 89 cb 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 <48> 85 c9
74 70 48 85 d2 74 6b 49 89 fd 49 89 f6 49 89 e4 eb 14 48
[ 441.663015] RSP: 0018:ffffc9000090b988 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffffde
[ 441.663017] RAX: 0000000000000000 RBX: ffff88021ab2d408 RCX:
ffff88021ab2d408
[ 441.663018] RDX: ffff88021ab2d388 RSI: ffffffffa021c440 RDI:
0000000000000000
[ 441.663019] RBP: ffff88021ab2d388 R08: 0000000000005ecf R09:
0000000000008500
[ 441.663020] R10: ffffea000877ec00 R11: ffff880236803500 R12:
ffffffffa021c440
[ 441.663021] R13: ffff88021ab2d448 R14: 0000000000000004 R15:
ffffc9000090b9e0
[ 441.663048] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663063] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663065] ? merge+0x57/0xb0
[ 441.663080] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663082] list_sort+0x8b/0x230
[ 441.663094] radeon_cs_parser_fini+0xdf/0x110 [radeon]
[ 441.663110] radeon_cs_ioctl+0x2a4/0x710 [radeon]
[ 441.663113] ? __switch_to_asm+0x34/0x70
[ 441.663114] ? __switch_to_asm+0x40/0x70
[ 441.663130] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663141] drm_ioctl_kernel+0xa3/0xe0 [drm]
[ 441.663149] drm_ioctl+0x2e2/0x380 [drm]
[ 441.663164] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663168] ? page_add_new_anon_rmap+0x42/0x70
[ 441.663171] do_vfs_ioctl+0x9a/0x600
[ 441.663173] ksys_ioctl+0x35/0x60
[ 441.663175] __x64_sys_ioctl+0x11/0x20
[ 441.663177] do_syscall_64+0x3d/0xf0
[ 441.663179] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 441.663180] RIP: 0033:0x7f9377377f37
[ 441.663182] Code: 00 00 00 75 0c 48 c7 c0 ff ff ff ff 48 83 c4 18 c3 e8 ad
db 01 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 10 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 21 4f 2c 00 f7 d8 64 89 01 48
[ 441.663183] RSP: 002b:00007f92c3130d28 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 441.663185] RAX: ffffffffffffffda RBX: 0000564498327ec0 RCX:
00007f9377377f37
[ 441.663186] RDX: 0000564498337ec8 RSI: 00000000c0206466 RDI:
0000000000000010
[ 441.663186] RBP: 0000564498337ec8 R08: 0000000000000000 R09:
0000000000000000
[ 441.663187] R10: 0000000000000000 R11: 0000000000000246 R12:
00000000c0206466
[ 441.663188] R13: 0000000000000010 R14: 0000000000000000 R15:
0000564497a38120
[ 462.833418] eth0: hw csum failure
[ 462.833428] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19
[ 462.833429] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 462.833429] Call Trace:
[ 462.833432] <IRQ>
[ 462.833438] dump_stack+0x46/0x5b
[ 462.833442] __skb_checksum_complete+0xb0/0xc0
[ 462.833446] tcp_v4_rcv+0x528/0xb60
[ 462.833449] ? ipt_do_table+0x2d0/0x400
[ 462.833452] ip_local_deliver_finish+0x5a/0x110
[ 462.833454] ip_local_deliver+0xe1/0xf0
[ 462.833455] ? ip_sublist_rcv_finish+0x60/0x60
[ 462.833457] ip_rcv+0xca/0xe0
[ 462.833459] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 462.833461] __netif_receive_skb_one_core+0x4b/0x70
[ 462.833464] netif_receive_skb_internal+0x4e/0x130
[ 462.833466] napi_gro_receive+0x6a/0x80
[ 462.833469] sky2_poll+0x707/0xd20
[ 462.833471] net_rx_action+0x237/0x380
[ 462.833474] __do_softirq+0xdc/0x1e0
[ 462.833477] irq_exit+0xa9/0xb0
[ 462.833479] do_IRQ+0x45/0xc0
[ 462.833481] common_interrupt+0xf/0xf
[ 462.833482] </IRQ>
[ 462.833486] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 462.833488] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f
ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1
4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 462.833489] RSP: 0018:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 462.833491] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 462.833492] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 462.833493] RBP: ffff880237b263a0 R08: 0000000000000000 R09:
0000000000000000
[ 462.833494] R10: 00000000ffffffff R11: 0000000000000273 R12:
0000006bc3052131
[ 462.833495] R13: 0000006bc2f99f57 R14: 0000000000000004 R15:
ffffffff8204af20
[ 462.833498] ? cpuidle_enter_state+0x119/0x200
[ 462.833503] do_idle+0x1bf/0x200
[ 462.833506] cpu_startup_entry+0x6a/0x70
[ 462.833510] start_secondary+0x17f/0x1c0
[ 462.833513] secondary_startup_64+0xa4/0xb0
Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
Hi Eric.
On Mon, 15 Oct 2018 at 16:42, Eric Dumazet [off-list ref] wrote:
On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger
[off-list ref] wrote: quoted
Begin forwarded message:
Date: Sun, 14 Oct 2018 10:42:48 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 201423] New: eth0: hw csum failure
https://bugzilla.kernel.org/show_bug.cgi?id=201423
Bug ID: 201423
Summary: eth0: hw csum failure
Product: Networking
Version: 2.5
Kernel Version: 4.19.0-rc7
Hardware: Intel
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: stephen@networkplumber.org
Reporter: rossi.f@inwind.it
Regression: No
I have a P6T DELUXE V2 motherboard and using the sky2 driver for the ethernet
ports. I get the following error message:
[ 433.727397] eth0: hw csum failure
[ 433.727406] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19
[ 433.727406] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 433.727407] Call Trace:
[ 433.727409] <IRQ>
[ 433.727415] dump_stack+0x46/0x5b
[ 433.727419] __skb_checksum_complete+0xb0/0xc0
[ 433.727423] tcp_v4_rcv+0x528/0xb60
[ 433.727426] ? ipt_do_table+0x2d0/0x400
[ 433.727429] ip_local_deliver_finish+0x5a/0x110
[ 433.727430] ip_local_deliver+0xe1/0xf0
[ 433.727431] ? ip_sublist_rcv_finish+0x60/0x60
[ 433.727432] ip_rcv+0xca/0xe0
[ 433.727434] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 433.727436] __netif_receive_skb_one_core+0x4b/0x70
[ 433.727438] netif_receive_skb_internal+0x4e/0x130
[ 433.727439] napi_gro_receive+0x6a/0x80
[ 433.727442] sky2_poll+0x707/0xd20
[ 433.727446] ? rcu_check_callbacks+0x1b4/0x900
[ 433.727447] net_rx_action+0x237/0x380
[ 433.727449] __do_softirq+0xdc/0x1e0
[ 433.727452] irq_exit+0xa9/0xb0
[ 433.727453] do_IRQ+0x45/0xc0
[ 433.727455] common_interrupt+0xf/0xf
[ 433.727456] </IRQ>
[ 433.727459] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 433.727461] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f
ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1
4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 433.727462] RSP: 0000:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 433.727463] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 433.727464] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 433.727465] RBP: ffff880237b263a0 R08: 0000000000000714 R09:
000000650512105d
[ 433.727465] R10: 00000000ffffffff R11: 0000000000000342 R12:
00000064fc2a8b1c
[ 433.727466] R13: 00000064fc25b35f R14: 0000000000000004 R15:
ffffffff8204af20
[ 433.727468] ? cpuidle_enter_state+0x119/0x200
[ 433.727471] do_idle+0x1bf/0x200
[ 433.727473] cpu_startup_entry+0x6a/0x70
[ 433.727475] start_secondary+0x17f/0x1c0
[ 433.727476] secondary_startup_64+0xa4/0xb0
[ 441.662954] eth0: hw csum failure
[ 441.662959] CPU: 4 PID: 4347 Comm: radeon_cs:0 Not tainted 4.19.0-rc7 #19
[ 441.662960] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 441.662960] Call Trace:
[ 441.662963] <IRQ>
[ 441.662968] dump_stack+0x46/0x5b
[ 441.662972] __skb_checksum_complete+0xb0/0xc0
[ 441.662975] tcp_v4_rcv+0x528/0xb60
[ 441.662979] ? ipt_do_table+0x2d0/0x400
[ 441.662981] ip_local_deliver_finish+0x5a/0x110
[ 441.662983] ip_local_deliver+0xe1/0xf0
[ 441.662985] ? ip_sublist_rcv_finish+0x60/0x60
[ 441.662986] ip_rcv+0xca/0xe0
[ 441.662988] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 441.662990] __netif_receive_skb_one_core+0x4b/0x70
[ 441.662993] netif_receive_skb_internal+0x4e/0x130
[ 441.662994] napi_gro_receive+0x6a/0x80
[ 441.662998] sky2_poll+0x707/0xd20
[ 441.663000] net_rx_action+0x237/0x380
[ 441.663002] __do_softirq+0xdc/0x1e0
[ 441.663005] irq_exit+0xa9/0xb0
[ 441.663007] do_IRQ+0x45/0xc0
[ 441.663009] common_interrupt+0xf/0xf
[ 441.663010] </IRQ>
[ 441.663012] RIP: 0010:merge+0x22/0xb0
[ 441.663014] Code: c3 31 c0 c3 90 90 90 90 41 56 41 55 41 54 55 48 89 d5 53
48 89 cb 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 <48> 85 c9
74 70 48 85 d2 74 6b 49 89 fd 49 89 f6 49 89 e4 eb 14 48
[ 441.663015] RSP: 0018:ffffc9000090b988 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffffde
[ 441.663017] RAX: 0000000000000000 RBX: ffff88021ab2d408 RCX:
ffff88021ab2d408
[ 441.663018] RDX: ffff88021ab2d388 RSI: ffffffffa021c440 RDI:
0000000000000000
[ 441.663019] RBP: ffff88021ab2d388 R08: 0000000000005ecf R09:
0000000000008500
[ 441.663020] R10: ffffea000877ec00 R11: ffff880236803500 R12:
ffffffffa021c440
[ 441.663021] R13: ffff88021ab2d448 R14: 0000000000000004 R15:
ffffc9000090b9e0
[ 441.663048] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663063] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663065] ? merge+0x57/0xb0
[ 441.663080] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663082] list_sort+0x8b/0x230
[ 441.663094] radeon_cs_parser_fini+0xdf/0x110 [radeon]
[ 441.663110] radeon_cs_ioctl+0x2a4/0x710 [radeon]
[ 441.663113] ? __switch_to_asm+0x34/0x70
[ 441.663114] ? __switch_to_asm+0x40/0x70
[ 441.663130] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663141] drm_ioctl_kernel+0xa3/0xe0 [drm]
[ 441.663149] drm_ioctl+0x2e2/0x380 [drm]
[ 441.663164] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663168] ? page_add_new_anon_rmap+0x42/0x70
[ 441.663171] do_vfs_ioctl+0x9a/0x600
[ 441.663173] ksys_ioctl+0x35/0x60
[ 441.663175] __x64_sys_ioctl+0x11/0x20
[ 441.663177] do_syscall_64+0x3d/0xf0
[ 441.663179] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 441.663180] RIP: 0033:0x7f9377377f37
[ 441.663182] Code: 00 00 00 75 0c 48 c7 c0 ff ff ff ff 48 83 c4 18 c3 e8 ad
db 01 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 10 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 21 4f 2c 00 f7 d8 64 89 01 48
[ 441.663183] RSP: 002b:00007f92c3130d28 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 441.663185] RAX: ffffffffffffffda RBX: 0000564498327ec0 RCX:
00007f9377377f37
[ 441.663186] RDX: 0000564498337ec8 RSI: 00000000c0206466 RDI:
0000000000000010
[ 441.663186] RBP: 0000564498337ec8 R08: 0000000000000000 R09:
0000000000000000
[ 441.663187] R10: 0000000000000000 R11: 0000000000000246 R12:
00000000c0206466
[ 441.663188] R13: 0000000000000010 R14: 0000000000000000 R15:
0000564497a38120
[ 462.833418] eth0: hw csum failure
[ 462.833428] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19
[ 462.833429] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 462.833429] Call Trace:
[ 462.833432] <IRQ>
[ 462.833438] dump_stack+0x46/0x5b
[ 462.833442] __skb_checksum_complete+0xb0/0xc0
[ 462.833446] tcp_v4_rcv+0x528/0xb60
[ 462.833449] ? ipt_do_table+0x2d0/0x400
[ 462.833452] ip_local_deliver_finish+0x5a/0x110
[ 462.833454] ip_local_deliver+0xe1/0xf0
[ 462.833455] ? ip_sublist_rcv_finish+0x60/0x60
[ 462.833457] ip_rcv+0xca/0xe0
[ 462.833459] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 462.833461] __netif_receive_skb_one_core+0x4b/0x70
[ 462.833464] netif_receive_skb_internal+0x4e/0x130
[ 462.833466] napi_gro_receive+0x6a/0x80
[ 462.833469] sky2_poll+0x707/0xd20
[ 462.833471] net_rx_action+0x237/0x380
[ 462.833474] __do_softirq+0xdc/0x1e0
[ 462.833477] irq_exit+0xa9/0xb0
[ 462.833479] do_IRQ+0x45/0xc0
[ 462.833481] common_interrupt+0xf/0xf
[ 462.833482] </IRQ>
[ 462.833486] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 462.833488] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f
ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1
4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 462.833489] RSP: 0018:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 462.833491] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 462.833492] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 462.833493] RBP: ffff880237b263a0 R08: 0000000000000000 R09:
0000000000000000
[ 462.833494] R10: 00000000ffffffff R11: 0000000000000273 R12:
0000006bc3052131
[ 462.833495] R13: 0000006bc2f99f57 R14: 0000000000000004 R15:
ffffffff8204af20
[ 462.833498] ? cpuidle_enter_state+0x119/0x200
[ 462.833503] do_idle+0x1bf/0x200
[ 462.833506] cpu_startup_entry+0x6a/0x70
[ 462.833510] start_secondary+0x17f/0x1c0
[ 462.833513] secondary_startup_64+0xa4/0xb0
Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
Just to say that we've also just hit this with both the LAN78xx and
SMSC9514 drivers, ie all Raspberry Pis with onboard ethernet. Likewise
that commit had been pinpointed as the cause, or at least exposing an
underlying issue.
As the patch has been backported in 4.14.71 it's hitting LTS users too.
Thanks for the pointer on sungem. I'll have a look into what's going
on and see if we can sort it, although I have cc'ed in the maintainers
of those chips in case they are already on the case.
Cheers.
Dave
On Mon, 15 Oct 2018 08:41:47 -0700
Eric Dumazet [off-list ref] wrote:
On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger
[off-list ref] wrote: quoted
Begin forwarded message:
Date: Sun, 14 Oct 2018 10:42:48 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 201423] New: eth0: hw csum failure
https://bugzilla.kernel.org/show_bug.cgi?id=201423
Bug ID: 201423
Summary: eth0: hw csum failure
Product: Networking
Version: 2.5
Kernel Version: 4.19.0-rc7
Hardware: Intel
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: stephen@networkplumber.org
Reporter: rossi.f@inwind.it
Regression: No
I have a P6T DELUXE V2 motherboard and using the sky2 driver for the ethernet
ports. I get the following error message:
[ 433.727397] eth0: hw csum failure
[ 433.727406] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19
[ 433.727406] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 433.727407] Call Trace:
[ 433.727409] <IRQ>
[ 433.727415] dump_stack+0x46/0x5b
[ 433.727419] __skb_checksum_complete+0xb0/0xc0
[ 433.727423] tcp_v4_rcv+0x528/0xb60
[ 433.727426] ? ipt_do_table+0x2d0/0x400
[ 433.727429] ip_local_deliver_finish+0x5a/0x110
[ 433.727430] ip_local_deliver+0xe1/0xf0
[ 433.727431] ? ip_sublist_rcv_finish+0x60/0x60
[ 433.727432] ip_rcv+0xca/0xe0
[ 433.727434] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 433.727436] __netif_receive_skb_one_core+0x4b/0x70
[ 433.727438] netif_receive_skb_internal+0x4e/0x130
[ 433.727439] napi_gro_receive+0x6a/0x80
[ 433.727442] sky2_poll+0x707/0xd20
[ 433.727446] ? rcu_check_callbacks+0x1b4/0x900
[ 433.727447] net_rx_action+0x237/0x380
[ 433.727449] __do_softirq+0xdc/0x1e0
[ 433.727452] irq_exit+0xa9/0xb0
[ 433.727453] do_IRQ+0x45/0xc0
[ 433.727455] common_interrupt+0xf/0xf
[ 433.727456] </IRQ>
[ 433.727459] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 433.727461] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f
ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1
4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 433.727462] RSP: 0000:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 433.727463] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 433.727464] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 433.727465] RBP: ffff880237b263a0 R08: 0000000000000714 R09:
000000650512105d
[ 433.727465] R10: 00000000ffffffff R11: 0000000000000342 R12:
00000064fc2a8b1c
[ 433.727466] R13: 00000064fc25b35f R14: 0000000000000004 R15:
ffffffff8204af20
[ 433.727468] ? cpuidle_enter_state+0x119/0x200
[ 433.727471] do_idle+0x1bf/0x200
[ 433.727473] cpu_startup_entry+0x6a/0x70
[ 433.727475] start_secondary+0x17f/0x1c0
[ 433.727476] secondary_startup_64+0xa4/0xb0
[ 441.662954] eth0: hw csum failure
[ 441.662959] CPU: 4 PID: 4347 Comm: radeon_cs:0 Not tainted 4.19.0-rc7 #19
[ 441.662960] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 441.662960] Call Trace:
[ 441.662963] <IRQ>
[ 441.662968] dump_stack+0x46/0x5b
[ 441.662972] __skb_checksum_complete+0xb0/0xc0
[ 441.662975] tcp_v4_rcv+0x528/0xb60
[ 441.662979] ? ipt_do_table+0x2d0/0x400
[ 441.662981] ip_local_deliver_finish+0x5a/0x110
[ 441.662983] ip_local_deliver+0xe1/0xf0
[ 441.662985] ? ip_sublist_rcv_finish+0x60/0x60
[ 441.662986] ip_rcv+0xca/0xe0
[ 441.662988] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 441.662990] __netif_receive_skb_one_core+0x4b/0x70
[ 441.662993] netif_receive_skb_internal+0x4e/0x130
[ 441.662994] napi_gro_receive+0x6a/0x80
[ 441.662998] sky2_poll+0x707/0xd20
[ 441.663000] net_rx_action+0x237/0x380
[ 441.663002] __do_softirq+0xdc/0x1e0
[ 441.663005] irq_exit+0xa9/0xb0
[ 441.663007] do_IRQ+0x45/0xc0
[ 441.663009] common_interrupt+0xf/0xf
[ 441.663010] </IRQ>
[ 441.663012] RIP: 0010:merge+0x22/0xb0
[ 441.663014] Code: c3 31 c0 c3 90 90 90 90 41 56 41 55 41 54 55 48 89 d5 53
48 89 cb 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 <48> 85 c9
74 70 48 85 d2 74 6b 49 89 fd 49 89 f6 49 89 e4 eb 14 48
[ 441.663015] RSP: 0018:ffffc9000090b988 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffffde
[ 441.663017] RAX: 0000000000000000 RBX: ffff88021ab2d408 RCX:
ffff88021ab2d408
[ 441.663018] RDX: ffff88021ab2d388 RSI: ffffffffa021c440 RDI:
0000000000000000
[ 441.663019] RBP: ffff88021ab2d388 R08: 0000000000005ecf R09:
0000000000008500
[ 441.663020] R10: ffffea000877ec00 R11: ffff880236803500 R12:
ffffffffa021c440
[ 441.663021] R13: ffff88021ab2d448 R14: 0000000000000004 R15:
ffffc9000090b9e0
[ 441.663048] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663063] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663065] ? merge+0x57/0xb0
[ 441.663080] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon]
[ 441.663082] list_sort+0x8b/0x230
[ 441.663094] radeon_cs_parser_fini+0xdf/0x110 [radeon]
[ 441.663110] radeon_cs_ioctl+0x2a4/0x710 [radeon]
[ 441.663113] ? __switch_to_asm+0x34/0x70
[ 441.663114] ? __switch_to_asm+0x40/0x70
[ 441.663130] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663141] drm_ioctl_kernel+0xa3/0xe0 [drm]
[ 441.663149] drm_ioctl+0x2e2/0x380 [drm]
[ 441.663164] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663168] ? page_add_new_anon_rmap+0x42/0x70
[ 441.663171] do_vfs_ioctl+0x9a/0x600
[ 441.663173] ksys_ioctl+0x35/0x60
[ 441.663175] __x64_sys_ioctl+0x11/0x20
[ 441.663177] do_syscall_64+0x3d/0xf0
[ 441.663179] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 441.663180] RIP: 0033:0x7f9377377f37
[ 441.663182] Code: 00 00 00 75 0c 48 c7 c0 ff ff ff ff 48 83 c4 18 c3 e8 ad
db 01 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 10 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 21 4f 2c 00 f7 d8 64 89 01 48
[ 441.663183] RSP: 002b:00007f92c3130d28 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 441.663185] RAX: ffffffffffffffda RBX: 0000564498327ec0 RCX:
00007f9377377f37
[ 441.663186] RDX: 0000564498337ec8 RSI: 00000000c0206466 RDI:
0000000000000010
[ 441.663186] RBP: 0000564498337ec8 R08: 0000000000000000 R09:
0000000000000000
[ 441.663187] R10: 0000000000000000 R11: 0000000000000246 R12:
00000000c0206466
[ 441.663188] R13: 0000000000000010 R14: 0000000000000000 R15:
0000564497a38120
[ 462.833418] eth0: hw csum failure
[ 462.833428] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19
[ 462.833429] Hardware name: System manufacturer System Product Name/P6T
DELUXE V2, BIOS 1202 12/22/2010
[ 462.833429] Call Trace:
[ 462.833432] <IRQ>
[ 462.833438] dump_stack+0x46/0x5b
[ 462.833442] __skb_checksum_complete+0xb0/0xc0
[ 462.833446] tcp_v4_rcv+0x528/0xb60
[ 462.833449] ? ipt_do_table+0x2d0/0x400
[ 462.833452] ip_local_deliver_finish+0x5a/0x110
[ 462.833454] ip_local_deliver+0xe1/0xf0
[ 462.833455] ? ip_sublist_rcv_finish+0x60/0x60
[ 462.833457] ip_rcv+0xca/0xe0
[ 462.833459] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 462.833461] __netif_receive_skb_one_core+0x4b/0x70
[ 462.833464] netif_receive_skb_internal+0x4e/0x130
[ 462.833466] napi_gro_receive+0x6a/0x80
[ 462.833469] sky2_poll+0x707/0xd20
[ 462.833471] net_rx_action+0x237/0x380
[ 462.833474] __do_softirq+0xdc/0x1e0
[ 462.833477] irq_exit+0xa9/0xb0
[ 462.833479] do_IRQ+0x45/0xc0
[ 462.833481] common_interrupt+0xf/0xf
[ 462.833482] </IRQ>
[ 462.833486] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 462.833488] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f
ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1
4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 462.833489] RSP: 0018:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 462.833491] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 462.833492] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 462.833493] RBP: ffff880237b263a0 R08: 0000000000000000 R09:
0000000000000000
[ 462.833494] R10: 00000000ffffffff R11: 0000000000000273 R12:
0000006bc3052131
[ 462.833495] R13: 0000006bc2f99f57 R14: 0000000000000004 R15:
ffffffff8204af20
[ 462.833498] ? cpuidle_enter_state+0x119/0x200
[ 462.833503] do_idle+0x1bf/0x200
[ 462.833506] cpu_startup_entry+0x6a/0x70
[ 462.833510] start_secondary+0x17f/0x1c0
[ 462.833513] secondary_startup_64+0xa4/0xb0
Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
There are two variants of this chip, one does 1's compliment checksum, and
the other one does TCP checksum. Maybe the 1's compliment version is incorrectly
including the CRC.
Side note, not sure why but the driver only calls gro for checksummed packets.
Is that necessary?
On 15 October 2018 17:41:47 CEST, Eric Dumazet [off-list ref] wrote: On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger
[off-list ref] wrote: quoted
Begin forwarded message:
Date: Sun, 14 Oct 2018 10:42:48 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 201423] New: eth0: hw csum failure
https://bugzilla.kernel.org/show_bug.cgi?id=201423
Bug ID: 201423
Summary: eth0: hw csum failure
Product: Networking
Version: 2.5
Kernel Version: 4.19.0-rc7
Hardware: Intel
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: stephen@networkplumber.org
Reporter: rossi.f@inwind.it
Regression: No
I have a P6T DELUXE V2 motherboard and using the sky2 driver for the ethernet quoted ports. I get the following error message:
[ 433.727397] eth0: hw csum failure
[ 433.727406] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19 quoted [ 433.727406] Hardware name: System manufacturer System Product Name/P6T quoted DELUXE V2, BIOS 1202 12/22/2010
[ 433.727407] Call Trace:
[ 433.727409] <IRQ>
[ 433.727415] dump_stack+0x46/0x5b
[ 433.727419] __skb_checksum_complete+0xb0/0xc0
[ 433.727423] tcp_v4_rcv+0x528/0xb60
[ 433.727426] ? ipt_do_table+0x2d0/0x400
[ 433.727429] ip_local_deliver_finish+0x5a/0x110
[ 433.727430] ip_local_deliver+0xe1/0xf0
[ 433.727431] ? ip_sublist_rcv_finish+0x60/0x60
[ 433.727432] ip_rcv+0xca/0xe0
[ 433.727434] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 433.727436] __netif_receive_skb_one_core+0x4b/0x70
[ 433.727438] netif_receive_skb_internal+0x4e/0x130
[ 433.727439] napi_gro_receive+0x6a/0x80
[ 433.727442] sky2_poll+0x707/0xd20
[ 433.727446] ? rcu_check_callbacks+0x1b4/0x900
[ 433.727447] net_rx_action+0x237/0x380
[ 433.727449] __do_softirq+0xdc/0x1e0
[ 433.727452] irq_exit+0xa9/0xb0
[ 433.727453] do_IRQ+0x45/0xc0
[ 433.727455] common_interrupt+0xf/0xf
[ 433.727456] </IRQ>
[ 433.727459] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 433.727461] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f quoted ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1 quoted 4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 433.727462] RSP: 0000:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 433.727463] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 433.727464] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 433.727465] RBP: ffff880237b263a0 R08: 0000000000000714 R09:
000000650512105d
[ 433.727465] R10: 00000000ffffffff R11: 0000000000000342 R12:
00000064fc2a8b1c
[ 433.727466] R13: 00000064fc25b35f R14: 0000000000000004 R15:
ffffffff8204af20
[ 433.727468] ? cpuidle_enter_state+0x119/0x200
[ 433.727471] do_idle+0x1bf/0x200
[ 433.727473] cpu_startup_entry+0x6a/0x70
[ 433.727475] start_secondary+0x17f/0x1c0
[ 433.727476] secondary_startup_64+0xa4/0xb0
[ 441.662954] eth0: hw csum failure
[ 441.662959] CPU: 4 PID: 4347 Comm: radeon_cs:0 Not tainted 4.19.0-rc7 #19 quoted [ 441.662960] Hardware name: System manufacturer System Product Name/P6T quoted DELUXE V2, BIOS 1202 12/22/2010
[ 441.662960] Call Trace:
[ 441.662963] <IRQ>
[ 441.662968] dump_stack+0x46/0x5b
[ 441.662972] __skb_checksum_complete+0xb0/0xc0
[ 441.662975] tcp_v4_rcv+0x528/0xb60
[ 441.662979] ? ipt_do_table+0x2d0/0x400
[ 441.662981] ip_local_deliver_finish+0x5a/0x110
[ 441.662983] ip_local_deliver+0xe1/0xf0
[ 441.662985] ? ip_sublist_rcv_finish+0x60/0x60
[ 441.662986] ip_rcv+0xca/0xe0
[ 441.662988] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 441.662990] __netif_receive_skb_one_core+0x4b/0x70
[ 441.662993] netif_receive_skb_internal+0x4e/0x130
[ 441.662994] napi_gro_receive+0x6a/0x80
[ 441.662998] sky2_poll+0x707/0xd20
[ 441.663000] net_rx_action+0x237/0x380
[ 441.663002] __do_softirq+0xdc/0x1e0
[ 441.663005] irq_exit+0xa9/0xb0
[ 441.663007] do_IRQ+0x45/0xc0
[ 441.663009] common_interrupt+0xf/0xf
[ 441.663010] </IRQ>
[ 441.663012] RIP: 0010:merge+0x22/0xb0
[ 441.663014] Code: c3 31 c0 c3 90 90 90 90 41 56 41 55 41 54 55 48 89 d5 53 quoted 48 89 cb 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 c0 <48> 85 c9 quoted 74 70 48 85 d2 74 6b 49 89 fd 49 89 f6 49 89 e4 eb 14 48
[ 441.663015] RSP: 0018:ffffc9000090b988 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffffde
[ 441.663017] RAX: 0000000000000000 RBX: ffff88021ab2d408 RCX:
ffff88021ab2d408
[ 441.663018] RDX: ffff88021ab2d388 RSI: ffffffffa021c440 RDI:
0000000000000000
[ 441.663019] RBP: ffff88021ab2d388 R08: 0000000000005ecf R09:
0000000000008500
[ 441.663020] R10: ffffea000877ec00 R11: ffff880236803500 R12:
ffffffffa021c440
[ 441.663021] R13: ffff88021ab2d448 R14: 0000000000000004 R15:
ffffc9000090b9e0
[ 441.663048] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon] quoted [ 441.663063] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon] quoted [ 441.663065] ? merge+0x57/0xb0
[ 441.663080] ? radeon_irq_kms_set_irq_n_enabled+0x120/0x120 [radeon] quoted [ 441.663082] list_sort+0x8b/0x230
[ 441.663094] radeon_cs_parser_fini+0xdf/0x110 [radeon]
[ 441.663110] radeon_cs_ioctl+0x2a4/0x710 [radeon]
[ 441.663113] ? __switch_to_asm+0x34/0x70
[ 441.663114] ? __switch_to_asm+0x40/0x70
[ 441.663130] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663141] drm_ioctl_kernel+0xa3/0xe0 [drm]
[ 441.663149] drm_ioctl+0x2e2/0x380 [drm]
[ 441.663164] ? radeon_cs_parser_init+0x20/0x20 [radeon]
[ 441.663168] ? page_add_new_anon_rmap+0x42/0x70
[ 441.663171] do_vfs_ioctl+0x9a/0x600
[ 441.663173] ksys_ioctl+0x35/0x60
[ 441.663175] __x64_sys_ioctl+0x11/0x20
[ 441.663177] do_syscall_64+0x3d/0xf0
[ 441.663179] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 441.663180] RIP: 0033:0x7f9377377f37
[ 441.663182] Code: 00 00 00 75 0c 48 c7 c0 ff ff ff ff 48 83 c4 18 c3 e8 ad quoted db 01 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 10 00 00 00 0f 05 <48> 3d 01 quoted f0 ff ff 73 01 c3 48 8b 0d 21 4f 2c 00 f7 d8 64 89 01 48
[ 441.663183] RSP: 002b:00007f92c3130d28 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 441.663185] RAX: ffffffffffffffda RBX: 0000564498327ec0 RCX:
00007f9377377f37
[ 441.663186] RDX: 0000564498337ec8 RSI: 00000000c0206466 RDI:
0000000000000010
[ 441.663186] RBP: 0000564498337ec8 R08: 0000000000000000 R09:
0000000000000000
[ 441.663187] R10: 0000000000000000 R11: 0000000000000246 R12:
00000000c0206466
[ 441.663188] R13: 0000000000000010 R14: 0000000000000000 R15:
0000564497a38120
[ 462.833418] eth0: hw csum failure
[ 462.833428] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.19.0-rc7 #19 quoted [ 462.833429] Hardware name: System manufacturer System Product Name/P6T quoted DELUXE V2, BIOS 1202 12/22/2010
[ 462.833429] Call Trace:
[ 462.833432] <IRQ>
[ 462.833438] dump_stack+0x46/0x5b
[ 462.833442] __skb_checksum_complete+0xb0/0xc0
[ 462.833446] tcp_v4_rcv+0x528/0xb60
[ 462.833449] ? ipt_do_table+0x2d0/0x400
[ 462.833452] ip_local_deliver_finish+0x5a/0x110
[ 462.833454] ip_local_deliver+0xe1/0xf0
[ 462.833455] ? ip_sublist_rcv_finish+0x60/0x60
[ 462.833457] ip_rcv+0xca/0xe0
[ 462.833459] ? ip_rcv_finish_core.isra.0+0x300/0x300
[ 462.833461] __netif_receive_skb_one_core+0x4b/0x70
[ 462.833464] netif_receive_skb_internal+0x4e/0x130
[ 462.833466] napi_gro_receive+0x6a/0x80
[ 462.833469] sky2_poll+0x707/0xd20
[ 462.833471] net_rx_action+0x237/0x380
[ 462.833474] __do_softirq+0xdc/0x1e0
[ 462.833477] irq_exit+0xa9/0xb0
[ 462.833479] do_IRQ+0x45/0xc0
[ 462.833481] common_interrupt+0xf/0xf
[ 462.833482] </IRQ>
[ 462.833486] RIP: 0010:cpuidle_enter_state+0x124/0x200
[ 462.833488] Code: 53 60 89 c3 e8 dd 90 ad ff 65 8b 3d 96 58 a7 7e e8 d1 8f quoted ad ff 31 ff 49 89 c4 e8 27 99 ad ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 89 e1 quoted 4c 29 e9 48 89 c8 48 c1 f9 3f 48 f7 ea b8 ff ff ff 7f 48
[ 462.833489] RSP: 0018:ffffc900000a3e98 EFLAGS: 00000282 ORIG_RAX:
ffffffffffffffde
[ 462.833491] RAX: ffff880237b1f280 RBX: 0000000000000004 RCX:
000000000000001f
[ 462.833492] RDX: 20c49ba5e353f7cf RSI: 000000002fe419c1 RDI:
0000000000000000
[ 462.833493] RBP: ffff880237b263a0 R08: 0000000000000000 R09:
0000000000000000
[ 462.833494] R10: 00000000ffffffff R11: 0000000000000273 R12:
0000006bc3052131
[ 462.833495] R13: 0000006bc2f99f57 R14: 0000000000000004 R15:
ffffffff8204af20
[ 462.833498] ? cpuidle_enter_state+0x119/0x200
[ 462.833503] do_idle+0x1bf/0x200
[ 462.833506] cpu_startup_entry+0x6a/0x70
[ 462.833510] start_secondary+0x17f/0x1c0
[ 462.833513] secondary_startup_64+0xa4/0xb0
Something is changed between 4.17.12 and 4.18, after bisecting the problem I quoted got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large quoted packets defeat CHECKSUM_COMPLETE optimization because of NIC adding quoted zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed quoted to CHECKSUM_NONE, forcing a full csum validation, even if all prior quoted fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
here is the chip found on my motherboard so it seems that flag is not set
# dmesg | grep sky2
[ 0.545661] sky2: driver version 1.30 [ 0.545781] sky2 0000:06:00.0: Yukon-2 EC Ultra chip revision 3 [ 0.546067] sky2 0000:06:00.0 eth0: addr 00:24:8c:xx:xx:xx [ 0.546188] sky2 0000:04:00.0: Yukon-2 EC Ultra chip revision 3 [ 0.546484] sky2 0000:04:00.0 eth1: addr 00:24:8c:xx:xx:xx [ 38.043074] sky2 0000:06:00.0 eth0: enabling interface [ 39.842161] sky2 0000:06:00.0 eth0: Link is up at 100 Mbps, full duplex, flow control rx
On 15.10.2018 17:41, Eric Dumazet wrote: On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger quoted Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
I've seen similar on several systems with mlx4 cards when using 4.18.x -
that is hw csum failure followed by some backtrace.
Only seems to happen on systems dealing with quite a bit of UDP.
Example from 4.18.10: [635607.740574] p0xe0: hw csum failure
[635607.740598] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740599] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740599] Call Trace:
[635607.740602] <IRQ>
[635607.740611] dump_stack+0x5c/0x7b
[635607.740617] __skb_gro_checksum_complete+0x9a/0xa0
[635607.740621] udp6_gro_receive+0x211/0x290
[635607.740624] ipv6_gro_receive+0x1a8/0x390
[635607.740627] dev_gro_receive+0x33e/0x550
[635607.740628] napi_gro_frags+0xa2/0x210
[635607.740635] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740648] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740654] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740657] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740658] net_rx_action+0xe0/0x2e0
[635607.740662] __do_softirq+0xd8/0x2e5
[635607.740666] irq_exit+0xb4/0xc0
[635607.740667] do_IRQ+0x85/0xd0
[635607.740670] common_interrupt+0xf/0xf
[635607.740671] </IRQ>
[635607.740675] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740675] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740701] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740703] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740703] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740704] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740705] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740706] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740709] ? cpuidle_enter_state+0x91/0x2a0
[635607.740712] do_idle+0x1d0/0x240
[635607.740715] cpu_startup_entry+0x5f/0x70
[635607.740719] start_secondary+0x185/0x1a0
[635607.740722] secondary_startup_64+0xa5/0xb0
[635607.740731] p0xe0: hw csum failure
[635607.740745] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740746] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740746] Call Trace:
[635607.740747] <IRQ>
[635607.740750] dump_stack+0x5c/0x7b
[635607.740755] __skb_checksum_complete+0xb8/0xd0
[635607.740760] __udp6_lib_rcv+0xa6b/0xa70
[635607.740767] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740770] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740774] ip6_input_finish+0xc0/0x460
[635607.740776] ip6_input+0x2b/0x90
[635607.740778] ? ip6_rcv_finish+0x110/0x110
[635607.740780] ipv6_rcv+0x2cd/0x4b0
[635607.740783] ? udp6_lib_lookup_skb+0x59/0x80
[635607.740785] __netif_receive_skb_core+0x455/0xb30
[635607.740788] ? ipv6_gro_receive+0x1a8/0x390
[635607.740790] ? netif_receive_skb_internal+0x24/0xb0
[635607.740792] netif_receive_skb_internal+0x24/0xb0
[635607.740793] napi_gro_frags+0x165/0x210
[635607.740796] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740802] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740807] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740810] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740811] net_rx_action+0xe0/0x2e0
[635607.740813] __do_softirq+0xd8/0x2e5
[635607.740816] irq_exit+0xb4/0xc0
[635607.740817] do_IRQ+0x85/0xd0
[635607.740820] common_interrupt+0xf/0xf
[635607.740821] </IRQ>
[635607.740823] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740823] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740848] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740849] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740850] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740851] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740852] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740853] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740855] ? cpuidle_enter_state+0x91/0x2a0
[635607.740857] do_idle+0x1d0/0x240
[635607.740859] cpu_startup_entry+0x5f/0x70
[635607.740861] start_secondary+0x185/0x1a0
[635607.740863] secondary_startup_64+0xa5/0xb0
On Mon, Oct 15, 2018 at 11:30 PM Andre Tomt [off-list ref] wrote:
On 15.10.2018 17:41, Eric Dumazet wrote: quoted On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger quoted Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
I've seen similar on several systems with mlx4 cards when using 4.18.x -
that is hw csum failure followed by some backtrace.
Only seems to happen on systems dealing with quite a bit of UDP.
Strange, because mlx4 on IPv6+UDP should not use CHECKSUM_COMPLETE,
but CHECKSUM_UNNECESSARY
I would be nice to track this a bit further, maybe by providing the
full packet content.
Example from 4.18.10: quoted [635607.740574] p0xe0: hw csum failure
[635607.740598] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740599] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740599] Call Trace:
[635607.740602] <IRQ>
[635607.740611] dump_stack+0x5c/0x7b
[635607.740617] __skb_gro_checksum_complete+0x9a/0xa0
[635607.740621] udp6_gro_receive+0x211/0x290
[635607.740624] ipv6_gro_receive+0x1a8/0x390
[635607.740627] dev_gro_receive+0x33e/0x550
[635607.740628] napi_gro_frags+0xa2/0x210
[635607.740635] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740648] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740654] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740657] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740658] net_rx_action+0xe0/0x2e0
[635607.740662] __do_softirq+0xd8/0x2e5
[635607.740666] irq_exit+0xb4/0xc0
[635607.740667] do_IRQ+0x85/0xd0
[635607.740670] common_interrupt+0xf/0xf
[635607.740671] </IRQ>
[635607.740675] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740675] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740701] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740703] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740703] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740704] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740705] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740706] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740709] ? cpuidle_enter_state+0x91/0x2a0
[635607.740712] do_idle+0x1d0/0x240
[635607.740715] cpu_startup_entry+0x5f/0x70
[635607.740719] start_secondary+0x185/0x1a0
[635607.740722] secondary_startup_64+0xa5/0xb0
[635607.740731] p0xe0: hw csum failure
[635607.740745] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740746] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740746] Call Trace:
[635607.740747] <IRQ>
[635607.740750] dump_stack+0x5c/0x7b
[635607.740755] __skb_checksum_complete+0xb8/0xd0
[635607.740760] __udp6_lib_rcv+0xa6b/0xa70
[635607.740767] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740770] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740774] ip6_input_finish+0xc0/0x460
[635607.740776] ip6_input+0x2b/0x90
[635607.740778] ? ip6_rcv_finish+0x110/0x110
[635607.740780] ipv6_rcv+0x2cd/0x4b0
[635607.740783] ? udp6_lib_lookup_skb+0x59/0x80
[635607.740785] __netif_receive_skb_core+0x455/0xb30
[635607.740788] ? ipv6_gro_receive+0x1a8/0x390
[635607.740790] ? netif_receive_skb_internal+0x24/0xb0
[635607.740792] netif_receive_skb_internal+0x24/0xb0
[635607.740793] napi_gro_frags+0x165/0x210
[635607.740796] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740802] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740807] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740810] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740811] net_rx_action+0xe0/0x2e0
[635607.740813] __do_softirq+0xd8/0x2e5
[635607.740816] irq_exit+0xb4/0xc0
[635607.740817] do_IRQ+0x85/0xd0
[635607.740820] common_interrupt+0xf/0xf
[635607.740821] </IRQ>
[635607.740823] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740823] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740848] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740849] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740850] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740851] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740852] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740853] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740855] ? cpuidle_enter_state+0x91/0x2a0
[635607.740857] do_idle+0x1d0/0x240
[635607.740859] cpu_startup_entry+0x5f/0x70
[635607.740861] start_secondary+0x185/0x1a0
[635607.740863] secondary_startup_64+0xa5/0xb0
On 10/16/2018 06:00 AM, Eric Dumazet wrote: On Mon, Oct 15, 2018 at 11:30 PM Andre Tomt [off-list ref] wrote: quoted
On 15.10.2018 17:41, Eric Dumazet wrote: quoted On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger quoted Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
I've seen similar on several systems with mlx4 cards when using 4.18.x -
that is hw csum failure followed by some backtrace.
Only seems to happen on systems dealing with quite a bit of UDP.
Strange, because mlx4 on IPv6+UDP should not use CHECKSUM_COMPLETE,
but CHECKSUM_UNNECESSARY
I would be nice to track this a bit further, maybe by providing the
full packet content.
quoted Example from 4.18.10: quoted [635607.740574] p0xe0: hw csum failure
[635607.740598] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740599] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740599] Call Trace:
[635607.740602] <IRQ>
[635607.740611] dump_stack+0x5c/0x7b
[635607.740617] __skb_gro_checksum_complete+0x9a/0xa0
[635607.740621] udp6_gro_receive+0x211/0x290
[635607.740624] ipv6_gro_receive+0x1a8/0x390
[635607.740627] dev_gro_receive+0x33e/0x550
[635607.740628] napi_gro_frags+0xa2/0x210
[635607.740635] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740648] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740654] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740657] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740658] net_rx_action+0xe0/0x2e0
[635607.740662] __do_softirq+0xd8/0x2e5
[635607.740666] irq_exit+0xb4/0xc0
[635607.740667] do_IRQ+0x85/0xd0
[635607.740670] common_interrupt+0xf/0xf
[635607.740671] </IRQ>
[635607.740675] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740675] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740701] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740703] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740703] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740704] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740705] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740706] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740709] ? cpuidle_enter_state+0x91/0x2a0
[635607.740712] do_idle+0x1d0/0x240
[635607.740715] cpu_startup_entry+0x5f/0x70
[635607.740719] start_secondary+0x185/0x1a0
[635607.740722] secondary_startup_64+0xa5/0xb0
[635607.740731] p0xe0: hw csum failure
[635607.740745] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740746] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740746] Call Trace:
[635607.740747] <IRQ>
[635607.740750] dump_stack+0x5c/0x7b
[635607.740755] __skb_checksum_complete+0xb8/0xd0
[635607.740760] __udp6_lib_rcv+0xa6b/0xa70
[635607.740767] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740770] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740774] ip6_input_finish+0xc0/0x460
[635607.740776] ip6_input+0x2b/0x90
[635607.740778] ? ip6_rcv_finish+0x110/0x110
[635607.740780] ipv6_rcv+0x2cd/0x4b0
[635607.740783] ? udp6_lib_lookup_skb+0x59/0x80
[635607.740785] __netif_receive_skb_core+0x455/0xb30
[635607.740788] ? ipv6_gro_receive+0x1a8/0x390
[635607.740790] ? netif_receive_skb_internal+0x24/0xb0
[635607.740792] netif_receive_skb_internal+0x24/0xb0
[635607.740793] napi_gro_frags+0x165/0x210
[635607.740796] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740802] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740807] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740810] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740811] net_rx_action+0xe0/0x2e0
[635607.740813] __do_softirq+0xd8/0x2e5
[635607.740816] irq_exit+0xb4/0xc0
[635607.740817] do_IRQ+0x85/0xd0
[635607.740820] common_interrupt+0xf/0xf
[635607.740821] </IRQ>
[635607.740823] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740823] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740848] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740849] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740850] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740851] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740852] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740853] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740855] ? cpuidle_enter_state+0x91/0x2a0
[635607.740857] do_idle+0x1d0/0x240
[635607.740859] cpu_startup_entry+0x5f/0x70
[635607.740861] start_secondary+0x185/0x1a0
[635607.740863] secondary_startup_64+0xa5/0xb0
As a matter of fact Dimitris found the issue in the patch and is working on a fix involving csum_block_sub()
Problems comes from trimming an odd number of bytes.
On 10/19/2018 02:58 PM, Eric Dumazet wrote:
On 10/16/2018 06:00 AM, Eric Dumazet wrote: quoted On Mon, Oct 15, 2018 at 11:30 PM Andre Tomt [off-list ref] wrote: quoted
On 15.10.2018 17:41, Eric Dumazet wrote: quoted On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger quoted Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
I've seen similar on several systems with mlx4 cards when using 4.18.x -
that is hw csum failure followed by some backtrace.
Only seems to happen on systems dealing with quite a bit of UDP.
Strange, because mlx4 on IPv6+UDP should not use CHECKSUM_COMPLETE,
but CHECKSUM_UNNECESSARY
I would be nice to track this a bit further, maybe by providing the
full packet content.
quoted Example from 4.18.10: quoted [635607.740574] p0xe0: hw csum failure
[635607.740598] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740599] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740599] Call Trace:
[635607.740602] <IRQ>
[635607.740611] dump_stack+0x5c/0x7b
[635607.740617] __skb_gro_checksum_complete+0x9a/0xa0
[635607.740621] udp6_gro_receive+0x211/0x290
[635607.740624] ipv6_gro_receive+0x1a8/0x390
[635607.740627] dev_gro_receive+0x33e/0x550
[635607.740628] napi_gro_frags+0xa2/0x210
[635607.740635] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740648] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740654] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740657] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740658] net_rx_action+0xe0/0x2e0
[635607.740662] __do_softirq+0xd8/0x2e5
[635607.740666] irq_exit+0xb4/0xc0
[635607.740667] do_IRQ+0x85/0xd0
[635607.740670] common_interrupt+0xf/0xf
[635607.740671] </IRQ>
[635607.740675] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740675] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740701] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740703] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740703] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740704] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740705] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740706] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740709] ? cpuidle_enter_state+0x91/0x2a0
[635607.740712] do_idle+0x1d0/0x240
[635607.740715] cpu_startup_entry+0x5f/0x70
[635607.740719] start_secondary+0x185/0x1a0
[635607.740722] secondary_startup_64+0xa5/0xb0
[635607.740731] p0xe0: hw csum failure
[635607.740745] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740746] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740746] Call Trace:
[635607.740747] <IRQ>
[635607.740750] dump_stack+0x5c/0x7b
[635607.740755] __skb_checksum_complete+0xb8/0xd0
[635607.740760] __udp6_lib_rcv+0xa6b/0xa70
[635607.740767] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740770] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740774] ip6_input_finish+0xc0/0x460
[635607.740776] ip6_input+0x2b/0x90
[635607.740778] ? ip6_rcv_finish+0x110/0x110
[635607.740780] ipv6_rcv+0x2cd/0x4b0
[635607.740783] ? udp6_lib_lookup_skb+0x59/0x80
[635607.740785] __netif_receive_skb_core+0x455/0xb30
[635607.740788] ? ipv6_gro_receive+0x1a8/0x390
[635607.740790] ? netif_receive_skb_internal+0x24/0xb0
[635607.740792] netif_receive_skb_internal+0x24/0xb0
[635607.740793] napi_gro_frags+0x165/0x210
[635607.740796] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740802] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740807] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740810] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740811] net_rx_action+0xe0/0x2e0
[635607.740813] __do_softirq+0xd8/0x2e5
[635607.740816] irq_exit+0xb4/0xc0
[635607.740817] do_IRQ+0x85/0xd0
[635607.740820] common_interrupt+0xf/0xf
[635607.740821] </IRQ>
[635607.740823] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740823] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740848] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740849] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740850] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740851] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740852] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740853] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740855] ? cpuidle_enter_state+0x91/0x2a0
[635607.740857] do_idle+0x1d0/0x240
[635607.740859] cpu_startup_entry+0x5f/0x70
[635607.740861] start_secondary+0x185/0x1a0
[635607.740863] secondary_startup_64+0xa5/0xb0
As a matter of fact Dimitris found the issue in the patch and is working on a fix involving csum_block_sub()
Problems comes from trimming an odd number of bytes.
More exactly, trimming bytes starting at an odd offset.
On 20.10.2018 00:25, Eric Dumazet wrote: On 10/19/2018 02:58 PM, Eric Dumazet wrote: quoted On 10/16/2018 06:00 AM, Eric Dumazet wrote: quoted On Mon, Oct 15, 2018 at 11:30 PM Andre Tomt [off-list ref] wrote: quoted I've seen similar on several systems with mlx4 cards when using 4.18.x -
that is hw csum failure followed by some backtrace.
Only seems to happen on systems dealing with quite a bit of UDP.
Strange, because mlx4 on IPv6+UDP should not use CHECKSUM_COMPLETE,
but CHECKSUM_UNNECESSARY
I would be nice to track this a bit further, maybe by providing the
full packet content. <snip> quoted
As a matter of fact Dimitris found the issue in the patch and is working on a fix involving csum_block_sub()
Problems comes from trimming an odd number of bytes.
More exactly, trimming bytes starting at an odd offset.
No hw csum failures here since I deployed Dimitris fix on top of 4.18.16
32 hours ago.
Thanks
On 21.10.2018 15:34, Andre Tomt wrote: On 20.10.2018 00:25, Eric Dumazet wrote: quoted On 10/19/2018 02:58 PM, Eric Dumazet wrote: quoted On 10/16/2018 06:00 AM, Eric Dumazet wrote: quoted On Mon, Oct 15, 2018 at 11:30 PM Andre Tomt [off-list ref] wrote: quoted I've seen similar on several systems with mlx4 cards when using
4.18.x -
that is hw csum failure followed by some backtrace.
Only seems to happen on systems dealing with quite a bit of UDP.
Strange, because mlx4 on IPv6+UDP should not use CHECKSUM_COMPLETE,
but CHECKSUM_UNNECESSARY
I would be nice to track this a bit further, maybe by providing the
full packet content. <snip> quoted quoted
As a matter of fact Dimitris found the issue in the patch and is
working on a fix involving csum_block_sub()
Problems comes from trimming an odd number of bytes.
More exactly, trimming bytes starting at an odd offset.
No hw csum failures here since I deployed Dimitris fix on top of 4.18.16
32 hours ago.
Thanks
It eventually showed up again with mlx4, on 4.18.16 + fix and also on
4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux distributions.
[116116.994519] p0xe0: hw csum failure
[116116.994550] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 4.19.0-1 #1
[116116.994551] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[116116.994555] Call Trace:
[116116.994558] <IRQ>
[116116.994567] dump_stack+0x5c/0x7b
[116116.994574] __skb_gro_checksum_complete+0x9a/0xa0
[116116.994580] udp6_gro_receive+0x211/0x290
[116116.994585] ipv6_gro_receive+0x1b1/0x3a0
[116116.994588] dev_gro_receive+0x3a0/0x620
[116116.994590] ? __build_skb+0x25/0xe0
[116116.994592] napi_gro_frags+0xa8/0x220
[116116.994598] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[116116.994611] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[116116.994621] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[116116.994629] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[116116.994635] net_rx_action+0xe0/0x2e0
[116116.994641] __do_softirq+0xd8/0x2ff
[116116.994646] irq_exit+0xbd/0xd0
[116116.994650] do_IRQ+0x85/0xd0
[116116.994656] common_interrupt+0xf/0xf
[116116.994659] </IRQ>
[116116.994665] RIP: 0010:cpuidle_enter_state+0xb3/0x310
[116116.994668] Code: 31 ff e8 e0 e0 bb ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3f 02 00 00 31 ff e8 64 cc c0 ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[116116.994669] RSP: 0018:ffff924a0635bea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffda
[116116.994671] RAX: ffff9016ffb60fc0 RBX: 0000699b9835d616 RCX: 000000000000001f
[116116.994673] RDX: 0000699b9835d616 RSI: 00000000229837f7 RDI: 0000000000000000
[116116.994674] RBP: 0000000000000001 R08: 0000000000000002 R09: 0000000000020840
[116116.994675] R10: ffff924a0635be88 R11: 0000000000000367 R12: ffff9016ffb69aa8
[116116.994676] R13: ffffffffa50ac638 R14: 0000000000000000 R15: 0000699b981c63b9
[116116.994680] ? cpuidle_enter_state+0x90/0x310
[116116.994685] do_idle+0x1d0/0x240
[116116.994687] cpu_startup_entry+0x5f/0x70
[116116.994690] start_secondary+0x185/0x1a0
[116116.994693] secondary_startup_64+0xa4/0xb0
[116116.994709] p0xe0: hw csum failure
[116116.994739] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 4.19.0-1 #1
[116116.994740] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[116116.994741] Call Trace:
[116116.994743] <IRQ>
[116116.994746] dump_stack+0x5c/0x7b
[116116.994751] __skb_checksum_complete+0xb8/0xd0
[116116.994755] __udp6_lib_rcv+0xa0e/0xa20
[116116.994764] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[116116.994768] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[116116.994771] ip6_input_finish+0xc0/0x460
[116116.994774] ip6_input+0x2b/0x90
[116116.994776] ? ip6_make_skb+0x1b0/0x1b0
[116116.994778] ipv6_rcv+0x54/0xb0
[116116.994781] __netif_receive_skb_one_core+0x42/0x50
[116116.994784] netif_receive_skb_internal+0x24/0xb0
[116116.994786] napi_gro_frags+0x171/0x220
[116116.994790] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[116116.994798] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[116116.994803] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[116116.994806] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[116116.994808] net_rx_action+0xe0/0x2e0
[116116.994810] __do_softirq+0xd8/0x2ff
[116116.994812] irq_exit+0xbd/0xd0
[116116.994814] do_IRQ+0x85/0xd0
[116116.994816] common_interrupt+0xf/0xf
[116116.994818] </IRQ>
[116116.994821] RIP: 0010:cpuidle_enter_state+0xb3/0x310
[116116.994823] Code: 31 ff e8 e0 e0 bb ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3f 02 00 00 31 ff e8 64 cc c0 ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[116116.994824] RSP: 0018:ffff924a0635bea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffda
[116116.994825] RAX: ffff9016ffb60fc0 RBX: 0000699b9835d616 RCX: 000000000000001f
[116116.994826] RDX: 0000699b9835d616 RSI: 00000000229837f7 RDI: 0000000000000000
[116116.994827] RBP: 0000000000000001 R08: 0000000000000002 R09: 0000000000020840
[116116.994828] R10: ffff924a0635be88 R11: 0000000000000367 R12: ffff9016ffb69aa8
[116116.994829] R13: ffffffffa50ac638 R14: 0000000000000000 R15: 0000699b981c63b9
[116116.994832] ? cpuidle_enter_state+0x90/0x310
[116116.994835] do_idle+0x1d0/0x240
[116116.994837] cpu_startup_entry+0x5f/0x70
[116116.994838] start_secondary+0x185/0x1a0
[116116.994840] secondary_startup_64+0xa4/0xb0
On 10/24/2018 12:41 PM, Andre Tomt wrote:
It eventually showed up again with mlx4, on 4.18.16 + fix and also on 4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
On 25.10.2018 19:38, Eric Dumazet wrote:
On 10/24/2018 12:41 PM, Andre Tomt wrote: quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and also on 4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything shows up.
On 26.10.2018 13:45, Andre Tomt wrote: On 25.10.2018 19:38, Eric Dumazet wrote: quoted
On 10/24/2018 12:41 PM, Andre Tomt wrote: quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and also on
4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything shows up.
And it tripped again with that commit; however on another box with a
much more complicated setup (VRFs, sch_cake, ifb, conntrack/nat, 6in4
tunnel, VF device on mlx4)
[ 8197.348260] wanib: hw csum failure
[ 8197.348288] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.0-1 #1
[ 8197.348289] Hardware name: Supermicro SYS-5018D-FN8T/X10SDV-TP8F, BIOS 1.3 03/19/2018
[ 8197.348290] Call Trace:
[ 8197.348296] <IRQ>
[ 8197.348304] dump_stack+0x5c/0x80
[ 8197.348308] __skb_checksum_complete+0xac/0xc0
[ 8197.348318] icmp_error+0x1c8/0x1f0 [nf_conntrack]
[ 8197.348325] ? ip_output+0x61/0xc0
[ 8197.348328] ? skb_copy_bits+0x13d/0x220
[ 8197.348334] nf_conntrack_in+0xd8/0x390 [nf_conntrack]
[ 8197.348339] ? ___pskb_trim+0x192/0x330
[ 8197.348343] nf_hook_slow+0x43/0xc0
[ 8197.348346] ip_rcv+0x90/0xb0
[ 8197.348349] ? ip_rcv_finish_core.isra.0+0x310/0x310
[ 8197.348354] __netif_receive_skb_one_core+0x42/0x50
[ 8197.348357] netif_receive_skb_internal+0x24/0xb0
[ 8197.348361] ifb_ri_tasklet+0x167/0x260 [ifb]
[ 8197.348365] tasklet_action_common.isra.3+0x49/0xb0
[ 8197.348369] __do_softirq+0xe7/0x2d3
[ 8197.348372] irq_exit+0x96/0xd0
[ 8197.348375] do_IRQ+0x85/0xd0
[ 8197.348378] common_interrupt+0xf/0xf
[ 8197.348379] </IRQ>
[ 8197.348382] RIP: 0010:cpuidle_enter_state+0xb9/0x320
[ 8197.348384] Code: e8 1c 16 bc ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 3e fb c0 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3
[ 8197.348386] RSP: 0018:ffff9f0441953ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd5
[ 8197.348388] RAX: ffff9759efae0fc0 RBX: 000007749807d911 RCX: 000000000000001f
[ 8197.348390] RDX: 000007749807d911 RSI: 000000003a2e8670 RDI: 0000000000000000
[ 8197.348393] RBP: ffff9759efae98a8 R08: 0000000000000002 R09: 0000000000020840
[ 8197.348396] R10: 00626b4810384abc R11: ffff9759efae01e8 R12: 0000000000000001
[ 8197.348398] R13: ffffffff8d0ac638 R14: 0000000000000001 R15: 0000000000000000
[ 8197.348402] ? cpuidle_enter_state+0x94/0x320
[ 8197.348407] do_idle+0x1e4/0x220
[ 8197.348411] cpu_startup_entry+0x5f/0x70
[ 8197.348415] start_secondary+0x185/0x1a0
[ 8197.348417] secondary_startup_64+0xa4/0xb0
On Fri, Oct 26, 2018 at 5:38 AM Andre Tomt [off-list ref] wrote:
On 26.10.2018 13:45, Andre Tomt wrote: quoted On 25.10.2018 19:38, Eric Dumazet wrote: quoted
On 10/24/2018 12:41 PM, Andre Tomt wrote: quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and also on
4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything shows up.
And it tripped again with that commit; however on another box with a
much more complicated setup (VRFs, sch_cake, ifb, conntrack/nat, 6in4
tunnel, VF device on mlx4)
quoted [ 8197.348260] wanib: hw csum failure
[ 8197.348288] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.0-1 #1
[ 8197.348289] Hardware name: Supermicro SYS-5018D-FN8T/X10SDV-TP8F, BIOS 1.3 03/19/2018
[ 8197.348290] Call Trace:
[ 8197.348296] <IRQ>
[ 8197.348304] dump_stack+0x5c/0x80
[ 8197.348308] __skb_checksum_complete+0xac/0xc0
[ 8197.348318] icmp_error+0x1c8/0x1f0 [nf_conntrack]
[ 8197.348325] ? ip_output+0x61/0xc0
[ 8197.348328] ? skb_copy_bits+0x13d/0x220
[ 8197.348334] nf_conntrack_in+0xd8/0x390 [nf_conntrack]
[ 8197.348339] ? ___pskb_trim+0x192/0x330
[ 8197.348343] nf_hook_slow+0x43/0xc0
[ 8197.348346] ip_rcv+0x90/0xb0
[ 8197.348349] ? ip_rcv_finish_core.isra.0+0x310/0x310
[ 8197.348354] __netif_receive_skb_one_core+0x42/0x50
[ 8197.348357] netif_receive_skb_internal+0x24/0xb0
[ 8197.348361] ifb_ri_tasklet+0x167/0x260 [ifb]
[ 8197.348365] tasklet_action_common.isra.3+0x49/0xb0
[ 8197.348369] __do_softirq+0xe7/0x2d3
[ 8197.348372] irq_exit+0x96/0xd0
[ 8197.348375] do_IRQ+0x85/0xd0
[ 8197.348378] common_interrupt+0xf/0xf
[ 8197.348379] </IRQ>
[ 8197.348382] RIP: 0010:cpuidle_enter_state+0xb9/0x320
[ 8197.348384] Code: e8 1c 16 bc ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 3e fb c0 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3
[ 8197.348386] RSP: 0018:ffff9f0441953ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd5
[ 8197.348388] RAX: ffff9759efae0fc0 RBX: 000007749807d911 RCX: 000000000000001f
[ 8197.348390] RDX: 000007749807d911 RSI: 000000003a2e8670 RDI: 0000000000000000
[ 8197.348393] RBP: ffff9759efae98a8 R08: 0000000000000002 R09: 0000000000020840
[ 8197.348396] R10: 00626b4810384abc R11: ffff9759efae01e8 R12: 0000000000000001
[ 8197.348398] R13: ffffffff8d0ac638 R14: 0000000000000001 R15: 0000000000000000
[ 8197.348402] ? cpuidle_enter_state+0x94/0x320
[ 8197.348407] do_idle+0x1e4/0x220
[ 8197.348411] cpu_startup_entry+0x5f/0x70
[ 8197.348415] start_secondary+0x185/0x1a0
[ 8197.348417] secondary_startup_64+0xa4/0xb0
Very different trace , yet another bug to track .
If you can, try to remove some components from this setup.
On 26.10.2018 14:59, Eric Dumazet wrote: On Fri, Oct 26, 2018 at 5:38 AM Andre Tomt [off-list ref] wrote: quoted And it tripped again with that commit; however on another box with a
much more complicated setup (VRFs, sch_cake, ifb, conntrack/nat, 6in4
tunnel, VF device on mlx4)
quoted [ 8197.348260] wanib: hw csum failure
[ 8197.348288] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.0-1 #1
[ 8197.348289] Hardware name: Supermicro SYS-5018D-FN8T/X10SDV-TP8F, BIOS 1.3 03/19/2018
[ 8197.348290] Call Trace:
[ 8197.348296] <IRQ>
[ 8197.348304] dump_stack+0x5c/0x80
[ 8197.348308] __skb_checksum_complete+0xac/0xc0
[ 8197.348318] icmp_error+0x1c8/0x1f0 [nf_conntrack]
[ 8197.348325] ? ip_output+0x61/0xc0
[ 8197.348328] ? skb_copy_bits+0x13d/0x220
[ 8197.348334] nf_conntrack_in+0xd8/0x390 [nf_conntrack]
[ 8197.348339] ? ___pskb_trim+0x192/0x330
[ 8197.348343] nf_hook_slow+0x43/0xc0
[ 8197.348346] ip_rcv+0x90/0xb0
[ 8197.348349] ? ip_rcv_finish_core.isra.0+0x310/0x310
[ 8197.348354] __netif_receive_skb_one_core+0x42/0x50
[ 8197.348357] netif_receive_skb_internal+0x24/0xb0
[ 8197.348361] ifb_ri_tasklet+0x167/0x260 [ifb]
[ 8197.348365] tasklet_action_common.isra.3+0x49/0xb0
[ 8197.348369] __do_softirq+0xe7/0x2d3
[ 8197.348372] irq_exit+0x96/0xd0
[ 8197.348375] do_IRQ+0x85/0xd0
[ 8197.348378] common_interrupt+0xf/0xf
[ 8197.348379] </IRQ>
[ 8197.348382] RIP: 0010:cpuidle_enter_state+0xb9/0x320
[ 8197.348384] Code: e8 1c 16 bc ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 3e fb c0 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3
[ 8197.348386] RSP: 0018:ffff9f0441953ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd5
[ 8197.348388] RAX: ffff9759efae0fc0 RBX: 000007749807d911 RCX: 000000000000001f
[ 8197.348390] RDX: 000007749807d911 RSI: 000000003a2e8670 RDI: 0000000000000000
[ 8197.348393] RBP: ffff9759efae98a8 R08: 0000000000000002 R09: 0000000000020840
[ 8197.348396] R10: 00626b4810384abc R11: ffff9759efae01e8 R12: 0000000000000001
[ 8197.348398] R13: ffffffff8d0ac638 R14: 0000000000000001 R15: 0000000000000000
[ 8197.348402] ? cpuidle_enter_state+0x94/0x320
[ 8197.348407] do_idle+0x1e4/0x220
[ 8197.348411] cpu_startup_entry+0x5f/0x70
[ 8197.348415] start_secondary+0x185/0x1a0
[ 8197.348417] secondary_startup_64+0xa4/0xb0
Very different trace , yet another bug to track .
If you can, try to remove some components from this setup.
Will do. Just remembered I took out the VF stuff a few days ago and that
netdev is just a normal vlan device now. Going to eliminate VRF and
cake/ifb as well.
On 26.10.2018 13:45, Andre Tomt wrote: On 25.10.2018 19:38, Eric Dumazet wrote: quoted
On 10/24/2018 12:41 PM, Andre Tomt wrote: quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and also on
4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything shows up.
Just hit it on the simpler server; no VRF, no tunnels, no nat/conntrack.
Only a basic stateless nftables ruleset and a vlan netdev (unlikely to
be the one triggering this I guess; it has only v4 traffic).
On 4.19 + above commit: [158269.360271] p0xe0: hw csum failure
[158269.360286] CPU: 3 PID: 0 Comm: swapper/3 Tainted: P O 4.19.0-1 #1
[158269.360287] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[158269.360288] Call Trace:
[158269.360290] <IRQ>
[158269.360295] dump_stack+0x5c/0x7b
[158269.360299] __skb_gro_checksum_complete+0x9a/0xa0
[158269.360301] udp6_gro_receive+0x211/0x290
[158269.360303] ipv6_gro_receive+0x1b1/0x3a0
[158269.360306] ? ip_sublist_rcv_finish+0x70/0x70
[158269.360307] dev_gro_receive+0x3a0/0x620
[158269.360309] ? __build_skb+0x25/0xe0
[158269.360310] napi_gro_frags+0xa8/0x220
[158269.360314] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[158269.360322] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[158269.360325] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[158269.360327] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[158269.360329] net_rx_action+0xe0/0x2e0
[158269.360330] __do_softirq+0xd8/0x2ff
[158269.360333] irq_exit+0xbd/0xd0
[158269.360334] do_IRQ+0x85/0xd0
[158269.360336] common_interrupt+0xf/0xf
[158269.360337] </IRQ>
[158269.360339] RIP: 0010:cpuidle_enter_state+0xb3/0x310
[158269.360340] Code: 31 ff e8 e0 e0 bb ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3f 02 00 00 31 ff e8 64 cc c0 ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[158269.360341] RSP: 0018:ffffaf28c634bea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[158269.360342] RAX: ffff9a9f7fae0fc0 RBX: 00008ff1f4ff622a RCX: 000000000000001f
[158269.360343] RDX: 00008ff1f4ff622a RSI: 0000000022983893 RDI: 0000000000000000
[158269.360343] RBP: 0000000000000001 R08: 0000000000000002 R09: 0000000000020840
[158269.360344] R10: ffffaf28c634be88 R11: 0000000000000036 R12: ffff9a9f7fae9aa8
[158269.360344] R13: ffffffffaa0ac638 R14: 0000000000000000 R15: 00008ff1f4f09d43
[158269.360347] ? cpuidle_enter_state+0x90/0x310
[158269.360349] do_idle+0x1d0/0x240
[158269.360351] cpu_startup_entry+0x5f/0x70
[158269.360352] start_secondary+0x185/0x1a0
[158269.360354] secondary_startup_64+0xa4/0xb0
On 27.10.2018 23:41, Andre Tomt wrote: On 26.10.2018 13:45, Andre Tomt wrote: quoted On 25.10.2018 19:38, Eric Dumazet wrote: quoted
On 10/24/2018 12:41 PM, Andre Tomt wrote: quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and also
on 4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything shows
up.
Just hit it on the simpler server; no VRF, no tunnels, no nat/conntrack.
Only a basic stateless nftables ruleset and a vlan netdev (unlikely to
be the one triggering this I guess; it has only v4 traffic).
I'm currently testing 4.19 with the recomended commit added, plus these
to sort out some GRO issues (on a hunch, unsure if related):
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=a8305bff685252e80b7c60f4f5e7dd2e63e38218
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=992cba7e276d438ac8b0a8c17b147b37c8c286f7
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ece23711dd956cd5053c9cb03e9fe0668f9c8894
and I *think* it is behaving better now? it's not conclusive as it could
take a while to trip in this environment but some of the test servers
have not shown anything bad in almost 24h.
On 30.10.2018 11:58, Andre Tomt wrote: On 27.10.2018 23:41, Andre Tomt wrote: quoted On 26.10.2018 13:45, Andre Tomt wrote: quoted On 25.10.2018 19:38, Eric Dumazet wrote: quoted
On 10/24/2018 12:41 PM, Andre Tomt wrote: quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and also
on 4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything
shows up.
Just hit it on the simpler server; no VRF, no tunnels, no
nat/conntrack. Only a basic stateless nftables ruleset and a vlan
netdev (unlikely to be the one triggering this I guess; it has only v4
traffic).
I'm currently testing 4.19 with the recomended commit added, plus these
to sort out some GRO issues (on a hunch, unsure if related):
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=a8305bff685252e80b7c60f4f5e7dd2e63e38218
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=992cba7e276d438ac8b0a8c17b147b37c8c286f7
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ece23711dd956cd5053c9cb03e9fe0668f9c8894
and I *think* it is behaving better now? it's not conclusive as it could
take a while to trip in this environment but some of the test servers
have not shown anything bad in almost 24h.
Sorry, s/some of the/none of the
On 10/16/2018 06:00 AM, Eric Dumazet wrote: quoted On Mon, Oct 15, 2018 at 11:30 PM Andre Tomt [off-list ref] wrote: quoted
On 15.10.2018 17:41, Eric Dumazet wrote: quoted On Mon, Oct 15, 2018 at 8:15 AM Stephen Hemminger quoted Something is changed between 4.17.12 and 4.18, after bisecting the problem I
got the following first bad commit:
commit 88078d98d1bb085d72af8437707279e203524fa5
Author: Eric Dumazet [off-list ref]
Date: Wed Apr 18 11:43:15 2018 -0700
net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends
After working on IP defragmentation lately, I found that some large
packets defeat CHECKSUM_COMPLETE optimization because of NIC adding
zero paddings on the last (small) fragment.
While removing the padding with pskb_trim_rcsum(), we set skb->ip_summed
to CHECKSUM_NONE, forcing a full csum validation, even if all prior
fragments had CHECKSUM_COMPLETE set.
We can instead compute the checksum of the part we are trimming,
usually smaller than the part we keep.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Thanks for bisecting !
This commit is known to expose some NIC/driver bugs.
Look at commit 12b03558cef6d655d0d394f5e98a6fd07c1f6c0f
("net: sungem: fix rx checksum support") for one driver needing a fix.
I assume SKY2_HW_NEW_LE is not set on your NIC ?
I've seen similar on several systems with mlx4 cards when using 4.18.x -
that is hw csum failure followed by some backtrace.
Only seems to happen on systems dealing with quite a bit of UDP.
Strange, because mlx4 on IPv6+UDP should not use CHECKSUM_COMPLETE,
but CHECKSUM_UNNECESSARY
I would be nice to track this a bit further, maybe by providing the
full packet content.
quoted Example from 4.18.10: quoted [635607.740574] p0xe0: hw csum failure
[635607.740598] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740599] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740599] Call Trace:
[635607.740602] <IRQ>
[635607.740611] dump_stack+0x5c/0x7b
[635607.740617] __skb_gro_checksum_complete+0x9a/0xa0
[635607.740621] udp6_gro_receive+0x211/0x290
[635607.740624] ipv6_gro_receive+0x1a8/0x390
[635607.740627] dev_gro_receive+0x33e/0x550
[635607.740628] napi_gro_frags+0xa2/0x210
[635607.740635] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740648] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740654] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740657] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740658] net_rx_action+0xe0/0x2e0
[635607.740662] __do_softirq+0xd8/0x2e5
[635607.740666] irq_exit+0xb4/0xc0
[635607.740667] do_IRQ+0x85/0xd0
[635607.740670] common_interrupt+0xf/0xf
[635607.740671] </IRQ>
[635607.740675] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740675] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740701] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740703] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740703] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740704] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740705] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740706] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740709] ? cpuidle_enter_state+0x91/0x2a0
[635607.740712] do_idle+0x1d0/0x240
[635607.740715] cpu_startup_entry+0x5f/0x70
[635607.740719] start_secondary+0x185/0x1a0
[635607.740722] secondary_startup_64+0xa5/0xb0
[635607.740731] p0xe0: hw csum failure
[635607.740745] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 4.18.0-1 #1
[635607.740746] Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017
[635607.740746] Call Trace:
[635607.740747] <IRQ>
[635607.740750] dump_stack+0x5c/0x7b
[635607.740755] __skb_checksum_complete+0xb8/0xd0
[635607.740760] __udp6_lib_rcv+0xa6b/0xa70
[635607.740767] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740770] ? nft_do_chain_inet+0x7a/0xd0 [nf_tables]
[635607.740774] ip6_input_finish+0xc0/0x460
[635607.740776] ip6_input+0x2b/0x90
[635607.740778] ? ip6_rcv_finish+0x110/0x110
[635607.740780] ipv6_rcv+0x2cd/0x4b0
[635607.740783] ? udp6_lib_lookup_skb+0x59/0x80
[635607.740785] __netif_receive_skb_core+0x455/0xb30
[635607.740788] ? ipv6_gro_receive+0x1a8/0x390
[635607.740790] ? netif_receive_skb_internal+0x24/0xb0
[635607.740792] netif_receive_skb_internal+0x24/0xb0
[635607.740793] napi_gro_frags+0x165/0x210
[635607.740796] mlx4_en_process_rx_cq+0xa01/0xb40 [mlx4_en]
[635607.740802] ? mlx4_cq_completion+0x23/0x70 [mlx4_core]
[635607.740807] ? mlx4_eq_int+0x373/0xc80 [mlx4_core]
[635607.740810] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[635607.740811] net_rx_action+0xe0/0x2e0
[635607.740813] __do_softirq+0xd8/0x2e5
[635607.740816] irq_exit+0xb4/0xc0
[635607.740817] do_IRQ+0x85/0xd0
[635607.740820] common_interrupt+0xf/0xf
[635607.740821] </IRQ>
[635607.740823] RIP: 0010:cpuidle_enter_state+0xb4/0x2a0
[635607.740823] Code: 31 ff e8 df a6 ba ff 45 84 f6 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 d8 01 00 00 31 ff e8 13 81 bf ff fb 66 0f 1f 44 00 00 <4c> 29 fb 48 ba cf f7 53 e3 a5 9b c4 20 48 89 d8 48 c1 fb 3f 48 f7
[635607.740848] RSP: 0018:ffffa5c206353ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[635607.740849] RAX: ffff8d72ffd20f00 RBX: 00024214f597c5b0 RCX: 000000000000001f
[635607.740850] RDX: 00024214f597c5b0 RSI: 0000000000020780 RDI: 0000000000000000
[635607.740851] RBP: 0000000000000004 R08: 002542bfbefa99fa R09: 00000000ffffffff
[635607.740852] R10: ffffa5c206353e88 R11: 00000000000000c5 R12: ffffffffaf0aaf78
[635607.740853] R13: ffff8d72ffd297d8 R14: 0000000000000000 R15: 00024214f58c2ed5
[635607.740855] ? cpuidle_enter_state+0x91/0x2a0
[635607.740857] do_idle+0x1d0/0x240
[635607.740859] cpu_startup_entry+0x5f/0x70
[635607.740861] start_secondary+0x185/0x1a0
[635607.740863] secondary_startup_64+0xa5/0xb0
As a matter of fact Dimitris found the issue in the patch and is working on a fix involving csum_block_sub()
Problems comes from trimming an odd number of bytes.
I have applied the commit db4f1be3ca9b0ef7330763d07bf4ace83ad6f913 on top of kernel 4.19.0 and the problem seems gone. The same commit applies to kernel 4.18.x but doesn't work. Thanks!
Fabio
On 30.10.2018 12:04, Andre Tomt wrote: On 30.10.2018 11:58, Andre Tomt wrote: quoted On 27.10.2018 23:41, Andre Tomt wrote: quoted On 26.10.2018 13:45, Andre Tomt wrote: quoted On 25.10.2018 19:38, Eric Dumazet wrote: quoted
On 10/24/2018 12:41 PM, Andre Tomt wrote: quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and also
on 4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux
distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything
shows up.
Just hit it on the simpler server; no VRF, no tunnels, no
nat/conntrack. Only a basic stateless nftables ruleset and a vlan
netdev (unlikely to be the one triggering this I guess; it has only
v4 traffic).
I'm currently testing 4.19 with the recomended commit added, plus
these to sort out some GRO issues (on a hunch, unsure if related):
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=a8305bff685252e80b7c60f4f5e7dd2e63e38218
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=992cba7e276d438ac8b0a8c17b147b37c8c286f7
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ece23711dd956cd5053c9cb03e9fe0668f9c8894
and I *think* it is behaving better now? it's not conclusive as it
could take a while to trip in this environment but some of the test
servers have not shown anything bad in almost 24h.
Sorry, s/some of the/none of the
I think it is fairly safe to say 4.19 + mlx4 + these 4 commits is OK. At
least for my workload. Servers are now 51-61 hours in, no splats. I also
added ntp pool traffic to one of them to make things a little more exciting.
Not sure what is needed for 4.18, I dont have the mental bandwidth to
test that right now. Also no idea about the similar looking mlx5 splats
reported elsewhere.
On 31.10.2018 05:08, Andre Tomt wrote: On 30.10.2018 12:04, Andre Tomt wrote: quoted On 30.10.2018 11:58, Andre Tomt wrote: quoted On 27.10.2018 23:41, Andre Tomt wrote: quoted On 26.10.2018 13:45, Andre Tomt wrote: quoted On 25.10.2018 19:38, Eric Dumazet wrote: quoted
On 10/24/2018 12:41 PM, Andre Tomt wrote: quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and
also on 4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux
distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything
shows up.
Just hit it on the simpler server; no VRF, no tunnels, no
nat/conntrack. Only a basic stateless nftables ruleset and a vlan
netdev (unlikely to be the one triggering this I guess; it has only
v4 traffic).
I'm currently testing 4.19 with the recomended commit added, plus
these to sort out some GRO issues (on a hunch, unsure if related):
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=a8305bff685252e80b7c60f4f5e7dd2e63e38218
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=992cba7e276d438ac8b0a8c17b147b37c8c286f7
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ece23711dd956cd5053c9cb03e9fe0668f9c8894
and I *think* it is behaving better now? it's not conclusive as it
could take a while to trip in this environment but some of the test
servers have not shown anything bad in almost 24h.
Sorry, s/some of the/none of the
I think it is fairly safe to say 4.19 + mlx4 + these 4 commits is OK. At
least for my workload. Servers are now 51-61 hours in, no splats. I also
added ntp pool traffic to one of them to make things a little more
exciting.
Not sure what is needed for 4.18, I dont have the mental bandwidth to
test that right now. Also no idea about the similar looking mlx5 splats
reported elsewhere.
As expected conntrack/nat + vlan + forwarding still splats.
sch_cake, IFB and VRF was removed from this setup.
Here is a conntrack splat without IFB/VRF/Cake inteference: [34458.506346] wanib: hw csum failure
[34458.506371] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.0-1 #1
[34458.506374] Hardware name: Supermicro Super Server/X10SDV-4C-TLN2F, BIOS 2.0 06/13/2018
[34458.506377] Call Trace:
[34458.506381] <IRQ>
[34458.506388] dump_stack+0x5c/0x80
[34458.506392] __skb_checksum_complete+0xac/0xc0
[34458.506402] icmp_error+0x1c8/0x1f0 [nf_conntrack]
[34458.506406] ? skb_copy_bits+0x13d/0x220
[34458.506411] nf_conntrack_in+0xd8/0x390 [nf_conntrack]
[34458.506416] ? ___pskb_trim+0x192/0x330
[34458.506421] nf_hook_slow+0x43/0xc0
[34458.506426] ip_rcv+0x90/0xb0
[34458.506430] ? ip_rcv_finish_core.isra.0+0x310/0x310
[34458.506435] __netif_receive_skb_one_core+0x42/0x50
[34458.506438] netif_receive_skb_internal+0x24/0xb0
[34458.506441] napi_gro_frags+0x177/0x210
[34458.506446] mlx4_en_process_rx_cq+0x8df/0xb50 [mlx4_en]
[34458.506459] ? mlx4_eq_int+0x38f/0xcb0 [mlx4_core]
[34458.506463] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[34458.506466] net_rx_action+0xe1/0x2c0
[34458.506469] __do_softirq+0xe7/0x2d3
[34458.506475] irq_exit+0x96/0xd0
[34458.506478] do_IRQ+0x85/0xd0
[34458.506483] common_interrupt+0xf/0xf
[34458.506486] </IRQ>
[34458.506491] RIP: 0010:cpuidle_enter_state+0xb9/0x320
[34458.506495] Code: e8 3c 16 bc ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 5e fb c0 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3
[34458.506497] RSP: 0018:ffff978d41943ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdb
[34458.506500] RAX: ffff8d8f6fa60fc0 RBX: 00001f56ff07af28 RCX: 000000000000001f
[34458.506501] RDX: 00001f56ff07af28 RSI: 000000003a2e90d6 RDI: 0000000000000000
[34458.506503] RBP: ffff8d8f6fa698c0 R08: 0000000000000002 R09: 0000000000020840
[34458.506504] R10: 0004ea58f2899595 R11: ffff8d8f6fa601e8 R12: 0000000000000001
[34458.506505] R13: ffffffff8a0ac638 R14: 0000000000000001 R15: 0000000000000000
[34458.506509] ? cpuidle_enter_state+0x94/0x320
[34458.506512] do_idle+0x1e4/0x220
[34458.506515] cpu_startup_entry+0x5f/0x70
[34458.506519] start_secondary+0x185/0x1a0
[34458.506521] secondary_startup_64+0xa4/0xb0
Stateless filtered non-forwarding host still looks like it has been
fixed (the udp6_gro_* splats are still all gone). Also seems fine when
moving the traffic over a vlan device. These fixes went into 4.19.1-rc1
(checksum_complete + unlink gro packets on overflow fixes)