Re: Fw: [Bug 201423] New: eth0: hw csum failure
From: Andre Tomt <hidden>
Date: 2018-11-04 14:56:50
On 31.10.2018 05:08, Andre Tomt wrote:
On 30.10.2018 12:04, Andre Tomt wrote:quoted
On 30.10.2018 11:58, Andre Tomt wrote:quoted
On 27.10.2018 23:41, Andre Tomt wrote:quoted
On 26.10.2018 13:45, Andre Tomt wrote:quoted
On 25.10.2018 19:38, Eric Dumazet wrote:quoted
On 10/24/2018 12:41 PM, Andre Tomt wrote:quoted
It eventually showed up again with mlx4, on 4.18.16 + fix and also on 4.19. I still do not have a useful packet capture. It is running a torrent client serving up various linux distributions.Have you also applied this fix ? https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913No. I've applied it now to 4.19 and will report back if anything shows up.Just hit it on the simpler server; no VRF, no tunnels, no nat/conntrack. Only a basic stateless nftables ruleset and a vlan netdev (unlikely to be the one triggering this I guess; it has only v4 traffic).I'm currently testing 4.19 with the recomended commit added, plus these to sort out some GRO issues (on a hunch, unsure if related): https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=a8305bff685252e80b7c60f4f5e7dd2e63e38218 https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=992cba7e276d438ac8b0a8c17b147b37c8c286f7 https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ece23711dd956cd5053c9cb03e9fe0668f9c8894 and I *think* it is behaving better now? it's not conclusive as it could take a while to trip in this environment but some of the test servers have not shown anything bad in almost 24h.Sorry, s/some of the/none of theI think it is fairly safe to say 4.19 + mlx4 + these 4 commits is OK. At least for my workload. Servers are now 51-61 hours in, no splats. I also added ntp pool traffic to one of them to make things a little more exciting. Not sure what is needed for 4.18, I dont have the mental bandwidth to test that right now. Also no idea about the similar looking mlx5 splats reported elsewhere.
As expected conntrack/nat + vlan + forwarding still splats. sch_cake, IFB and VRF was removed from this setup. Here is a conntrack splat without IFB/VRF/Cake inteference:
[34458.506346] wanib: hw csum failure [34458.506371] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.0-1 #1 [34458.506374] Hardware name: Supermicro Super Server/X10SDV-4C-TLN2F, BIOS 2.0 06/13/2018 [34458.506377] Call Trace: [34458.506381] <IRQ> [34458.506388] dump_stack+0x5c/0x80 [34458.506392] __skb_checksum_complete+0xac/0xc0 [34458.506402] icmp_error+0x1c8/0x1f0 [nf_conntrack] [34458.506406] ? skb_copy_bits+0x13d/0x220 [34458.506411] nf_conntrack_in+0xd8/0x390 [nf_conntrack] [34458.506416] ? ___pskb_trim+0x192/0x330 [34458.506421] nf_hook_slow+0x43/0xc0 [34458.506426] ip_rcv+0x90/0xb0 [34458.506430] ? ip_rcv_finish_core.isra.0+0x310/0x310 [34458.506435] __netif_receive_skb_one_core+0x42/0x50 [34458.506438] netif_receive_skb_internal+0x24/0xb0 [34458.506441] napi_gro_frags+0x177/0x210 [34458.506446] mlx4_en_process_rx_cq+0x8df/0xb50 [mlx4_en] [34458.506459] ? mlx4_eq_int+0x38f/0xcb0 [mlx4_core] [34458.506463] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en] [34458.506466] net_rx_action+0xe1/0x2c0 [34458.506469] __do_softirq+0xe7/0x2d3 [34458.506475] irq_exit+0x96/0xd0 [34458.506478] do_IRQ+0x85/0xd0 [34458.506483] common_interrupt+0xf/0xf [34458.506486] </IRQ> [34458.506491] RIP: 0010:cpuidle_enter_state+0xb9/0x320 [34458.506495] Code: e8 3c 16 bc ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 5e fb c0 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3 [34458.506497] RSP: 0018:ffff978d41943ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdb [34458.506500] RAX: ffff8d8f6fa60fc0 RBX: 00001f56ff07af28 RCX: 000000000000001f [34458.506501] RDX: 00001f56ff07af28 RSI: 000000003a2e90d6 RDI: 0000000000000000 [34458.506503] RBP: ffff8d8f6fa698c0 R08: 0000000000000002 R09: 0000000000020840 [34458.506504] R10: 0004ea58f2899595 R11: ffff8d8f6fa601e8 R12: 0000000000000001 [34458.506505] R13: ffffffff8a0ac638 R14: 0000000000000001 R15: 0000000000000000 [34458.506509] ? cpuidle_enter_state+0x94/0x320 [34458.506512] do_idle+0x1e4/0x220 [34458.506515] cpu_startup_entry+0x5f/0x70 [34458.506519] start_secondary+0x185/0x1a0 [34458.506521] secondary_startup_64+0xa4/0xb0
Stateless filtered non-forwarding host still looks like it has been fixed (the udp6_gro_* splats are still all gone). Also seems fine when moving the traffic over a vlan device. These fixes went into 4.19.1-rc1 (checksum_complete + unlink gro packets on overflow fixes)