Re: 4.6.3, pppoe + shaper workload, skb_panic / skb_push / ppp_start_xmit
From: Denys Fedoryshchenko <hidden>
Date: 2016-07-28 11:28:26
On 2016-07-28 14:09, Guillaume Nault wrote:
On Tue, Jul 12, 2016 at 10:31:18AM -0700, Cong Wang wrote:quoted
On Mon, Jul 11, 2016 at 12:45 PM, [off-list ref] wrote:quoted
Hi On latest kernel i noticed kernel panic happening 1-2 times per day. It is also happening on older kernel (at least 4.5.3)....quoted
[42916.426463] Call Trace: [42916.426658] <IRQ> [42916.426719] [<ffffffff81843786>] skb_push+0x36/0x37 [42916.427111] [<ffffffffa00e8ce5>] ppp_start_xmit+0x10f/0x150 [ppp_generic] [42916.427314] [<ffffffff81853467>] dev_hard_start_xmit+0x25a/0x2d3 [42916.427516] [<ffffffff818530f2>] ? validate_xmit_skb.isra.107.part.108+0x11d/0x238 [42916.427858] [<ffffffff8186dee3>] sch_direct_xmit+0x89/0x1b5 [42916.428060] [<ffffffff8186e142>] __qdisc_run+0x133/0x170 [42916.428261] [<ffffffff81850034>] net_tx_action+0xe3/0x148 [42916.428462] [<ffffffff810c401a>] __do_softirq+0xb9/0x1a9 [42916.428663] [<ffffffff810c4251>] irq_exit+0x37/0x7c [42916.428862] [<ffffffff8102b8f7>] smp_apic_timer_interrupt+0x3d/0x48 [42916.429063] [<ffffffff818cb15c>] apic_timer_interrupt+0x7c/0x90Interesting, we call a skb_cow_head() before skb_push() in ppp_start_xmit(), I have no idea why this could happen.The skb is corrupted: head is at ffff8800b0bf2800 while data is at ffa00500b0bf284c. Figuring out how this corruption happened is going to be hard without a way to reproduce the problem. Denys, can you confirm you're using a vanilla kernel? Also I guess the ppp devices and tc settings are handled by accel-ppp. If so, can you share more info about your setup (accel-ppp.conf, radius attributes, iptables...) so that I can try to reproduce it on my machines?
I have slight modification from vanilla:
--- linux/net/sched/sch_htb.c 2016-06-08 01:23:53.000000000 +0000
+++ linux-new/net/sched/sch_htb.c 2016-06-21 14:03:08.398486593 +0000@@ -1495,10 +1495,10 @@ cl->common.classid); cl->quantum = 1000; } - if (!hopt->quantum && cl->quantum > 200000) { + if (!hopt->quantum && cl->quantum > 2000000) { pr_warn("HTB: quantum of class %X is big. Consider r2q change.\n", cl->common.classid); - cl->quantum = 200000; + cl->quantum = 2000000; } if (hopt->quantum) cl->quantum = hopt->quantum;
But i guess it should not be reason of crash (it is related to another system, without it i was unable to shape over 7Gbps, maybe with latest kernel i will not need this patch). I'm trying to make reproducible conditions of crash, because right now it happens only on some servers in large networks (completely different ISPs, so i excluded possible hardware fault of specific server). It is complex config, i have accel-ppp, plus my own "shaping daemon" that apply several shapers on ppp interfaces. Wost thing it happens only on live customers, i am unable to reproduce same on stress tests. Also until recent kernel i was getting different panic messages (but all related to ppp). I think also at least one reason of crash also was fixed by "ppp: defer netns reference release for ppp channel" in 4.7.0 (maybe thats why i am getting less crashes recently). I tried also various kernel debug options that doesn't cause major performance degradation (locks checking, freed memory poisoning and etc), without any luck yet. Is it useful if i will post panics that at least occurs twice? (I will post below example, got recently) Sure if i will be able to reproducible conditions i will send them immediately. <server19> [ 5449.900988] general protection fault: 0000 [#1] SMP <server19> [ 5449.901263] Modules linked in: <server19> cls_fw <server19> act_police <server19> cls_u32 <server19> sch_ingress <server19> sch_sfq <server19> sch_htb <server19> pppoe <server19> pppox <server19> ppp_generic <server19> slhc <server19> netconsole <server19> configfs <server19> xt_nat <server19> ts_bm <server19> xt_string <server19> xt_connmark <server19> xt_TCPMSS <server19> xt_tcpudp <server19> xt_mark <server19> iptable_filter <server19> iptable_nat <server19> nf_conntrack_ipv4 <server19> nf_defrag_ipv4 <server19> nf_nat_ipv4 <server19> nf_nat <server19> nf_conntrack <server19> iptable_mangle <server19> ip_tables <server19> x_tables <server19> 8021q <server19> garp <server19> mrp <server19> stp <server19> llc <server19> ixgbe <server19> dca <server19> <server19> [ 5449.904989] CPU: 1 PID: 6359 Comm: ip Not tainted 4.7.0-build-0109 #2 <server19> [ 5449.905255] Hardware name: Supermicro X10SLM+-LN4F/X10SLM+-LN4F, BIOS 3.0 04/24/2015 <server19> [ 5449.905712] task: ffff8803eef40000 ti: ffff8803fd754000 task.ti: ffff8803fd754000 <server19> [ 5449.906168] RIP: 0010:[<ffffffff818a994d>] <server19> [<ffffffff818a994d>] inet_fill_ifaddr+0x5a/0x264 <server19> [ 5449.906710] RSP: 0018:ffff8803fd757b98 EFLAGS: 00010286 <server19> [ 5449.906976] RAX: ffff8803ef65cb90 RBX: ffff8803f7d2cd00 RCX: 0000000000000000 <server19> [ 5449.907248] RDX: 0000000800000002 RSI: ffff8803ef65cb90 RDI: ffff8803ef65cba8 <server19> [ 5449.907519] RBP: ffff8803fd757be0 R08: 0000000000000008 R09: 0000000000000002 <server19> [ 5449.907792] R10: ffa005040269f480 R11: ffffffff820a1c00 R12: ffa005040269f480 <server19> [ 5449.908067] R13: ffff8803ef65cb90 R14: 0000000000000000 R15: ffff8803f7d2cd00 <server19> [ 5449.908339] FS: 00007f660674d700(0000) GS:ffff88041fc40000(0000) knlGS:0000000000000000 <server19> [ 5449.908796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <server19> [ 5449.909067] CR2: 00000000008b9018 CR3: 00000003f2a11000 CR4: 00000000001406e0 <server19> [ 5449.909339] Stack: <server19> [ 5449.909598] 0163a8c0869711ac <server19> 0000008000000000 <server19> ffffffffffffffff <server19> 0003e1d50003e1d5 <server19> <server19> [ 5449.910329] ffff8800d54c0ac8 <server19> ffff8803f0d90000 <server19> 0000000000000005 <server19> 0000000000000000 <server19> <server19> [ 5449.911066] ffff8803f7d2cd00 <server19> ffff8803fd757c40 <server19> ffffffff818a9f73 <server19> ffffffff820a1c00 <server19> <server19> [ 5449.911803] Call Trace: <server19> [ 5449.912061] [<ffffffff818a9f73>] inet_dump_ifaddr+0xfb/0x185 <server19> [ 5449.912332] [<ffffffff8185de4b>] rtnl_dump_all+0xa9/0xc2 <server19> [ 5449.912601] [<ffffffff818756d8>] netlink_dump+0xf0/0x25c <server19> [ 5449.912873] [<ffffffff818759ed>] netlink_recvmsg+0x1a9/0x2d3 <server19> [ 5449.913142] [<ffffffff81838412>] sock_recvmsg+0x14/0x16 <server19> [ 5449.913407] [<ffffffff8183a743>] ___sys_recvmsg+0xea/0x1a1 <server19> [ 5449.913675] [<ffffffff811658e6>] ? alloc_pages_vma+0x167/0x1a0 <server19> [ 5449.913945] [<ffffffff81159a8b>] ? page_add_new_anon_rmap+0xb4/0xbd <server19> [ 5449.914212] [<ffffffff8113b0d0>] ? lru_cache_add_active_or_unevictable+0x31/0x9d <server19> [ 5449.914664] [<ffffffff81151762>] ? handle_mm_fault+0x632/0x112d <server19> [ 5449.914940] [<ffffffff811550fe>] ? vma_merge+0x27e/0x2b1 <server19> [ 5449.915208] [<ffffffff8183b4db>] __sys_recvmsg+0x3d/0x5e <server19> [ 5449.915478] [<ffffffff8183b4db>] ? __sys_recvmsg+0x3d/0x5e <server19> [ 5449.915747] [<ffffffff8183b509>] SyS_recvmsg+0xd/0x17 <server19> [ 5449.916017] [<ffffffff818cb85f>] entry_SYSCALL_64_fastpath+0x17/0x93 <server19> [ 5449.916287] Code: <server19> e5 <server19> 41 <server19> 57 <server19> 41 <server19> 56 <server19> 41 <server19> 55 <server19> 41 <server19> 54 <server19> 49 <server19> 89 <server19> f4 <server19> 53 <server19> 89 <server19> c6 <server19> 48 <server19> 89 <server19> fb <server19> 48 <server19> 83 <server19> ec <server19> 20 <server19> e8 <server19> be <server19> b0 <server19> fc <server19> ff <server19> 48 <server19> 85 <server19> c0 <server19> 49 <server19> 89 <server19> c5 <server19> 0f <server19> 84 <server19> f4 <server19> 01 <server19> 00 <server19> 00 <server19> c6 <server19> 40 <server19> 10 <server19> 02 <server19> <server19> 8a <server19> 44 <server19> 24 <server19> 41 <server19> 41 <server19> 83 <server19> ce <server19> ff <server19> 45 <server19> 89 <server19> f7 <server19> 41 <server19> 88 <server19> 45 <server19> 11 <server19> 41 <server19> 8b <server19> 44 <server19> 24 <server19> 44 <server19> <server19> [ 5449.921684] RIP <server19> [<ffffffff818a994d>] inet_fill_ifaddr+0x5a/0x264 <server19> [ 5449.922028] RSP <ffff8803fd757b98> <server19> [ 5449.922547] ---[ end trace 18580d58f51e3038 ]--- <server19> [ 5449.923705] Kernel panic - not syncing: Fatal exception <server19> [ 5449.923979] Kernel Offset: disabled <server19> [ 5449.925873] Rebooting in 5 seconds.. <server19> [43221.432450] general protection fault: 0000 [#1] SMP <server19> [43221.432656] Modules linked in: <server19> intel_ips <server19> intel_smartconnect <server19> intel_rst <server19> cls_fw <server19> act_police <server19> cls_u32 <server19> sch_ingress <server19> sch_sfq <server19> sch_htb <server19> pppoe <server19> pppox <server19> ppp_generic <server19> slhc <server19> netconsole <server19> configfs <server19> xt_nat <server19> ts_bm <server19> xt_string <server19> xt_connmark <server19> xt_TCPMSS <server19> xt_tcpudp <server19> xt_mark <server19> iptable_filter <server19> iptable_nat <server19> nf_conntrack_ipv4 <server19> nf_defrag_ipv4 <server19> nf_nat_ipv4 <server19> nf_nat <server19> nf_conntrack <server19> iptable_mangle <server19> ip_tables <server19> x_tables <server19> 8021q <server19> garp <server19> mrp <server19> stp <server19> llc <server19> ixgbe <server19> dca <server19> <server19> [43221.433815] CPU: 3 PID: 29196 Comm: accel-cmd Not tainted 4.7.0-build-0110 #2 <server19> [43221.434024] Hardware name: Supermicro X10SLM+-LN4F/X10SLM+-LN4F, BIOS 3.0 04/24/2015 <server19> [43221.434414] task: ffff8803dcc39780 ti: ffff8800cdb18000 task.ti: ffff8800cdb18000 <server19> [43221.434805] RIP: 0010:[<ffffffff818a7fd0>] <server19> [<ffffffff818a7fd0>] inet_fill_ifaddr+0x5a/0x264 <server19> [43221.435202] RSP: 0018:ffff8800cdb1bb98 EFLAGS: 00010282 <server19> [43221.435406] RAX: ffff8803fe89efb0 RBX: ffff8803de661500 RCX: 0000000000000000 <server19> [43221.435616] RDX: 0000000800000002 RSI: ffff8803fe89efb0 RDI: ffff8803fe89efc8 <server19> [43221.435823] RBP: ffff8800cdb1bbe0 R08: 0000000000000008 R09: 0000000000000002 <server19> [43221.436030] R10: ffa0050402880f80 R11: ffffffff820a1680 R12: ffa0050402880f80 <server19> [43221.436234] R13: ffff8803fe89efb0 R14: 0000000000000000 R15: ffff8803de661500 <server19> [43221.436436] FS: 00007f25a2539700(0000) GS:ffff88041fcc0000(0000) knlGS:0000000000000000 <server19> [43221.436821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <server19> [43221.437023] CR2: 000000000060f000 CR3: 00000000cd2e8000 CR4: 00000000001406e0 <server19> [43221.437227] Stack: <server19> [43221.437419] 0163a8c0818411ac <server19> 0000008000000000 <server19> ffffffffffffffff <server19> 003a44db003a44db <server19> <server19> [43221.437827] ffff8803fe5992c8 <server19> ffff8803f5b04000 <server19> 0000000000000003 <server19> 0000000000000000 <server19> <server19> [43221.438230] ffff8803de661500 <server19> ffff8800cdb1bc40 <server19> ffffffff818a85f6 <server19> ffffffff820a1680 <server19> <server19> [43221.438636] Call Trace: <server19> [43221.438834] [<ffffffff818a85f6>] inet_dump_ifaddr+0xfb/0x185 <server19> [43221.439035] [<ffffffff8185c4ce>] rtnl_dump_all+0xa9/0xc2 <server19> [43221.439241] [<ffffffff81873d5b>] netlink_dump+0xf0/0x25c <server19> [43221.439441] [<ffffffff81874070>] netlink_recvmsg+0x1a9/0x2d3 <server19> [43221.439641] [<ffffffff81836a95>] sock_recvmsg+0x14/0x16 <server19> [43221.439841] [<ffffffff81838dc6>] ___sys_recvmsg+0xea/0x1a1 <server19> [43221.440043] [<ffffffff8116765f>] ? alloc_pages_vma+0x167/0x1a0 <server19> [43221.440247] [<ffffffff8115b804>] ? page_add_new_anon_rmap+0xb4/0xbd <server19> [43221.440449] [<ffffffff8113ce49>] ? lru_cache_add_active_or_unevictable+0x31/0x9d <server19> [43221.440837] [<ffffffff811534db>] ? handle_mm_fault+0x632/0x112d <server19> [43221.441038] [<ffffffff81839636>] ? SyS_sendto+0xef/0x120 <server19> [43221.441241] [<ffffffff81839b5e>] __sys_recvmsg+0x3d/0x5e <server19> [43221.441443] [<ffffffff81839b5e>] ? __sys_recvmsg+0x3d/0x5e <server19> [43221.441644] [<ffffffff81839b8c>] SyS_recvmsg+0xd/0x17 <server19> [43221.441849] [<ffffffff818c9edf>] entry_SYSCALL_64_fastpath+0x17/0x93 <server19> [43221.442055] Code: <server19> e5 <server19> 41 <server19> 57 <server19> 41 <server19> 56 <server19> 41 <server19> 55 <server19> 41 <server19> 54 <server19> 49 <server19> 89 <server19> f4 <server19> 53 <server19> 89 <server19> c6 <server19> 48 <server19> 89 <server19> fb <server19> 48 <server19> 83 <server19> ec <server19> 20 <server19> e8 <server19> be <server19> b0 <server19> fc <server19> ff <server19> 48 <server19> 85 <server19> c0 <server19> 49 <server19> 89 <server19> c5 <server19> 0f <server19> 84 <server19> f4 <server19> 01 <server19> 00 <server19> 00 <server19> c6 <server19> 40 <server19> 10 <server19> 02 <server19> <server19> 8a <server19> 44 <server19> 24 <server19> 41 <server19> 41 <server19> 83 <server19> ce <server19> ff <server19> 45 <server19> 89 <server19> f7 <server19> 41 <server19> 88 <server19> 45 <server19> 11 <server19> 41 <server19> 8b <server19> 44 <server19> 24 <server19> 44 <server19> <server19> [43221.442945] RIP <server19> [<ffffffff818a7fd0>] inet_fill_ifaddr+0x5a/0x264 <server19> [43221.443151] RSP <ffff8800cdb1bb98> <server19> [43221.445125] ---[ end trace 99273d413e56a193 ]--- <server19> [43221.446262] Kernel panic - not syncing: Fatal exception <server19> [43221.446536] Kernel Offset: disabled <server19> [43221.448446] Rebooting in 5 seconds.. Jul 27 23:41:44 10.0.253.19 Jul 27 23:41:44 10.0.253.19 [43226.451328] ACPI MEMORY or I/O RESET_REG.