Re: [Bug 205427] New: Qdisc running seqcount may cause performance issues
From: Stephen Hemminger <stephen@networkplumber.org>
Date: 2019-11-05 05:36:39
On Tue, 05 Nov 2019 05:27:57 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=205427 Bug ID: 205427 Summary: Qdisc running seqcount may cause performance issues Product: Networking Version: 2.5 Kernel Version: 4.9 Hardware: All OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: IPV4 Assignee: stephen@networkplumber.org Reporter: yellowriver2010@hotmail.com Regression: No [ 1261.949409] NMI watchdog: BUG: soft lockup - CPU#80 stuck for 23s! [tc:12076] [ 1261.956565] Modules linked in: kpatch_D871570(O) kpatch(OE) intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel lrw glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr iohub_sriov(O) mei_me ioatdma i2c_i801 lpc_ich ipmi_si mei dca shpchp mfd_core i2c_smbus wmi ipmi_msghandler acpi_power_meter acpi_pad ip_tables ext4 jbd2 mbcache virtio_net virtio_blk i2c_algo_bit drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci virtio_pci drm virtio_ring(E) libahci virtio libata nvme nvme_core i2c_core [ 1262.014116] CPU: 80 PID: 12076 Comm: tc Tainted: G OE K 4.9.151-015.ali3000.alios7.x86_64 #1 [ 1262.023437] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.23.34 02/14/2019 [ 1262.032154] task: ffff887f48dc8000 task.stack: ffffc90037128000 [ 1262.038091] RIP: 0010:[<ffffffff8160fbee>] [<ffffffff8160fbee>] __gnet_stats_copy_basic+0x8e/0x90 [ 1262.047093] RSP: 0018:ffffc9003712b930 EFLAGS: 00000202 [ 1262.052418] RAX: 000000000011845b RBX: ffffc9003712b9a0 RCX: ffff887f4e39b2b0 [ 1262.059569] RDX: 0000000000000000 RSI: ffffc9003712b93c RDI: ffff887f4e3950bc [ 1262.066871] RBP: ffffc9003712b968 R08: 0000000000000004 R09: ffff887f5252bf7c [ 1262.074166] R10: ffff887f58718100 R11: 0000000000000008 R12: 0000000000000000 [ 1262.081465] R13: ffff887f5252bf4c R14: ffff887f4e39b200 R15: ffff887f4e39b2b0 [ 1262.088765] FS: 00007f0a974cd740(0000) GS:ffff887f7f000000(0000) knlGS:0000000000000000 [ 1262.097157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1262.103067] CR2: 00000000006474c0 CR3: 0000007f50006000 CR4: 00000000007606f0 [ 1262.110368] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1262.117665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1262.124962] PKRU: 55555554 [ 1262.127822] Stack: [ 1262.129985] ffffffff8160fce8 0000000000000009 0000000000000000 ffffffff8160ff24 [ 1262.137779] ffff887f58718100 d0e0e5e734dba31b ffff887f58718100 ffffc9003712ba30 [ 1262.145574] ffffffff8164d383 0000000000000009 0000000000000000 0000000000000000 [ 1262.153363] Call Trace: [ 1262.155968] [<ffffffff8160fce8>] ? gnet_stats_copy_basic+0x38/0xe0 [ 1262.162397] [<ffffffff8160ff24>] ? gnet_stats_start_copy_compat+0x94/0x100 [ 1262.169527] [<ffffffff8164d383>] tc_fill_qdisc+0x283/0x400 [ 1262.175258] [<ffffffff8164d635>] tc_dump_qdisc_root+0x135/0x1a0 [ 1262.181430] [<ffffffff8164d744>] tc_dump_qdisc+0xa4/0x150 [ 1262.187077] [<ffffffff81654eb1>] netlink_dump+0x231/0x2c0 [ 1262.192721] [<ffffffff81655d68>] __netlink_dump_start+0x168/0x1a0 [ 1262.199067] [<ffffffff8164d6a0>] ? tc_dump_qdisc_root+0x1a0/0x1a0 [ 1262.205413] [<ffffffff81634093>] rtnetlink_rcv_msg+0x1c3/0x230 [ 1262.211494] [<ffffffff8164d6a0>] ? tc_dump_qdisc_root+0x1a0/0x1a0 [ 1262.217835] [<ffffffff81633ed0>] ? rtnl_newlink+0x860/0x860 [ 1262.223651] [<ffffffff81657b64>] netlink_rcv_skb+0xa4/0xc0 [ 1262.229387] [<ffffffff8162e785>] rtnetlink_rcv+0x15/0x20 [ 1262.234942] [<ffffffff8165754c>] netlink_unicast+0x18c/0x220 [ 1262.240844] [<ffffffff8165793b>] netlink_sendmsg+0x35b/0x3b0 [ 1262.246758] [<ffffffff815fca58>] sock_sendmsg+0x38/0x50 [ 1262.252233] [<ffffffff815fd4ad>] ___sys_sendmsg+0x29d/0x2b0 [ 1262.258060] [<ffffffff811ed21f>] ? do_wp_page+0x39f/0x850 [ 1262.263708] [<ffffffff811f04ea>] ? handle_mm_fault+0x6da/0xd50 [ 1262.269791] [<ffffffff815fde04>] __sys_sendmsg+0x54/0x90 [ 1262.275359] [<ffffffff815fde52>] SyS_sendmsg+0x12/0x20 [ 1262.280744] [<ffffffff81003c04>] do_syscall_64+0x74/0x180 [ 1262.286387] [<ffffffff81741c8e>] entry_SYSCALL_64_after_swapgs+0x58/0xc6 [ 1262.293337] Code: 85 ff 74 18 8b 07 a8 01 75 1f 48 8b 11 48 89 16 8b 51 08 89 56 08 39 07 75 ea f3 c3 48 8b 01 48 89 06 8b 41 08 89 46 08 c3 f3 90 <eb> d7 0f 1f 44 00 00 55 48 89 e5 41 55 41 54 53 8b 47 1c 48 89 [ 1262.314295] Kernel panic - not syncing: softlockup: hung tasks [ 1262.320291] CPU: 80 PID: 12076 Comm: tc Tainted: G OELK 4.9.151-015.ali3000.alios7.x86_64 #1 [ 1262.329904] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.23.34 02/14/2019 [ 1262.338904] ffff887f7f003e50 ffffffff8139f342 0000000000000000 ffffffff81a4f7f1 [ 1262.346674] ffff887f7f003ed8 ffffffff811b2245 ffffc90000000008 ffff887f7f003ee8 [ 1262.354456] ffff887f7f003e80 d0e0e5e734dba31b ffff887f7f003ea7 0000000000000000 [ 1262.362249] Call Trace: [ 1262.364846] <IRQ> [ 1262.366785] [<ffffffff8139f342>] dump_stack+0x63/0x81 [ 1262.372095] [<ffffffff811b2245>] panic+0xf8/0x244 [ 1262.377049] [<ffffffff81158296>] watchdog_timer_fn+0x226/0x230 [ 1262.383126] [<ffffffff81158070>] ? watchdog_park_threads+0x70/0x70 [ 1262.389555] [<ffffffff811085e3>] __hrtimer_run_queues+0xf3/0x270 [ 1262.395807] [<ffffffff81108d8a>] hrtimer_interrupt+0x9a/0x180 [ 1262.401796] [<ffffffff810580f8>] local_apic_timer_interrupt+0x38/0x60 [ 1262.408491] [<ffffffff81745405>] smp_apic_timer_interrupt+0x45/0x60 [ 1262.415006] [<ffffffff81743b90>] apic_timer_interrupt+0xa0/0xb0 [ 1262.421177] <EOI> [ 1262.423118] [<ffffffff8160fbee>] ? __gnet_stats_copy_basic+0x8e/0x90 [ 1262.429871] [<ffffffff8160fce8>] ? gnet_stats_copy_basic+0x38/0xe0 [ 1262.436301] [<ffffffff8160ff24>] ? gnet_stats_start_copy_compat+0x94/0x100 [ 1262.443428] [<ffffffff8164d383>] tc_fill_qdisc+0x283/0x400 [ 1262.449164] [<ffffffff8164d635>] tc_dump_qdisc_root+0x135/0x1a0 [ 1262.455329] [<ffffffff8164d744>] tc_dump_qdisc+0xa4/0x150 [ 1262.460978] [<ffffffff81654eb1>] netlink_dump+0x231/0x2c0 [ 1262.466620] [<ffffffff81655d68>] __netlink_dump_start+0x168/0x1a0 [ 1262.472965] [<ffffffff8164d6a0>] ? tc_dump_qdisc_root+0x1a0/0x1a0 [ 1262.479311] [<ffffffff81634093>] rtnetlink_rcv_msg+0x1c3/0x230 [ 1262.485394] [<ffffffff8164d6a0>] ? tc_dump_qdisc_root+0x1a0/0x1a0 [ 1262.491737] [<ffffffff81633ed0>] ? rtnl_newlink+0x860/0x860 [ 1262.497560] [<ffffffff81657b64>] netlink_rcv_skb+0xa4/0xc0 [ 1262.503295] [<ffffffff8162e785>] rtnetlink_rcv+0x15/0x20 [ 1262.508854] [<ffffffff8165754c>] netlink_unicast+0x18c/0x220 [ 1262.514760] [<ffffffff8165793b>] netlink_sendmsg+0x35b/0x3b0 [ 1262.520666] [<ffffffff815fca58>] sock_sendmsg+0x38/0x50 [ 1262.526140] [<ffffffff815fd4ad>] ___sys_sendmsg+0x29d/0x2b0 [ 1262.531960] [<ffffffff811ed21f>] ? do_wp_page+0x39f/0x850 [ 1262.537607] [<ffffffff811f04ea>] ? handle_mm_fault+0x6da/0xd50 [ 1262.543691] [<ffffffff815fde04>] __sys_sendmsg+0x54/0x90 [ 1262.549360] [<ffffffff815fde52>] SyS_sendmsg+0x12/0x20 [ 1262.554750] [<ffffffff81003c04>] do_syscall_64+0x74/0x180 [ 1262.560402] [<ffffffff81741c8e>] entry_SYSCALL_64_after_swapgs+0x58/0xc6 CPU 80 CPU 1 execute : tc qd show dev bond0 netperf rtnetlink_rcv (acquired rtnl_mutex) ... ... __dev_xmit_skb __gnet_stats_copy_basic If netperf on CPU 1 runs for a long time, the tc program may repeatedly acquire seqcount for a long time, resulting in softlockup. At this time, the tc program holds the large lock of rtnl_mutex, which also affects other programs in the system.