Thread (2 messages) flat view 2 messages, 2 authors, 2018-06-08

INFO: task hung in netdev_run_todo

From: syzbot <hidden>
Date: 2018-02-12 10:03:04

Hello,

syzbot hit the following crash on upstream commit
581e400ff935d34d95811258586128bf11baef15 (Wed Feb 7 22:29:34 2018 +0000)
Merge tag 'modules-for-v4.16' of  
git://git.kernel.org/pub/scm/linux/kernel/git/jeyu/linux

Unfortunately, I don't have any reproducer for this crash yet.
Raw console output is attached.
compiler: gcc (GCC) 7.1.1 20170620
.config is attached.
user-space arch: i386

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+8cd13ab48843d67c58f0@syzkaller.appspotmail.com
It will help syzbot understand when the bug is fixed. See footer for  
details.
If you forward the report, please keep this part and the footer.

INFO: task kworker/u4:1:21 blocked for more than 120 seconds.
       Not tainted 4.15.0+ #213
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u4:1    D12208    21      2 0x80000000
Workqueue: netns cleanup_net
Call Trace:
  context_switch kernel/sched/core.c:2857 [inline]
  __schedule+0x8e2/0x2040 kernel/sched/core.c:3435
  schedule+0xf5/0x430 kernel/sched/core.c:3494
  schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3552
  __mutex_lock_common kernel/locking/mutex.c:833 [inline]
  __mutex_lock+0xaad/0x1a80 kernel/locking/mutex.c:893
  mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
  rtnl_lock+0x17/0x20 net/core/rtnetlink.c:74
  netdev_wait_allrefs+0x1cb/0x410 net/core/dev.c:8035
  netdev_run_todo+0x478/0xae0 net/core/dev.c:8129
  rtnl_unlock+0xe/0x10 net/core/rtnetlink.c:108
  default_device_exit_batch+0x4b4/0x5d0 net/core/dev.c:8930
  ops_exit_list.isra.6+0x100/0x150 net/core/net_namespace.c:145
  cleanup_net+0x6a3/0xcc0 net/core/net_namespace.c:517
  process_one_work+0xbbf/0x1af0 kernel/workqueue.c:2113
  worker_thread+0x223/0x1990 kernel/workqueue.c:2247
  kthread+0x33c/0x400 kernel/kthread.c:238
  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:429

Showing all locks held in the system:
4 locks held by kworker/u4:1/21:
  #0:  ((wq_completion)"%s""netns"){+.+.}, at: [<000000008671663e>]  
process_one_work+0xaaf/0x1af0 kernel/workqueue.c:2084
  #1:  (net_cleanup_work){+.+.}, at: [<0000000098e1951b>]  
process_one_work+0xb01/0x1af0 kernel/workqueue.c:2088
  #2:  (net_mutex){+.+.}, at: [<000000009da3dcc8>] cleanup_net+0x242/0xcc0  
net/core/net_namespace.c:484
  #3:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74
2 locks held by khungtaskd/758:
  #0:  (rcu_read_lock){....}, at: [<0000000032c7899a>]  
check_hung_uninterruptible_tasks kernel/hung_task.c:175 [inline]
  #0:  (rcu_read_lock){....}, at: [<0000000032c7899a>] watchdog+0x1c5/0xd60  
kernel/hung_task.c:249
  #1:  (tasklist_lock){.+.+}, at: [<000000008744ddaf>]  
debug_show_all_locks+0xd3/0x3d0 kernel/locking/lockdep.c:4470
1 lock held by rsyslogd/3947:
  #0:  (&f->f_pos_lock){+.+.}, at: [<00000000f50ca942>]  
__fdget_pos+0x12b/0x190 fs/file.c:765
2 locks held by getty/4070:
  #0:  (&tty->ldisc_sem){++++}, at: [<000000004fdd9acb>]  
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<0000000033740686>]  
n_tty_read+0x2ef/0x1a00 drivers/tty/n_tty.c:2131
2 locks held by getty/4071:
  #0:  (&tty->ldisc_sem){++++}, at: [<000000004fdd9acb>]  
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<0000000033740686>]  
n_tty_read+0x2ef/0x1a00 drivers/tty/n_tty.c:2131
2 locks held by getty/4072:
  #0:  (&tty->ldisc_sem){++++}, at: [<000000004fdd9acb>]  
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<0000000033740686>]  
n_tty_read+0x2ef/0x1a00 drivers/tty/n_tty.c:2131
2 locks held by getty/4073:
  #0:  (&tty->ldisc_sem){++++}, at: [<000000004fdd9acb>]  
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<0000000033740686>]  
n_tty_read+0x2ef/0x1a00 drivers/tty/n_tty.c:2131
2 locks held by getty/4074:
  #0:  (&tty->ldisc_sem){++++}, at: [<000000004fdd9acb>]  
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<0000000033740686>]  
n_tty_read+0x2ef/0x1a00 drivers/tty/n_tty.c:2131
2 locks held by getty/4075:
  #0:  (&tty->ldisc_sem){++++}, at: [<000000004fdd9acb>]  
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<0000000033740686>]  
n_tty_read+0x2ef/0x1a00 drivers/tty/n_tty.c:2131
2 locks held by getty/4076:
  #0:  (&tty->ldisc_sem){++++}, at: [<000000004fdd9acb>]  
ldsem_down_read+0x37/0x40 drivers/tty/tty_ldsem.c:365
  #1:  (&ldata->atomic_read_lock){+.+.}, at: [<0000000033740686>]  
n_tty_read+0x2ef/0x1a00 drivers/tty/n_tty.c:2131
1 lock held by syz-executor2/4154:
  #0:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74
3 locks held by kworker/1:3/7524:
  #0:  ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at:  
[<000000008671663e>] process_one_work+0xaaf/0x1af0 kernel/workqueue.c:2084
  #1:  ((addr_chk_work).work){+.+.}, at: [<0000000098e1951b>]  
process_one_work+0xb01/0x1af0 kernel/workqueue.c:2088
  #2:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74
1 lock held by syz-executor0/10595:
  #0:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74
1 lock held by syz-executor3/13841:
  #0:  (sk_lock-AF_INET6){+.+.}, at: [<0000000063f9b07b>] lock_sock  
include/net/sock.h:1463 [inline]
  #0:  (sk_lock-AF_INET6){+.+.}, at: [<0000000063f9b07b>]  
compat_ipv6_getsockopt+0x1f1/0x390 net/ipv6/ipv6_sockglue.c:1412
1 lock held by syz-executor3/13845:
  #0:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74
1 lock held by syz-executor3/13851:
  #0:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74
1 lock held by syz-executor3/13862:
  #0:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74
1 lock held by syz-executor0/13866:
  #0:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74
1 lock held by syz-executor2/13868:
  #0:  (rtnl_mutex){+.+.}, at: [<000000000b60ddd2>] rtnl_lock+0x17/0x20  
net/core/rtnetlink.c:74

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 758 Comm: khungtaskd Not tainted 4.15.0+ #213
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  __dump_stack lib/dump_stack.c:17 [inline]
  dump_stack+0x194/0x257 lib/dump_stack.c:53
  nmi_cpu_backtrace+0x1d2/0x210 lib/nmi_backtrace.c:103
  nmi_trigger_cpumask_backtrace+0x122/0x180 lib/nmi_backtrace.c:62
  arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
  trigger_all_cpu_backtrace include/linux/nmi.h:138 [inline]
  check_hung_task kernel/hung_task.c:132 [inline]
  check_hung_uninterruptible_tasks kernel/hung_task.c:190 [inline]
  watchdog+0x90c/0xd60 kernel/hung_task.c:249
  kthread+0x33c/0x400 kernel/kthread.c:238
  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:429
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0+ #213
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
RIP: 0010:rcu_lockdep_current_cpu_online+0x52/0x190 kernel/rcu/tree.c:1123
RSP: 0018:ffff8801db407780 EFLAGS: 00000282
RAX: ffffed003b680ef0 RBX: 1ffff1003b680ef0 RCX: 1ffff1003b680ef6
RDX: dffffc0000000000 RSI: ffffffff86b290c0 RDI: ffff8801db423750
RBP: ffff8801db407808 R08: 1ffff1003b680e47 R09: 0000000000000002
R10: ffff8801db4076f8 R11: 0000000000000020 R12: 1ffff1003b680f20
R13: ffff8801d9e66a00 R14: ffff8801db4079c0 R15: 1ffff1003b680f14
FS:  0000000000000000(0000) GS:ffff8801db400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000240f000 CR3: 0000000006a22003 CR4: 00000000001606f0
DR0: 0000000020000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
Call Trace:
  <IRQ>
  rcu_read_lock_held+0x94/0xc0 kernel/rcu/update.c:331
  rebalance_domains+0x945/0xcc0 kernel/sched/fair.c:9147
  run_rebalance_domains+0x378/0x770 kernel/sched/fair.c:9405
  __do_softirq+0x2d7/0xb85 kernel/softirq.c:285
  invoke_softirq kernel/softirq.c:365 [inline]
  irq_exit+0x1cc/0x200 kernel/softirq.c:405
  scheduler_ipi+0x318/0x820 kernel/sched/core.c:1804
  smp_reschedule_interrupt+0xe6/0x650 arch/x86/kernel/smp.c:277
  reschedule_interrupt+0xa9/0xb0 arch/x86/entry/entry_64.S:853
  </IRQ>
RIP: 0010:native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:54
RSP: 0018:ffffffff86a07c38 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff02
RAX: dffffc0000000000 RBX: 1ffffffff0d40f8a RCX: 0000000000000000
RDX: 1ffffffff0d59278 RSI: 0000000000000001 RDI: ffffffff86ac93c0
RBP: ffffffff86a07c38 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff86a07cf0 R14: ffffffff87269560 R15: 0000000000000000
  arch_safe_halt arch/x86/include/asm/paravirt.h:93 [inline]
  default_idle+0xbf/0x430 arch/x86/kernel/process.c:354
  arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:345
  default_idle_call+0x36/0x90 kernel/sched/idle.c:98
  cpuidle_idle_call kernel/sched/idle.c:156 [inline]
  do_idle+0x24a/0x3b0 kernel/sched/idle.c:246
  cpu_startup_entry+0x104/0x120 kernel/sched/idle.c:351
  rest_init+0xed/0xf0 init/main.c:436
  start_kernel+0x7f1/0x819 init/main.c:716
  x86_64_start_reservations+0x2a/0x2c arch/x86/kernel/head64.c:378
  x86_64_start_kernel+0x77/0x7a arch/x86/kernel/head64.c:359
  secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:237
Code: c7 85 78 ff ff ff b3 8a b5 41 48 8d 04 13 48 c7 45 80 b5 14 7d 86 48  
c7 45 88 50 e2 5d 81 c7 00 f1 f1 f1 f1 c7 40 04 00 f2 f2 f2 <c7> 40 08 f3  
f3 f3 f3 b8 01 00 00 00 65 8b 0d 9b 0a a4 7e 81 e1


---
This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to syzkaller@googlegroups.com.

syzbot will keep track of this bug report.
If you forgot to add the Reported-by tag, once the fix for this bug is  
merged
into any tree, please reply to this email with:
#syz fix: exact-commit-title
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug  
report.
Note: all commands must start from beginning of the line in the email body.

Attachments

Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help