Re: sysrq-b fails miserably to reboot PowerNV box
From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: 2016-10-08 23:06:30
On Sat, 2016-10-08 at 22:16 +1100, Anton Blanchard wrote:
Hi, Unfortunately sysrq-b seems to tie us up in knots, instead of rebooting the box. This is mainline from today.
Argh yeah reboot from interrupt context, what a PITA... ppc_md.machine_shutdown is called which is supposed to be a kexec thing, it clears the PCI IODA tables and does OPAL shutdown which frees the IRQs among other things. Then the actual reboot is called which does things that are duplicates of what machine_shutdown does ... Somebody needs to dig in there ... I can try next week if I have time but somebody else feel free to beat me to it. Ben.
quoted hunk ↗ jump to hunk
Anton -- Trying to free IRQ 17 from IRQ context! ------------[ cut here ]------------ WARNING: CPU: 32 PID: 0 at kernel/irq/manage.c:1460 __free_irq+0x298/0x380 Modules linked in: kvm_hv kvm binfmt_misc vmx_crypto crc32c_vpmsum leds_powernv powernv_rng led_class powernv_op_panel rng_core ib_iser rdma_cm iw_cm ib_cm ib_core libiscsi scsi_transport_iscsi autofs4 CPU: 32 PID: 0 Comm: swapper/32 Tainted: G D 4.8.0- 11288-gb66484c #785 task: c0000017f91fce00 task.stack: c0000017f927c000 NIP: c00000000012b4e8 LR: c00000000012b4e4 CTR: c000000000668950 REGS: c000001fffe8b5c0 TRAP: 0700 Tainted: G D (4.8.0-11288-gb66484c) MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28004224 XER: 20000000 CFAR: c00000000000c2e0 SOFTE: 1 GPR00: c00000000012b4e4 c000001fffe8b840 c000000000dbc100 0000000000000027 GPR04: c0000007fb80ada0 c0000007fb81f598 0000000000000000 c000000000dee9a8 GPR08: 0000000000000007 0000000000000000 00000007fab60000 9000000000001003 GPR12: 0000000000004400 c00000000fe08000 c0000017f927ff90 0000000000000000 GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000001 c000000000d35d70 c000001fffe8bb00 0000000000000000 GPR24: 0000000000000000 c0000007e3330400 c000000000d31de0 000000000000000f GPR28: 0000000000000011 c000000ff7ed1400 0000000000000000 0000000000000011 NIP [c00000000012b4e8] __free_irq+0x298/0x380 LR [c00000000012b4e4] __free_irq+0x294/0x380 Call Trace: [c000001fffe8b840] [c00000000012b4e4] __free_irq+0x294/0x380 (unreliable) [c000001fffe8b8d0] [c00000000012b6ec] free_irq+0x7c/0xb0 [c000001fffe8b900] [c000000000072748] opal_event_shutdown+0x88/0xd0 [c000001fffe8b940] [c00000000006db8c] opal_shutdown+0x1c/0x90 [c000001fffe8b970] [c000000000069d00] pnv_shutdown+0x20/0x40 [c000001fffe8b990] [c000000000026a18] machine_restart+0x38/0x90 [c000001fffe8ba00] [c0000000000e7538] emergency_restart+0x28/0x40 [c000001fffe8ba20] [c000000000568c44] sysrq_handle_reboot+0x24/0x40 [c000001fffe8ba40] [c000000000569490] __handle_sysrq+0xe0/0x250 [c000001fffe8bae0] [c00000000057f1a0] hvc_poll+0x230/0x380 [c000001fffe8bb80] [c0000000005803cc] hvc_handle_interrupt+0x1c/0x50 [c000001fffe8bba0] [c00000000012a2b0] __handle_irq_event_percpu+0x90/0x2f0 [c000001fffe8bc60] [c00000000012a540] handle_irq_event_percpu+0x30/0x90 [c000001fffe8bca0] [c00000000012a5fc] handle_irq_event+0x5c/0xc0 [c000001fffe8bcd0] [c00000000012f5b4] handle_level_irq+0xd4/0x1e0 [c000001fffe8bd00] [c000000000128e44] generic_handle_irq+0x44/0x70 [c000001fffe8bd20] [c000000000072600] opal_handle_events+0xd0/0x120 [c000001fffe8bd70] [c0000000000726a8] opal_interrupt+0x38/0x50 [c000001fffe8bda0] [c00000000012a2b0] __handle_irq_event_percpu+0x90/0x2f0 [c000001fffe8be60] [c00000000012a540] handle_irq_event_percpu+0x30/0x90 [c000001fffe8bea0] [c00000000012a5fc] handle_irq_event+0x5c/0xc0 [c000001fffe8bed0] [c00000000012f9f0] handle_fasteoi_irq+0xe0/0x290 [c000001fffe8bf00] [c000000000128e44] generic_handle_irq+0x44/0x70 [c000001fffe8bf20] [c00000000001641c] __do_irq+0x8c/0x210 [c000001fffe8bf90] [c000000000027ed4] call_do_irq+0x14/0x24 [c0000017f927fac0] [c000000000016634] do_IRQ+0x94/0x140 [c0000017f927fb10] [c000000000008e8c] hardware_interrupt_common+0x10c/0x180--- interrupt: 501 at arch_local_irq_restore+0x5c/0x90LR = arch_local_irq_restore+0x40/0x90 [c0000017f927fe00] [c000000000155e44] tick_broadcast_oneshot_control+0x54/0x70 (unreliable) [c0000017f927fe20] [c0000000007b6da4] cpuidle_enter_state+0x1b4/0x440 [c0000017f927fe80] [c000000000114da4] call_cpuidle+0x44/0x90 [c0000017f927fea0] [c000000000115138] cpu_startup_entry+0x1f8/0x320 [c0000017f927ff20] [c000000000043540] start_secondary+0x320/0x340 [c0000017f927ff90] [c00000000000b56c] start_secondary_prolog+0x10/0x14 Instruction dump: 7d2903a6 7fc3f378 f8410018 7d2c4b78 4e800421 e8410018 4bfffe68 3c62ffd6 7f84e378 38639270 48811f35 60000000 <0fe00000> 4bfffdb4 7d2903a6 7fc3f378 ---[ end trace dab28cb0b01424b6 ]--- EEH: Frozen PHB#3-PE#1 detected EEH: PE location: U78C9.001.WZS003V-P1-C10 , PHB location: N/A CPU: 25 PID: 0 Comm: swapper/25 Tainted: G D W 4.8.0- 11288-gb66484c #785 Call Trace: [c0000007f6483690] [c00000000093e6fc] dump_stack+0xb0/0xf0 (unreliable) [c0000007f64836d0] [c000000000039c30] eeh_dev_check_failure+0x3f0/0x530 [c0000007f6483780] [c000000000039df4] eeh_check_failure+0x84/0xd0 [c0000007f64837c0] [c000000000677fb0] tg3_read32+0x80/0x90 [c0000007f64837f0] [c000000000698acc] tg3_timer+0x5ec/0x1170 [c0000007f6483850] [c000000000142334] call_timer_fn+0x54/0x1d0 [c0000007f64838e0] [c000000000142600] expire_timers+0x120/0x1d0 [c0000007f6483950] [c0000000001428a8] run_timer_softirq+0x1f8/0x260 [c0000007f64839e0] [c0000000000c0478] __do_softirq+0x158/0x3e0 [c0000007f6483ac0] [c0000000000c0958] irq_exit+0xe8/0x120 [c0000007f6483ae0] [c0000000000400f0] doorbell_exception+0xa0/0xd0 [c0000007f6483b10] [c00000000000a20c] h_doorbell_common+0x10c/0x180--- interrupt: e81 at arch_local_irq_restore+0x5c/0x90LR = arch_local_irq_restore+0x40/0x90 [c0000007f6483e00] [c000000000155e44] tick_broadcast_oneshot_control+0x54/0x70 (unreliable) [c0000007f6483e20] [c0000000007b6da4] cpuidle_enter_state+0x1b4/0x440 [c0000007f6483e80] [c000000000114da4] call_cpuidle+0x44/0x90 [c0000007f6483ea0] [c000000000115138] cpu_startup_entry+0x1f8/0x320 [c0000007f6483f20] [c000000000043540] start_secondary+0x320/0x340 [c0000007f6483f90] [c00000000000b56c] start_secondary_prolog+0x10/0x14 EEH: Detected PCI bus error on PHB#3-PE#1 EEH: This PCI device has failed 1 times in the last hour EEH: Notify device drivers to shutdown tg3 0003:09:00.0 eth0: PCI I/O error detected tg3 0003:09:00.1 eth1: PCI I/O error detected tg3 0003:09:00.2 eth2: PCI I/O error detected tg3 0003:09:00.3 eth3: PCI I/O error detected EEH: Collect temporary log PHB3 PHB#3 Diag-data (Version: 1) brdgCtl: 00000002 RootSts: 0002002f 00400000 b0830008 00100147 00002800 RootErrSts: 00000000 00008000 00000000 PhbSts: 0000001c00000000 0000001c00000000 Lem: 4000000050000000 42498e327f502eae 0000000000000000 OutErr: 2000000000000000 2000000000000000 0204006000003b50 1000080000000000 InAErr: 0000008800000000 0000000800000000 001d090000000800 0800000ff3b47380 PE[ 0] A/B: 8000002100000000 8000000000000000 PE[ 1] A/B: 8000000000000000 8000000000000000 PE[ 2] A/B: 8000000000000000 8000000000000000 PE[ 3] A/B: 8000000000000000 8000000000000000 PE[ 4] A/B: 8000000000000000 8000000000000000 PE[ 5] A/B: 8000000000000000 8000000000000000 PE[ 6] A/B: 8000000000000000 8000000000000000 PE[ 7] A/B: 8000000000000000 8000000000000000 PE[ 8] A/B: 8000000000000000 8000000000000000 PE[ 9] A/B: 8000000000000000 8000000000000000 PE[ 10] A/B: 8000000000000000 8000000000000000 PE[ 11] A/B: 8000000000000000 8000000000000000 PE[ 12] A/B: 8000000000000000 8000000000000000 PE[ 13] A/B: 8000000000000000 8000000000000000 PE[ 14] A/B: 8000000000000000 8000000000000000 PE[ 15] A/B: 8000000000000000 8000000000000000 PE[ 16] A/B: 8000000000000000 8000000000000000 PE[ 17] A/B: 8000000000000000 8000000000000000 PE[ 18] A/B: 8000000000000000 8000000000000000 PE[ 19] A/B: 8000000000000000 8000000000000000 PE[ 20] A/B: 8000000000000000 8000000000000000 PE[ 21] A/B: 8000000000000000 8000000000000000 PE[ 22] A/B: 8000000000000000 8000000000000000 PE[ 23] A/B: 8000000000000000 8000000000000000 PE[ 24] A/B: 8000000000000000 8000000000000000 PE[ 25] A/B: 8000000000000000 8000000000000000 PE[ 26] A/B: 8000000000000000 8000000000000000 PE[ 27] A/B: 8000000000000000 8000000000000000 PE[ 28] A/B: 8000000000000000 8000000000000000 PE[ 29] A/B: 8000000000000000 8000000000000000 PE[ 30] A/B: 8000000000000000 8000000000000000 PE[ 31] A/B: 8000000000000000 8000000000000000 PE[ 32] A/B: 8000000000000000 8000000000000000 PE[ 33] A/B: 8000000000000000 8000000000000000 PE[ 34] A/B: 8000000000000000 8000000000000000 PE[ 35] A/B: 8000000000000000 8000000000000000 PE[ 36] A/B: 8000000000000000 8000000000000000 PE[ 37] A/B: 8000000000000000 8000000000000000 PE[ 38] A/B: 8000000000000000 8000000000000000 PE[ 39] A/B: 8000000000000000 8000000000000000 PE[ 40] A/B: 8000000000000000 8000000000000000 PE[ 41] A/B: 8000000000000000 8000000000000000 PE[ 42] A/B: 8000000000000000 8000000000000000 PE[ 43] A/B: 8000000000000000 8000000000000000 PE[ 44] A/B: 8000000000000000 8000000000000000 PE[ 45] A/B: 8000000000000000 8000000000000000 PE[ 46] A/B: 8000000000000000 8000000000000000 PE[ 47] A/B: 8000000000000000 8000000000000000 PE[ 48] A/B: 8000000000000000 8000000000000000 PE[ 49] A/B: 8000000000000000 8000000000000000 PE[ 50] A/B: 8000000000000000 8000000000000000 PE[ 51] A/B: 8000000000000000 8000000000000000 PE[ 52] A/B: 8000000000000000 8000000000000000 PE[ 53] A/B: 8000000000000000 8000000000000000 PE[ 54] A/B: 8000000000000000 8000000000000000 PE[ 55] A/B: 8000000000000000 8000000000000000 PE[ 56] A/B: 8000000000000000 8000000000000000 PE[ 57] A/B: 8000000000000000 8000000000000000 PE[ 58] A/B: 8000000000000000 8000000000000000 PE[ 59] A/B: 8000000000000000 8000000000000000 PE[ 60] A/B: 8000000000000000 8000000000000000 PE[ 61] A/B: 8000000000000000 8000000000000000 PE[ 62] A/B: 8000000000000000 8000000000000000 PE[ 63] A/B: 8000000000000000 8000000000000000 PE[ 64] A/B: 8000000000000000 8000000000000000 PE[ 65] A/B: 8000000000000000 8000000000000000 PE[ 66] A/B: 8000000000000000 8000000000000000 PE[ 67] A/B: 8000000000000000 8000000000000000 PE[ 68] A/B: 8000000000000000 8000000000000000 PE[ 69] A/B: 8000000000000000 8000000000000000 PE[ 70] A/B: 8000000000000000 8000000000000000 PE[ 71] A/B: 8000000000000000 8000000000000000 PE[ 72] A/B: 8000000000000000 8000000000000000 PE[ 73] A/B: 8000000000000000 8000000000000000 PE[ 74] A/B: 8000000000000000 8000000000000000 PE[ 75] A/B: 8000000000000000 8000000000000000 PE[ 76] A/B: 8000000000000000 8000000000000000 PE[ 77] A/B: 8000000000000000 8000000000000000 PE[ 78] A/B: 8000000000000000 8000000000000000 PE[ 79] A/B: 8000000000000000 8000000000000000 PE[ 80] A/B: 8000000000000000 8000000000000000 PE[ 81] A/B: 8000000000000000 8000000000000000 PE[ 82] A/B: 8000000000000000 8000000000000000 PE[ 83] A/B: 8000000000000000 8000000000000000 PE[ 84] A/B: 8000000000000000 8000000000000000 PE[ 85] A/B: 8000000000000000 8000000000000000 PE[ 86] A/B: 8000000000000000 8000000000000000 PE[ 87] A/B: 8000000000000000 8000000000000000 PE[ 88] A/B: 8000000000000000 8000000000000000 PE[ 89] A/B: 8000000000000000 8000000000000000 PE[ 90] A/B: 8000000000000000 8000000000000000 PE[ 91] A/B: 8000000000000000 8000000000000000 PE[ 92] A/B: 8000000000000000 8000000000000000 PE[ 93] A/B: 8000000000000000 8000000000000000 PE[ 94] A/B: 8000000000000000 8000000000000000 PE[ 95] A/B: 8000000000000000 8000000000000000 PE[ 96] A/B: 8000000000000000 8000000000000000 PE[ 97] A/B: 8000000000000000 8000000000000000 PE[ 98] A/B: 8000000000000000 8000000000000000 PE[ 99] A/B: 8000000000000000 8000000000000000 PE[100] A/B: 8000000000000000 8000000000000000 PE[101] A/B: 8000000000000000 8000000000000000 PE[102] A/B: 8000000000000000 8000000000000000 PE[103] A/B: 8000000000000000 8000000000000000 PE[104] A/B: 8000000000000000 8000000000000000 PE[105] A/B: 8000000000000000 8000000000000000 PE[106] A/B: 8000000000000000 8000000000000000 PE[107] A/B: 8000000000000000 8000000000000000 PE[108] A/B: 8000000000000000 8000000000000000 PE[109] A/B: 8000000000000000 8000000000000000 PE[110] A/B: 8000000000000000 8000000000000000 PE[111] A/B: 8000000000000000 8000000000000000 PE[112] A/B: 8000000000000000 8000000000000000 PE[113] A/B: 8000000000000000 8000000000000000 PE[114] A/B: 8000000000000000 8000000000000000 PE[115] A/B: 8000000000000000 8000000000000000 PE[116] A/B: 8000000000000000 8000000000000000 PE[117] A/B: 8000000000000000 8000000000000000 PE[118] A/B: 8000000000000000 8000000000000000 PE[119] A/B: 8000000000000000 8000000000000000 PE[120] A/B: 8000000000000000 8000000000000000 PE[121] A/B: 8000000000000000 8000000000000000 PE[122] A/B: 8000000000000000 8000000000000000 PE[123] A/B: 8000000000000000 8000000000000000 PE[124] A/B: 8000000000000000 8000000000000000 PE[125] A/B: 8000000000000000 8000000000000000 PE[126] A/B: 8000000000000000 8000000000000000 PE[127] A/B: 8000000000000000 8000000000000000 PE[128] A/B: 8000000000000000 8000000000000000 PE[129] A/B: 8000000000000000 8000000000000000 PE[130] A/B: 8000000000000000 8000000000000000 PE[131] A/B: 8000000000000000 8000000000000000 PE[132] A/B: 8000000000000000 8000000000000000 PE[133] A/B: 8000000000000000 8000000000000000 PE[134] A/B: 8000000000000000 8000000000000000 PE[135] A/B: 8000000000000000 8000000000000000 PE[136] A/B: 8000000000000000 8000000000000000 PE[137] A/B: 8000000000000000 8000000000000000 PE[138] A/B: 8000000000000000 8000000000000000 PE[139] A/B: 8000000000000000 8000000000000000 PE[140] A/B: 8000000000000000 8000000000000000 PE[141] A/B: 8000000000000000 8000000000000000 PE[142] A/B: 8000000000000000 8000000000000000 PE[143] A/B: 8000000000000000 8000000000000000 PE[144] A/B: 8000000000000000 8000000000000000 PE[145] A/B: 8000000000000000 8000000000000000 PE[146] A/B: 8000000000000000 8000000000000000 PE[147] A/B: 8000000000000000 8000000000000000 PE[148] A/B: 8000000000000000 8000000000000000 PE[149] A/B: 8000000000000000 8000000000000000 PE[150] A/B: 8000000000000000 8000000000000000 PE[151] A/B: 8000000000000000 8000000000000000 PE[152] A/B: 8000000000000000 8000000000000000 PE[153] A/B: 8000000000000000 8000000000000000 PE[154] A/B: 8000000000000000 8000000000000000 PE[155] A/B: 8000000000000000 8000000000000000 PE[156] A/B: 8000000000000000 8000000000000000 PE[157] A/B: 8000000000000000 8000000000000000 PE[158] A/B: 8000000000000000 8000000000000000 PE[159] A/B: 8000000000000000 8000000000000000 PE[160] A/B: 8000000000000000 8000000000000000 PE[161] A/B: 8000000000000000 8000000000000000 PE[162] A/B: 8000000000000000 8000000000000000 PE[163] A/B: 8000000000000000 8000000000000000 PE[164] A/B: 8000000000000000 8000000000000000 PE[165] A/B: 8000000000000000 8000000000000000 PE[166] A/B: 8000000000000000 8000000000000000 PE[167] A/B: 8000000000000000 8000000000000000 PE[168] A/B: 8000000000000000 8000000000000000 PE[169] A/B: 8000000000000000 8000000000000000 PE[170] A/B: 8000000000000000 8000000000000000 PE[171] A/B: 8000000000000000 8000000000000000 PE[172] A/B: 8000000000000000 8000000000000000 PE[173] A/B: 8000000000000000 8000000000000000 PE[174] A/B: 8000000000000000 8000000000000000 PE[175] A/B: 8000000000000000 8000000000000000 PE[176] A/B: 8000000000000000 8000000000000000 PE[177] A/B: 8000000000000000 8000000000000000 PE[178] A/B: 8000000000000000 8000000000000000 PE[179] A/B: 8000000000000000 8000000000000000 PE[180] A/B: 8000000000000000 8000000000000000 PE[181] A/B: 8000000000000000 8000000000000000 PE[182] A/B: 8000000000000000 8000000000000000 PE[183] A/B: 8000000000000000 8000000000000000 PE[184] A/B: 8000000000000000 8000000000000000 PE[185] A/B: 8000000000000000 8000000000000000 PE[186] A/B: 8000000000000000 8000000000000000 PE[187] A/B: 8000000000000000 8000000000000000 PE[188] A/B: 8000000000000000 8000000000000000 PE[189] A/B: 8000000000000000 8000000000000000 PE[190] A/B: 8000000000000000 8000000000000000 PE[191] A/B: 8000000000000000 8000000000000000 PE[192] A/B: 8000000000000000 8000000000000000 PE[193] A/B: 8000000000000000 8000000000000000 PE[194] A/B: 8000000000000000 8000000000000000 PE[195] A/B: 8000000000000000 8000000000000000 PE[196] A/B: 8000000000000000 8000000000000000 PE[197] A/B: 8000000000000000 8000000000000000 PE[198] A/B: 8000000000000000 8000000000000000 PE[199] A/B: 8000000000000000 8000000000000000 PE[200] A/B: 8000000000000000 8000000000000000 PE[201] A/B: 8000000000000000 8000000000000000 PE[202] A/B: 8000000000000000 8000000000000000 PE[203] A/B: 8000000000000000 8000000000000000 PE[204] A/B: 8000000000000000 8000000000000000 PE[205] A/B: 8000000000000000 8000000000000000 PE[206] A/B: 8000000000000000 8000000000000000 PE[207] A/B: 8000000000000000 8000000000000000 PE[208] A/B: 8000000000000000 8000000000000000 PE[209] A/B: 8000000000000000 8000000000000000 PE[210] A/B: 8000000000000000 8000000000000000 PE[211] A/B: 8000000000000000 8000000000000000 PE[212] A/B: 8000000000000000 8000000000000000 PE[213] A/B: 8000000000000000 8000000000000000 PE[214] A/B: 8000000000000000 8000000000000000 PE[215] A/B: 8000000000000000 8000000000000000 PE[216] A/B: 8000000000000000 8000000000000000 PE[217] A/B: 8000000000000000 8000000000000000 PE[218] A/B: 8000000000000000 8000000000000000 PE[219] A/B: 8000000000000000 8000000000000000 PE[220] A/B: 8000000000000000 8000000000000000 PE[221] A/B: 8000000000000000 8000000000000000 PE[222] A/B: 8000000000000000 8000000000000000 PE[223] A/B: 8000000000000000 8000000000000000 PE[224] A/B: 8000000000000000 8000000000000000 PE[225] A/B: 8000000000000000 8000000000000000 PE[226] A/B: 8000000000000000 8000000000000000 PE[227] A/B: 8000000000000000 8000000000000000 PE[228] A/B: 8000000000000000 8000000000000000 PE[229] A/B: 8000000000000000 8000000000000000 PE[230] A/B: 8000000000000000 8000000000000000 PE[231] A/B: 8000000000000000 8000000000000000 PE[232] A/B: 8000000000000000 8000000000000000 PE[233] A/B: 8000000000000000 8000000000000000 PE[234] A/B: 8000000000000000 8000000000000000 PE[235] A/B: 8000000000000000 8000000000000000 PE[236] A/B: 8000000000000000 8000000000000000 PE[237] A/B: 8000000000000000 8000000000000000 PE[238] A/B: 8000000000000000 8000000000000000 PE[239] A/B: 8000000000000000 8000000000000000 PE[240] A/B: 8000000000000000 8000000000000000 PE[241] A/B: 8000000000000000 8000000000000000 PE[242] A/B: 8000000000000000 8000000000000000 PE[243] A/B: 8000000000000000 8000000000000000 PE[244] A/B: 8000000000000000 8000000000000000 PE[245] A/B: 8000000000000000 8000000000000000 PE[246] A/B: 8000000000000000 8000000000000000 PE[247] A/B: 8000000000000000 8000000000000000 PE[248] A/B: 8000000000000000 8000000000000000 PE[249] A/B: 8000000000000000 8000000000000000 PE[250] A/B: 8000000000000000 8000000000000000 PE[251] A/B: 8000000000000000 8000000000000000 PE[252] A/B: 8000000000000000 8000000000000000 PE[253] A/B: 8000000000000000 8000000000000000 PE[254] A/B: 8000000000000000 8000000000000000 PE[255] A/B: 8000802309000000 8800000ff3b47380 EEH: Reset without hotplug activity EEH: Notify device drivers the completion of reset NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [kworker/19:2:2695] Modules linked in: kvm_hv kvm binfmt_misc vmx_crypto crc32c_vpmsum leds_powernv powernv_rng led_class powernv_op_panel rng_core ib_iser rdma_cm iw_cm ib_cm ib_core libiscsi scsi_transport_iscsi autofs4 CPU: 19 PID: 2695 Comm: kworker/19:2 Tainted: G D W 4.8.0- 11288-gb66484c #785 Workqueue: events wait_rcu_exp_gp task: c0000007f4c10300 task.stack: c0000007f4e88000 NIP: c00000000015ff28 LR: c000000000137d94 CTR: c0000000000681e0 REGS: c0000007f4e8b890 TRAP: 0901 Tainted: G D W (4.8.0-11288-gb66484c) MSR: 900000010280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]> CR: 48002422 XER: 20000000 CFAR: c00000000015ff30 SOFTE: 1 GPR00: c000000000137d94 c0000007f4e8bb10 c000000000dbc100 0000000000000020 GPR04: c0000000001374a0 c000000000ce1280 0000000000000000 c000000000dee9a8 GPR08: c00000000fe04c00 0000000000000001 c0000007fb4e3680 0000000000000000 GPR12: c0000000000681e0 c00000000fe04c00 NIP [c00000000015ff28] smp_call_function_single+0x98/0x180 LR [c000000000137d94] wait_rcu_exp_gp+0x294/0xaf0 Call Trace: [c0000007f4e8bb10] [c00000000015ff58] smp_call_function_single+0xc8/0x180 (unreliable) [c0000007f4e8bb50] [c000000000137d94] wait_rcu_exp_gp+0x294/0xaf0 [c0000007f4e8bc50] [c0000000000dbab0] process_one_work+0x2a0/0x5a0 [c0000007f4e8bce0] [c0000000000dbe58] worker_thread+0xa8/0x6a0 [c0000007f4e8bd80] [c0000000000e41d0] kthread+0x120/0x130 [c0000007f4e8be30] [c00000000000bf68] ret_from_kernel_thread+0x5c/0x74 Instruction dump: 409e0078 e9280030 3d42fff0 394a7580 7d4a4a14 812a0018 71290001 41820020 60000000 60420000 7c210b78 7c421378 <812a0018> 71290001 4082fff0 7c2004ac NMI watchdog: BUG: soft lockup - CPU#32 stuck for 22s! [swapper/32:0] Modules linked in: kvm_hv kvm binfmt_misc vmx_crypto crc32c_vpmsum leds_powernv powernv_rng led_class powernv_op_panel rng_core ib_iser rdma_cm iw_cm ib_cm ib_core libiscsi scsi_transport_iscsi autofs4 CPU: 32 PID: 0 Comm: swapper/32 Tainted: G D W L 4.8.0- 11288-gb66484c #785 task: c0000017f91fce00 task.stack: c0000017f927c000 NIP: c00000000012a698 LR: c00000000012a840 CTR: 0000000000000000 REGS: c000001fffe8b520 TRAP: 0901 Tainted: G D W L (4.8.0-11288-gb66484c) MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44004882 XER: 00000000 CFAR: c00000000012a6a4 SOFTE: 1 GPR00: c00000000012a840 c000001fffe8b7a0 c000000000dbc100 c000000ff7ed1400 GPR04: 0000000000000011 0000000000000000 0000000000000000 0000000000000000 GPR08: c000000ff7ed1400 0000000000040000 c0000007f90000b0 c0000007f9000028 GPR12: 0000000000000040 c00000000fe08000 NIP [c00000000012a698] __synchronize_hardirq+0x38/0xa0 LR [c00000000012a840] synchronize_irq+0x30/0xd0 Call Trace: [c000001fffe8b7a0] [c000000000339548] remove_proc_subtree+0x168/0x1c0 (unreliable) [c000001fffe8b7e0] [c00000000012a840] synchronize_irq+0x30/0xd0 [c000001fffe8b840] [c00000000012b37c] __free_irq+0x12c/0x380 [c000001fffe8b8d0] [c00000000012b6ec] free_irq+0x7c/0xb0 [c000001fffe8b900] [c000000000072748] opal_event_shutdown+0x88/0xd0 [c000001fffe8b940] [c00000000006db8c] opal_shutdown+0x1c/0x90 [c000001fffe8b970] [c000000000069d00] pnv_shutdown+0x20/0x40 [c000001fffe8b990] [c000000000026a18] machine_restart+0x38/0x90 [c000001fffe8ba00] [c0000000000e7538] emergency_restart+0x28/0x40 [c000001fffe8ba20] [c000000000568c44] sysrq_handle_reboot+0x24/0x40 [c000001fffe8ba40] [c000000000569490] __handle_sysrq+0xe0/0x250 [c000001fffe8bae0] [c00000000057f1a0] hvc_poll+0x230/0x380 [c000001fffe8bb80] [c0000000005803cc] hvc_handle_interrupt+0x1c/0x50 [c000001fffe8bba0] [c00000000012a2b0] __handle_irq_event_percpu+0x90/0x2f0 [c000001fffe8bc60] [c00000000012a540] handle_irq_event_percpu+0x30/0x90 [c000001fffe8bca0] [c00000000012a5fc] handle_irq_event+0x5c/0xc0 [c000001fffe8bcd0] [c00000000012f5b4] handle_level_irq+0xd4/0x1e0 [c000001fffe8bd00] [c000000000128e44] generic_handle_irq+0x44/0x70 [c000001fffe8bd20] [c000000000072600] opal_handle_events+0xd0/0x120 [c000001fffe8bd70] [c0000000000726a8] opal_interrupt+0x38/0x50 [c000001fffe8bda0] [c00000000012a2b0] __handle_irq_event_percpu+0x90/0x2f0 [c000001fffe8be60] [c00000000012a540] handle_irq_event_percpu+0x30/0x90 [c000001fffe8bea0] [c00000000012a5fc] handle_irq_event+0x5c/0xc0 [c000001fffe8bed0] [c00000000012f9f0] handle_fasteoi_irq+0xe0/0x290 [c000001fffe8bf00] [c000000000128e44] generic_handle_irq+0x44/0x70 [c000001fffe8bf20] [c00000000001641c] __do_irq+0x8c/0x210 [c000001fffe8bf90] [c000000000027ed4] call_do_irq+0x14/0x24 [c0000017f927fac0] [c000000000016634] do_IRQ+0x94/0x140 [c0000017f927fb10] [c000000000008e8c] hardware_interrupt_common+0x10c/0x180--- interrupt: 501 at arch_local_irq_restore+0x5c/0x90LR = arch_local_irq_restore+0x40/0x90 [c0000017f927fe00] [c000000000155e44] tick_broadcast_oneshot_control+0x54/0x70 (unreliable) [c0000017f927fe20] [c0000000007b6da4] cpuidle_enter_state+0x1b4/0x440 [c0000017f927fe80] [c000000000114da4] call_cpuidle+0x44/0x90 [c0000017f927fea0] [c000000000115138] cpu_startup_entry+0x1f8/0x320 [c0000017f927ff20] [c000000000043540] start_secondary+0x320/0x340 [c0000017f927ff90] [c00000000000b56c] start_secondary_prolog+0x10/0x14 Instruction dump: 7c0802a6 fba1ffe8 fbc1fff0 3ba3018c fbe1fff8 7c7e1b78 f8010010 f821ffc1 48000010 60420000 7c210b78 7c421378 <e93e0128> 81290000 75290004 4082ffec