ppc: hard lockup / hang in v5.17-rc1 under QEMU

Started by Miguel Ojeda on Jan 26, 2022 · 7 messages from 4 authors · Last reply Jan 27, 2022

open the first message on its own page

Message details and related content
Hi PPC folks,

Our ppc64le CI deterministically triggers a hard lockup / hang under
QEMU since v5.17-rc1 (upgrading from v5.16).

Bisecting points to 0faf20a1ad16 ("powerpc/64s/interrupt: Don't enable
MSR[EE] in irq handlers unless perf is in use").

Cheers,
Miguel

[   16.328310] watchdog: CPU 1 detected hard LOCKUP on other CPUs 0
[   16.328955] watchdog: CPU 1 TB:16743325700, last SMP heartbeat
TB:8453096925 (16191ms ago)
[   16.330786] watchdog: CPU 0 Hard LOCKUP
[   16.331078] watchdog: CPU 0 TB:16744720354, last heartbeat
TB:8453109168 (16194ms ago)
[   16.331295] Kernel panic - not syncing: Hard LOCKUP
[   16.331312] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc2+ #28
[   16.331729] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.16.0-rc2+ #28
[   16.332294] NIP:  c000000000009784 LR: c00000000034ee60 CTR: c0000000000096e0
[   16.332339] REGS: c00000001ff87d60 TRAP: 0100   Not tainted  (5.16.0-rc2+)
[   16.332410] MSR:  8000000000001031 <SF
[   16.332520] Call Trace:
[   16.334429] ,ME,IR,DR,LE>  CR: 24000088  XER: 20040000
[   16.334770] CFAR: c00000000000977c IRQMASK: 3
[   16.334770] GPR00: c00000000034ee60 c00000000111f590
c00000000111f400 0000000000008002
[   16.334770] GPR04: 0000000000000001 c00000000113f400
c00000000106f400 c00000000110f400
[   16.334770] GPR08: c0000000003af400 0000000024000088
c00000000111f8a0 c000000000011d00
[   16.334770] GPR12: 8000000000009033 c0000000011d0000
0000000000000000 0000000000000000
[   16.334770] GPR16: c00000000110ab90 c0000000010722f0
c000000001142100 0000000000000001
[   16.334770] GPR20: 0000000000000000 000000000000000a
ffffffff00049233 c0000000010722a8
[   16.334770] GPR24: 0000000000000282 c000000001076580
c000000001143a00 0000000004200002

121 more lines

Message details and related content
On 1/26/22 15:16, Miguel Ojeda wrote:
Hi PPC folks,

Our ppc64le CI deterministically triggers a hard lockup / hang under
QEMU since v5.17-rc1 (upgrading from v5.16).

Bisecting points to 0faf20a1ad16 ("powerpc/64s/interrupt: Don't enable
MSR[EE] in irq handlers unless perf is in use").
Indeed. I could reproduce.

Could you please send the QEMU command line and the full dmesg ? and
possibly open an issue on :

  https://gitlab.com/qemu-project/qemu/-/issues/

I guess it's a QEMU modeling issue.

Thanks,

C.

Cheers,
Miguel

[   16.328310] watchdog: CPU 1 detected hard LOCKUP on other CPUs 0
[   16.328955] watchdog: CPU 1 TB:16743325700, last SMP heartbeat
TB:8453096925 (16191ms ago)
[   16.330786] watchdog: CPU 0 Hard LOCKUP
[   16.331078] watchdog: CPU 0 TB:16744720354, last heartbeat
TB:8453109168 (16194ms ago)
[   16.331295] Kernel panic - not syncing: Hard LOCKUP
[   16.331312] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.16.0-rc2+ #28
[   16.331729] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.16.0-rc2+ #28
[   16.332294] NIP:  c000000000009784 LR: c00000000034ee60 CTR: c0000000000096e0
[   16.332339] REGS: c00000001ff87d60 TRAP: 0100   Not tainted  (5.16.0-rc2+)
[   16.332410] MSR:  8000000000001031 <SF
[   16.332520] Call Trace:
[   16.334429] ,ME,IR,DR,LE>  CR: 24000088  XER: 20040000
[   16.334770] CFAR: c00000000000977c IRQMASK: 3
[   16.334770] GPR00: c00000000034ee60 c00000000111f590
c00000000111f400 0000000000008002
[   16.334770] GPR04: 0000000000000001 c00000000113f400
c00000000106f400 c00000000110f400
[   16.334770] GPR08: c0000000003af400 0000000024000088
c00000000111f8a0 c000000000011d00
[   16.334770] GPR12: 8000000000009033 c0000000011d0000
0000000000000000 0000000000000000
[   16.334770] GPR16: c00000000110ab90 c0000000010722f0
c000000001142100 0000000000000001
[   16.334770] GPR20: 0000000000000000 000000000000000a
ffffffff00049233 c0000000010722a8
[   16.334770] GPR24: 0000000000000282 c000000001076580
c000000001143a00 0000000004200002
[   16.334770] GPR28: c0000000010b0480 c0000000010722a8
c0000000003b6cfa 00000000015e0000
[   16.334464] [c0000000062675c0] [c0000000003283e0] dump_stack_lvl+0x78/0xb8
[   16.335274] NIP [c000000000009784] decrementer_common_virt+0xa4/0x210
[   16.336451]  (unreliable)
[   16.335629] LR [c00000000034ee60] __do_softirq+0xe0/0x2c4
[   16.336658] [c000000006267600] [c00000000009378c] panic+0x150/0x3a4

115 more quoted lines

Message details and related content
On Wed, Jan 26, 2022 at 4:03 PM Cédric Le Goater [off-list ref] wrote:
Indeed. I could reproduce.
Thanks for the quick confirmation!
Could you please send the QEMU command line and the full dmesg ? and
possibly open an issue on :

   https://gitlab.com/qemu-project/qemu/-/issues/

I guess it's a QEMU modeling issue.
Of course -- done (details there):

    https://gitlab.com/qemu-project/qemu/-/issues/842

Cheers,
Miguel
Message details and related content
Excerpts from Miguel Ojeda's message of January 27, 2022 4:57 am:
On Wed, Jan 26, 2022 at 4:03 PM Cédric Le Goater [off-list ref] wrote:
quoted
Indeed. I could reproduce.
Thanks for the quick confirmation!
quoted
Could you please send the QEMU command line and the full dmesg ? and
possibly open an issue on :

   https://gitlab.com/qemu-project/qemu/-/issues/

I guess it's a QEMU modeling issue.
Of course -- done (details there):

    https://gitlab.com/qemu-project/qemu/-/issues/842
That sounds like my fault actually.

https://lists.ozlabs.org/pipermail/linuxppc-dev/2022-January/239178.html

Thanks,
Nick
Message details and related content
Miguel Ojeda [off-list ref] writes:
Hi PPC folks,

Our ppc64le CI deterministically triggers a hard lockup / hang under
QEMU since v5.17-rc1 (upgrading from v5.16).

Bisecting points to 0faf20a1ad16 ("powerpc/64s/interrupt: Don't enable
MSR[EE] in irq handlers unless perf is in use").
Hi Miguel,

Thanks for the report.

Nick has posted one fix for the commit you identified:

  http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20220124143930.3923442-1-npiggin@gmail.com/


It looks like your kernel-ppc64le-release.config does not have the
hardlockup detector enabled, so I suspect you're hitting the bug
described in that patch.

That fix will hit linux-next in the next day or so and should be in rc2.

cheers
Message details and related content
Hi Michael,

On Thu, Jan 27, 2022 at 11:54 AM Michael Ellerman [off-list ref] wrote:
It looks like your kernel-ppc64le-release.config does not have the
hardlockup detector enabled, so I suspect you're hitting the bug
described in that patch.
On -release it was a hang; but please note that on -debug the hard
lockup was actually being detected (so it was also reproducible with
the detectors enabled).
That fix will hit linux-next in the next day or so and should be in rc2.
I confirm the fix works for us -- thanks a lot!

Cheers,
Miguel
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help