From: Michal Suchánek <hidden> Date: 2020-08-30 20:13:36
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
Thanks
Michal
commit 87706fefb43f4b9320fb5b18b408237c6200468a
Author: Russell Currey [off-list ref]
Date: Thu Apr 18 16:51:16 2019 +1000
powerpc/powernv/idle: Restore IAMR after idle
...
commit b01a02e38c2a8d93e81133cb4ed5d92644453fbf
Author: Michal Suchanek [off-list ref]
Date: Sun Aug 30 12:14:36 2020 +0200
Revert "powerpc/64s: Reimplement book3s idle code in C"
This reverts commit 10d91611f426d4bafd2a83d966c36da811b2f7ad.
commit 2718c37d8e1e37f35e596727a7d39275a602f071
Author: Michal Suchanek [off-list ref]
Date: Sun Aug 30 12:10:17 2020 +0200
Revert "powerpc/powernv/idle: Restore AMR/UAMOR/AMOR/IAMR after idle"
This reverts commit e9cef0189c5b217fcd4788562862defc27632a01.
commit 7cf36babc860e5d2cd130dba1360b132a74f16d7
Author: Michal Suchanek [off-list ref]
Date: Sun Aug 30 12:09:23 2020 +0200
Revert "powerpc/powernv/idle: Fix restore of SPRN_LDBAR for POWER9 stop state."
This reverts commit d71acd3f404e65b4cfcbdd3ae096c15c847d7d04.
From: Nicholas Piggin <npiggin@gmail.com> Date: 2020-08-31 01:16:17
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately. Do the messages always come in pairs of CPUs?
I'm not sure where to start with reproducing, I'll have to try. How many
vCPUs in the guests? Do you have several guests running at once?
Thanks,
Nick
From: Michal Suchánek <hidden> Date: 2020-08-31 09:09:19
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately. Do the messages always come in pairs of CPUs?
I'm not sure where to start with reproducing, I'll have to try. How many
vCPUs in the guests? Do you have several guests running at once?
The guests are spawned on demand - there are like 20-30 'slots'
configured where a VM may be running or it may be idle with no VM
spawned when there are no jobs available.
Thanks
Michal
From: Michal Suchánek <hidden> Date: 2020-08-31 09:17:20
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Thanks
Michal
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately. Do the messages always come in pairs of CPUs?
I'm not sure where to start with reproducing, I'll have to try. How many
vCPUs in the guests? Do you have several guests running at once?
Hello all,
some details on the setup...
these machines are buildservice workers, (build.opensuse.org) and all they
do is spawn new VMs, run a package building job inside (rpmbuild, debbuild,...)
The machines are running in OPAL/PowerNV mode, with "ppc64_cpu --smt=off".
The number of VMs varies across the machines:
obs-power8-01: 18 instances, "-smp 16,threads=8"
obs-power8-02: 20 instances, "-smp 8,threads=8"
obs-power8-03: 30 instances, "-smp 8,threads=8"
obs-power8-04: 20 instances, "-smp 8,threads=8"
obs-power8-05: 36 instances, "-smp 4,threads=2" (this one with "ppc64_cpu --subcores-per-core=4"
but anyway the stalls can be seen on all of them, sometimes after 4 hours
sometimes just after about a day. The 01 with more cpu overcommit seems
a little faster reproducing the problem, but that's more gut feeling than
anything backed by real numbers.
--
with kind regards (mit freundlichem Grinsen),
Ruediger Oertel (ro@suse.com,ro@suse.de,bugfinder@t-online.de)
--------Do-Not-Accept-Binary-Blobs.----Ever.----From-Anyone.------------
Key fingerprint = 17DC 6553 86A7 384B 53C5 CA5C 3CE4 F2E7 23F2 B417
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg,
Germany, (HRB 36809, AG Nürnberg), Geschäftsführer: Felix Imendörffer
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
Do the messages always come in pairs of CPUs?
well,
- one problem is that at some point the machine just locks up completely,
so I can not tell if there were lines not printed any more and in some
cases all I get is a single line
- looking at the stats in generally it's either one cpu printed several
times or a pair ("not strictly") alternatingly
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.029821] KVM: couldn't grab cpu 92
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.058630] KVM: couldn't grab cpu 92
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.108268] KVM: couldn't grab cpu 92
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.210206] KVM: couldn't grab cpu 92
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.323465] KVM: couldn't grab cpu 92
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.334420] KVM: couldn't grab cpu 92
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.345470] KVM: couldn't grab cpu 92
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.395185] KVM: couldn't grab cpu 92
2020-07-30T03:16:16+00:00 obs-power8-03 kernel: [51284.517182] KVM: couldn't grab cpu 92
2020-07-30T03:16:17+00:00 obs-power8-03 kernel: [51284.600716] KVM: couldn't grab cpu 92
2020-07-30T03:16:18+00:00 obs-power8-03 kernel: [51286.201589] KVM: couldn't grab cpu 92
2020-07-30T03:16:19+00:00 obs-power8-03 kernel: [51286.627273] KVM: couldn't grab cpu 92
2020-07-30T16:44:16+00:00 obs-power8-04 kernel: [30099.726288] KVM: couldn't grab cpu 61
2020-07-30T16:44:16+00:00 obs-power8-04 kernel: [30099.736843] KVM: couldn't grab cpu 125
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30099.747429] KVM: couldn't grab cpu 125
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30099.877138] KVM: couldn't grab cpu 61
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30099.916422] KVM: couldn't grab cpu 125
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30099.931755] KVM: couldn't grab cpu 61
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30100.029003] KVM: couldn't grab cpu 61
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30100.334895] KVM: couldn't grab cpu 125
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30100.392713] KVM: couldn't grab cpu 61
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30100.569011] KVM: couldn't grab cpu 125
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30100.617048] KVM: couldn't grab cpu 125
2020-07-30T16:44:17+00:00 obs-power8-04 kernel: [30100.628107] KVM: couldn't grab cpu 125
2020-07-30T16:44:18+00:00 obs-power8-04 kernel: [30100.809046] KVM: couldn't grab cpu 125
2020-07-30T16:44:18+00:00 obs-power8-04 kernel: [30101.001097] KVM: couldn't grab cpu 61
2020-07-30T16:44:19+00:00 obs-power8-04 kernel: [30102.109007] KVM: couldn't grab cpu 125
2020-07-30T16:44:19+00:00 obs-power8-04 kernel: [30102.254470] KVM: couldn't grab cpu 61
I'm not sure where to start with reproducing, I'll have to try. How many
vCPUs in the guests? Do you have several guests running at once?
Thanks,
Nick
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2020-08-31 10:53:22
Michal Suchánek [off-list ref] writes:
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
cheers
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2020-08-31 13:02:16
Ruediger Oertel [off-list ref] writes:
Am 31.08.20 um 03:14 schrieb Nicholas Piggin:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately. Do the messages always come in pairs of CPUs?
I'm not sure where to start with reproducing, I'll have to try. How many
vCPUs in the guests? Do you have several guests running at once?
Hello all,
some details on the setup...
these machines are buildservice workers, (build.opensuse.org) and all they
do is spawn new VMs, run a package building job inside (rpmbuild, debbuild,...)
The machines are running in OPAL/PowerNV mode, with "ppc64_cpu --smt=off".
The number of VMs varies across the machines:
obs-power8-01: 18 instances, "-smp 16,threads=8"
obs-power8-02: 20 instances, "-smp 8,threads=8"
obs-power8-03: 30 instances, "-smp 8,threads=8"
obs-power8-04: 20 instances, "-smp 8,threads=8"
Can you send us the output of:
# grep . /sys/module/kvm_hv/parameters/*
cheers
Am 31.08.20 um 14:58 schrieb Michael Ellerman:
[...]
quoted
The machines are running in OPAL/PowerNV mode, with "ppc64_cpu --smt=off".
The number of VMs varies across the machines:
obs-power8-01: 18 instances, "-smp 16,threads=8"
obs-power8-02: 20 instances, "-smp 8,threads=8"
obs-power8-03: 30 instances, "-smp 8,threads=8"
obs-power8-04: 20 instances, "-smp 8,threads=8"
Can you send us the output of:
# grep . /sys/module/kvm_hv/parameters/*
of course, the current values are:
/sys/module/kvm_hv/parameters/dynamic_mt_modes:6
/sys/module/kvm_hv/parameters/h_ipi_redirect:1
/sys/module/kvm_hv/parameters/indep_threads_mode:Y
/sys/module/kvm_hv/parameters/kvm_irq_bypass:1
/sys/module/kvm_hv/parameters/nested:Y
/sys/module/kvm_hv/parameters/one_vm_per_core:N
/sys/module/kvm_hv/parameters/target_smt_mode:0
(actually identical on all 5 above)
--
with kind regards (mit freundlichem Grinsen),
Ruediger Oertel (ro@suse.com,ro@suse.de,bugfinder@t-online.de)
--------Do-Not-Accept-Binary-Blobs.----Ever.----From-Anyone.------------
Key fingerprint = 17DC 6553 86A7 384B 53C5 CA5C 3CE4 F2E7 23F2 B417
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg,
Germany, (HRB 36809, AG Nürnberg), Geschäftsführer: Felix Imendörffer
From: Nicholas Piggin <npiggin@gmail.com> Date: 2020-09-07 13:16:48
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
I don't know if there's much we can add to debug it. Can we run pdbg
on the BMCs on these things?
Thanks,
Nick
From: Michal Suchánek <hidden> Date: 2020-09-07 13:27:58
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
quoted
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
I don't know if there's much we can add to debug it. Can we run pdbg
on the BMCs on these things?
I suppose it depends on the machine type?
Thanks
Michal
From: Michal Suchánek <hidden> Date: 2020-10-16 20:16:10
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
quoted
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
From: Nicholas Piggin <npiggin@gmail.com> Date: 2020-10-19 01:03:11
Excerpts from Michal Suchánek's message of October 17, 2020 6:14 am:
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
quoted
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
Hello,
I tried running some KVM guests to simulate the workload and what I get
is guests failing to start with a rcu stall. Tried both 5.3 and 5.9
kernel and qemu 4.2.1 and 5.1.0
To start some guests I run
for i in $(seq 0 9) ; do /opt/qemu/bin/qemu-system-ppc64 -m 2048 -accel kvm -smp 8 -kernel /boot/vmlinux -initrd /boot/initrd -nodefaults -nographic -serial mon:telnet::444$i,server,wait & done
To simulate some workload I run
xz -zc9T0 < /dev/zero > /dev/null &
while true; do
killall -STOP xz; sleep 1; killall -CONT xz; sleep 1;
done &
on the host and add a job that executes this to the ramdisk. However, most
guests never get to the point where the job is executed.
Any idea what might be the problem?
I would say try without pv queued spin locks (but if the same thing is
happening with 5.3 then it must be something else I guess).
I'll try to test a similar setup on a POWER8 here.
Thanks,
Nick
From: Nicholas Piggin <npiggin@gmail.com> Date: 2020-10-19 04:52:35
Excerpts from Nicholas Piggin's message of October 19, 2020 11:00 am:
Excerpts from Michal Suchánek's message of October 17, 2020 6:14 am:
quoted
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
quoted
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
Hello,
I tried running some KVM guests to simulate the workload and what I get
is guests failing to start with a rcu stall. Tried both 5.3 and 5.9
kernel and qemu 4.2.1 and 5.1.0
To start some guests I run
for i in $(seq 0 9) ; do /opt/qemu/bin/qemu-system-ppc64 -m 2048 -accel kvm -smp 8 -kernel /boot/vmlinux -initrd /boot/initrd -nodefaults -nographic -serial mon:telnet::444$i,server,wait & done
To simulate some workload I run
xz -zc9T0 < /dev/zero > /dev/null &
while true; do
killall -STOP xz; sleep 1; killall -CONT xz; sleep 1;
done &
on the host and add a job that executes this to the ramdisk. However, most
guests never get to the point where the job is executed.
Any idea what might be the problem?
I would say try without pv queued spin locks (but if the same thing is
happening with 5.3 then it must be something else I guess).
I'll try to test a similar setup on a POWER8 here.
Couldn't reproduce the guest hang, they seem to run fine even with
queued spinlocks. Might have a different .config.
I might have got a lockup in the host (although different symptoms than
the original report). I'll look into that a bit further.
Thanks,
Nick
From: Michal Suchánek <hidden> Date: 2020-10-21 17:02:23
On Mon, Oct 19, 2020 at 02:50:51PM +1000, Nicholas Piggin wrote:
Excerpts from Nicholas Piggin's message of October 19, 2020 11:00 am:
quoted
Excerpts from Michal Suchánek's message of October 17, 2020 6:14 am:
quoted
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
quoted
quoted
quoted
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
Hello,
I tried running some KVM guests to simulate the workload and what I get
is guests failing to start with a rcu stall. Tried both 5.3 and 5.9
kernel and qemu 4.2.1 and 5.1.0
To start some guests I run
for i in $(seq 0 9) ; do /opt/qemu/bin/qemu-system-ppc64 -m 2048 -accel kvm -smp 8 -kernel /boot/vmlinux -initrd /boot/initrd -nodefaults -nographic -serial mon:telnet::444$i,server,wait & done
To simulate some workload I run
xz -zc9T0 < /dev/zero > /dev/null &
while true; do
killall -STOP xz; sleep 1; killall -CONT xz; sleep 1;
done &
on the host and add a job that executes this to the ramdisk. However, most
guests never get to the point where the job is executed.
Any idea what might be the problem?
I would say try without pv queued spin locks (but if the same thing is
happening with 5.3 then it must be something else I guess).
I'll try to test a similar setup on a POWER8 here.
Couldn't reproduce the guest hang, they seem to run fine even with
queued spinlocks. Might have a different .config.
I might have got a lockup in the host (although different symptoms than
the original report). I'll look into that a bit further.
Hello,
attaching the config for reference.
I am running with --smt=off because KVM refuses to start VMs otherwise.
Thanks
Michal
From: Michal Suchánek <hidden> Date: 2021-01-14 12:44:38
On Mon, Oct 19, 2020 at 02:50:51PM +1000, Nicholas Piggin wrote:
Excerpts from Nicholas Piggin's message of October 19, 2020 11:00 am:
quoted
Excerpts from Michal Suchánek's message of October 17, 2020 6:14 am:
quoted
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
quoted
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
Hello,
I tried running some KVM guests to simulate the workload and what I get
is guests failing to start with a rcu stall. Tried both 5.3 and 5.9
kernel and qemu 4.2.1 and 5.1.0
To start some guests I run
for i in $(seq 0 9) ; do /opt/qemu/bin/qemu-system-ppc64 -m 2048 -accel kvm -smp 8 -kernel /boot/vmlinux -initrd /boot/initrd -nodefaults -nographic -serial mon:telnet::444$i,server,wait & done
To simulate some workload I run
xz -zc9T0 < /dev/zero > /dev/null &
while true; do
killall -STOP xz; sleep 1; killall -CONT xz; sleep 1;
done &
on the host and add a job that executes this to the ramdisk. However, most
guests never get to the point where the job is executed.
Any idea what might be the problem?
I would say try without pv queued spin locks (but if the same thing is
happening with 5.3 then it must be something else I guess).
I'll try to test a similar setup on a POWER8 here.
Couldn't reproduce the guest hang, they seem to run fine even with
queued spinlocks. Might have a different .config.
I might have got a lockup in the host (although different symptoms than
the original report). I'll look into that a bit further.
Hello,
any progress on this?
I considered reinstating the old assembly code for POWER[78] but even
the way it's called has changed slightly.
Thanks
Michal
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-01-14 13:12:08
Excerpts from Michal Suchánek's message of January 14, 2021 10:40 pm:
On Mon, Oct 19, 2020 at 02:50:51PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Nicholas Piggin's message of October 19, 2020 11:00 am:
quoted
Excerpts from Michal Suchánek's message of October 17, 2020 6:14 am:
quoted
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
quoted
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
Hello,
I tried running some KVM guests to simulate the workload and what I get
is guests failing to start with a rcu stall. Tried both 5.3 and 5.9
kernel and qemu 4.2.1 and 5.1.0
To start some guests I run
for i in $(seq 0 9) ; do /opt/qemu/bin/qemu-system-ppc64 -m 2048 -accel kvm -smp 8 -kernel /boot/vmlinux -initrd /boot/initrd -nodefaults -nographic -serial mon:telnet::444$i,server,wait & done
To simulate some workload I run
xz -zc9T0 < /dev/zero > /dev/null &
while true; do
killall -STOP xz; sleep 1; killall -CONT xz; sleep 1;
done &
on the host and add a job that executes this to the ramdisk. However, most
guests never get to the point where the job is executed.
Any idea what might be the problem?
I would say try without pv queued spin locks (but if the same thing is
happening with 5.3 then it must be something else I guess).
I'll try to test a similar setup on a POWER8 here.
Couldn't reproduce the guest hang, they seem to run fine even with
queued spinlocks. Might have a different .config.
I might have got a lockup in the host (although different symptoms than
the original report). I'll look into that a bit further.
Hello,
any progress on this?
No progress, I still wasn't able to reproduce, and it fell off the
radar sorry.
I expect hwthred_state must be getting corrupted somewhere or a
secondary thread getting stuck but I couldn't see where. I try pick
it up again thanks for the reminder.
Thanks,
Nick
From: Michal Suchánek <hidden> Date: 2021-11-02 15:49:25
On Thu, Jan 14, 2021 at 11:08:03PM +1000, Nicholas Piggin wrote:
Excerpts from Michal Suchánek's message of January 14, 2021 10:40 pm:
quoted
On Mon, Oct 19, 2020 at 02:50:51PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Nicholas Piggin's message of October 19, 2020 11:00 am:
quoted
Excerpts from Michal Suchánek's message of October 17, 2020 6:14 am:
quoted
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
quoted
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
Hello,
I tried running some KVM guests to simulate the workload and what I get
is guests failing to start with a rcu stall. Tried both 5.3 and 5.9
kernel and qemu 4.2.1 and 5.1.0
To start some guests I run
for i in $(seq 0 9) ; do /opt/qemu/bin/qemu-system-ppc64 -m 2048 -accel kvm -smp 8 -kernel /boot/vmlinux -initrd /boot/initrd -nodefaults -nographic -serial mon:telnet::444$i,server,wait & done
To simulate some workload I run
xz -zc9T0 < /dev/zero > /dev/null &
while true; do
killall -STOP xz; sleep 1; killall -CONT xz; sleep 1;
done &
on the host and add a job that executes this to the ramdisk. However, most
guests never get to the point where the job is executed.
Any idea what might be the problem?
I would say try without pv queued spin locks (but if the same thing is
happening with 5.3 then it must be something else I guess).
I'll try to test a similar setup on a POWER8 here.
Couldn't reproduce the guest hang, they seem to run fine even with
queued spinlocks. Might have a different .config.
I might have got a lockup in the host (although different symptoms than
the original report). I'll look into that a bit further.
Hello,
any progress on this?
No progress, I still wasn't able to reproduce, and it fell off the
radar sorry.
I expect hwthred_state must be getting corrupted somewhere or a
secondary thread getting stuck but I couldn't see where. I try pick
it up again thanks for the reminder.
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-11-05 01:48:15
Excerpts from Michal Suchánek's message of November 3, 2021 1:48 am:
On Thu, Jan 14, 2021 at 11:08:03PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of January 14, 2021 10:40 pm:
quoted
On Mon, Oct 19, 2020 at 02:50:51PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Nicholas Piggin's message of October 19, 2020 11:00 am:
quoted
Excerpts from Michal Suchánek's message of October 17, 2020 6:14 am:
quoted
On Mon, Sep 07, 2020 at 11:13:47PM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michael Ellerman's message of August 31, 2020 8:50 pm:
quoted
Michal Suchánek [off-list ref] writes:
quoted
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
quoted
Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
quoted
Hello,
on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
Reimplement book3s idle code in C").
The symptom is host locking up completely after some hours of KVM
workload with messages like
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
printed before the host locks up.
The machines run sandboxed builds which is a mixed workload resulting in
IO/single core/mutiple core load over time and there are periods of no
activity and no VMS runnig as well. The VMs are shortlived so VM
setup/terdown is somewhat excercised as well.
POWER9 with the new guest entry fast path does not seem to be affected.
Reverted the patch and the followup idle fixes on top of 5.2.14 and
re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
after idle") which gives same idle code as 5.1.16 and the kernel seems
stable.
Config is attached.
I cannot easily revert this commit, especially if I want to use the same
kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
only to the new idle code.
Any idea what can be the problem?
So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
those threads. I wonder what they are doing. POWER8 doesn't have a good
NMI IPI and I don't know if it supports pdbg dumping registers from the
BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Your firmware won't support it AFAIK.
You could try kdump, but if we have CPUs stuck in KVM then there's a
good chance it won't work :/
I haven't had any luck yet reproducing this still. Testing with sub
cores of various different combinations, etc. I'll keep trying though.
Hello,
I tried running some KVM guests to simulate the workload and what I get
is guests failing to start with a rcu stall. Tried both 5.3 and 5.9
kernel and qemu 4.2.1 and 5.1.0
To start some guests I run
for i in $(seq 0 9) ; do /opt/qemu/bin/qemu-system-ppc64 -m 2048 -accel kvm -smp 8 -kernel /boot/vmlinux -initrd /boot/initrd -nodefaults -nographic -serial mon:telnet::444$i,server,wait & done
To simulate some workload I run
xz -zc9T0 < /dev/zero > /dev/null &
while true; do
killall -STOP xz; sleep 1; killall -CONT xz; sleep 1;
done &
on the host and add a job that executes this to the ramdisk. However, most
guests never get to the point where the job is executed.
Any idea what might be the problem?
I would say try without pv queued spin locks (but if the same thing is
happening with 5.3 then it must be something else I guess).
I'll try to test a similar setup on a POWER8 here.
Couldn't reproduce the guest hang, they seem to run fine even with
queued spinlocks. Might have a different .config.
I might have got a lockup in the host (although different symptoms than
the original report). I'll look into that a bit further.
Hello,
any progress on this?
No progress, I still wasn't able to reproduce, and it fell off the
radar sorry.
I expect hwthred_state must be getting corrupted somewhere or a
secondary thread getting stuck but I couldn't see where. I try pick
it up again thanks for the reminder.