On nestedv2 the L1 converts a pending doorbell into guest DPDES state at
the top of kvmhv_vcpu_entry_nestedv2() and immediately forgets about it:
if (vcpu->arch.doorbell_request) {
vcpu->arch.doorbell_request = 0;
kvmppc_set_dpdes(vcpu, 1);
}
Clearing 'doorbell_request' at this point assumes that handing DPDES to the
L0 is equivalent to the L2 having taken the doorbell. That is not true, and
the doorbell can be lost in two ways:
- The block runs before the lazy_irq_pending() check, so the doorbell is
consumed even on the path that returns 0 without ever calling
H_GUEST_RUN_VCPU.
- DPDES stays pending in the L2 until it is actually delivered. The L2
may exit for an unrelated reason (hcall, page fault, HDEC) with the
doorbell still set, typically because it was running with MSR[EE]=0.
Nothing reloads DPDES afterwards, so the L1 never learns this.
Once 'doorbell_request' has been cleared, the L1 has no record of the
pending doorbell. kvmppc_doorbell_pending() returns false, so
kvmppc_read_dpdes() reports the target thread as idle when a sibling vCPU
emulates 'mfspr DPDES', and the vCPU can be treated as having no work
pending and blocked. From the L2's point of view the doorbell is silently
lost, which shows up as an SMT guest hanging on a doorbell-based IPI.
Fix this by making 'doorbell_request' track the L2's DPDES rather than
being consumed by entry:
- inject DPDES after the early-return paths and before
kvmhv_nestedv2_flush_vcpu() serializes it into the vcpu run input
buffer, and no longer clear 'doorbell_request' there,
- after H_GUEST_RUN_VCPU, reload DPDES from the L0. The run output only
carries the state the L0 chose to return and the 'valids' bitmap is
zeroed on exit, so an explicit kvmhv_nestedv2_cached_reload() is
needed to see the L2's current value,
- if DPDES is still set the doorbell was not delivered, so keep
'doorbell_request' pending so that it is re-injected on the next
entry; otherwise clear it.
This keeps a pending doorbell visible to the L1 for as long as the L2 has
not consumed it, so vCPU wakeup and DPDES emulation on sibling vCPUs stay
consistent with the L2's actual state.
Fixes: 54ec2bd9e017 ("KVM: PPC: Book3S HV nestedv2: Fix doorbell emulation")
Signed-off-by: Vaibhav Jain <redacted>
Assisted-by: Claude:Opus-5
---
arch/powerpc/kvm/book3s_hv.c | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
@@ -4296,6 +4295,17 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,if(rc<0)return-EINVAL;+/* Check if privileged door bell was requested and handled */+if(vcpu->arch.vcore->dpdes){+kvmhv_nestedv2_cached_reload(vcpu,KVMPPC_GSID_DPDES);+if(vcpu->arch.vcore->dpdes)+vcpu->arch.doorbell_request|=vcpu->arch.vcore->dpdes;+else+cpu->arch.doorbell_request=0;+}else{+vcpu->arch.doorbell_request=0;+}+timer_rearm_host_dec(*tb);/* Record context switch and guest_run_time data */
On Mon, Aug 03, 2026 at 09:14:25AM +0530, Vaibhav Jain wrote:
quoted hunk
On nestedv2 the L1 converts a pending doorbell into guest DPDES state at
the top of kvmhv_vcpu_entry_nestedv2() and immediately forgets about it:
if (vcpu->arch.doorbell_request) {
vcpu->arch.doorbell_request = 0;
kvmppc_set_dpdes(vcpu, 1);
}
Clearing 'doorbell_request' at this point assumes that handing DPDES to the
L0 is equivalent to the L2 having taken the doorbell. That is not true, and
the doorbell can be lost in two ways:
- The block runs before the lazy_irq_pending() check, so the doorbell is
consumed even on the path that returns 0 without ever calling
H_GUEST_RUN_VCPU.
- DPDES stays pending in the L2 until it is actually delivered. The L2
may exit for an unrelated reason (hcall, page fault, HDEC) with the
doorbell still set, typically because it was running with MSR[EE]=0.
Nothing reloads DPDES afterwards, so the L1 never learns this.
Once 'doorbell_request' has been cleared, the L1 has no record of the
pending doorbell. kvmppc_doorbell_pending() returns false, so
kvmppc_read_dpdes() reports the target thread as idle when a sibling vCPU
emulates 'mfspr DPDES', and the vCPU can be treated as having no work
pending and blocked. From the L2's point of view the doorbell is silently
lost, which shows up as an SMT guest hanging on a doorbell-based IPI.
Fix this by making 'doorbell_request' track the L2's DPDES rather than
being consumed by entry:
- inject DPDES after the early-return paths and before
kvmhv_nestedv2_flush_vcpu() serializes it into the vcpu run input
buffer, and no longer clear 'doorbell_request' there,
- after H_GUEST_RUN_VCPU, reload DPDES from the L0. The run output only
carries the state the L0 chose to return and the 'valids' bitmap is
zeroed on exit, so an explicit kvmhv_nestedv2_cached_reload() is
needed to see the L2's current value,
- if DPDES is still set the doorbell was not delivered, so keep
'doorbell_request' pending so that it is re-injected on the next
entry; otherwise clear it.
This keeps a pending doorbell visible to the L1 for as long as the L2 has
not consumed it, so vCPU wakeup and DPDES emulation on sibling vCPUs stay
consistent with the L2's actual state.
Fixes: 54ec2bd9e017 ("KVM: PPC: Book3S HV nestedv2: Fix doorbell emulation")
Signed-off-by: Vaibhav Jain <redacted>
Assisted-by: Claude:Opus-5
---
arch/powerpc/kvm/book3s_hv.c | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
@@ -4296,6 +4295,17 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,if(rc<0)return-EINVAL;+/* Check if privileged door bell was requested and handled */+if(vcpu->arch.vcore->dpdes){+kvmhv_nestedv2_cached_reload(vcpu,KVMPPC_GSID_DPDES);+if(vcpu->arch.vcore->dpdes)+vcpu->arch.doorbell_request|=vcpu->arch.vcore->dpdes;+else+cpu->arch.doorbell_request=0;
This is a compile error, should be "vcpu->"
quoted hunk
+ } else {+ vcpu->arch.doorbell_request = 0;+ }+
I think this entire block after vcpu exit can be reduced to just
if (vcpu->arch.doorbell_request)
vcpu->arch.doorbell_request = kvmppc_get_dpdes(vcpu);
- Gautam
On Mon, Aug 10, 2026 at 04:54:18PM +0530, Gautam Menghani wrote:
On Mon, Aug 03, 2026 at 09:14:25AM +0530, Vaibhav Jain wrote:
quoted
On nestedv2 the L1 converts a pending doorbell into guest DPDES state at
the top of kvmhv_vcpu_entry_nestedv2() and immediately forgets about it:
if (vcpu->arch.doorbell_request) {
vcpu->arch.doorbell_request = 0;
kvmppc_set_dpdes(vcpu, 1);
}
Clearing 'doorbell_request' at this point assumes that handing DPDES to the
L0 is equivalent to the L2 having taken the doorbell. That is not true, and
the doorbell can be lost in two ways:
- The block runs before the lazy_irq_pending() check, so the doorbell is
consumed even on the path that returns 0 without ever calling
H_GUEST_RUN_VCPU.
- DPDES stays pending in the L2 until it is actually delivered. The L2
may exit for an unrelated reason (hcall, page fault, HDEC) with the
doorbell still set, typically because it was running with MSR[EE]=0.
Nothing reloads DPDES afterwards, so the L1 never learns this.
Once 'doorbell_request' has been cleared, the L1 has no record of the
pending doorbell. kvmppc_doorbell_pending() returns false, so
kvmppc_read_dpdes() reports the target thread as idle when a sibling vCPU
emulates 'mfspr DPDES', and the vCPU can be treated as having no work
pending and blocked. From the L2's point of view the doorbell is silently
lost, which shows up as an SMT guest hanging on a doorbell-based IPI.
Fix this by making 'doorbell_request' track the L2's DPDES rather than
being consumed by entry:
- inject DPDES after the early-return paths and before
kvmhv_nestedv2_flush_vcpu() serializes it into the vcpu run input
buffer, and no longer clear 'doorbell_request' there,
- after H_GUEST_RUN_VCPU, reload DPDES from the L0. The run output only
carries the state the L0 chose to return and the 'valids' bitmap is
zeroed on exit, so an explicit kvmhv_nestedv2_cached_reload() is
needed to see the L2's current value,
- if DPDES is still set the doorbell was not delivered, so keep
'doorbell_request' pending so that it is re-injected on the next
entry; otherwise clear it.
This keeps a pending doorbell visible to the L1 for as long as the L2 has
not consumed it, so vCPU wakeup and DPDES emulation on sibling vCPUs stay
consistent with the L2's actual state.
Fixes: 54ec2bd9e017 ("KVM: PPC: Book3S HV nestedv2: Fix doorbell emulation")
Signed-off-by: Vaibhav Jain <redacted>
Assisted-by: Claude:Opus-5
---
arch/powerpc/kvm/book3s_hv.c | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
@@ -4296,6 +4295,17 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,if(rc<0)return-EINVAL;+/* Check if privileged door bell was requested and handled */+if(vcpu->arch.vcore->dpdes){+kvmhv_nestedv2_cached_reload(vcpu,KVMPPC_GSID_DPDES);+if(vcpu->arch.vcore->dpdes)+vcpu->arch.doorbell_request|=vcpu->arch.vcore->dpdes;+else+cpu->arch.doorbell_request=0;
This is a compile error, should be "vcpu->"
quoted
+ } else {+ vcpu->arch.doorbell_request = 0;+ }+
I think this entire block after vcpu exit can be reduced to just
if (vcpu->arch.doorbell_request)
vcpu->arch.doorbell_request = kvmppc_get_dpdes(vcpu);
Gave this a bit more thought - this can miss the doorbell that was sent
to this vcpu while the vcpu was running. I think the below code should
work fine:
if (vcpu->arch.doorbell_request) {
vcpu->arch.doorbell_request = 0;
kvmppc_set_dpdes(vcpu, 1);
}
rc = plpar_guest_run_vcpu(...);
/* Restore the doorbell status and also account for the doorbell
* received when this vcpu was running */
if (vcpu->arch.vcore->dpdes)
vcpu->arch.doorbell_request |= kvmppc_get_dpdes(vcpu);
- Gautam
On nestedv2 the L1 converts a pending doorbell into guest DPDES state at
the top of kvmhv_vcpu_entry_nestedv2() and immediately forgets about it:
if (vcpu->arch.doorbell_request) {
vcpu->arch.doorbell_request = 0;
kvmppc_set_dpdes(vcpu, 1);
}
Clearing 'doorbell_request' at this point assumes that handing DPDES to the
L0 is equivalent to the L2 having taken the doorbell. That is not true, and
the doorbell can be lost in two ways:
- The block runs before the lazy_irq_pending() check, so the doorbell is
consumed even on the path that returns 0 without ever calling
H_GUEST_RUN_VCPU.
- DPDES stays pending in the L2 until it is actually delivered. The L2
may exit for an unrelated reason (hcall, page fault, HDEC) with the
doorbell still set, typically because it was running with MSR[EE]=0.
Nothing reloads DPDES afterwards, so the L1 never learns this.
Once 'doorbell_request' has been cleared, the L1 has no record of the
pending doorbell. kvmppc_doorbell_pending() returns false, so
kvmppc_read_dpdes() reports the target thread as idle when a sibling vCPU
emulates 'mfspr DPDES', and the vCPU can be treated as having no work
pending and blocked. From the L2's point of view the doorbell is silently
lost, which shows up as an SMT guest hanging on a doorbell-based IPI.
Fix this by making 'doorbell_request' track the L2's DPDES rather than
being consumed by entry:
- inject DPDES after the early-return paths and before
kvmhv_nestedv2_flush_vcpu() serializes it into the vcpu run input
buffer, and no longer clear 'doorbell_request' there,
- after H_GUEST_RUN_VCPU, reload DPDES from the L0. The run output only
carries the state the L0 chose to return and the 'valids' bitmap is
zeroed on exit, so an explicit kvmhv_nestedv2_cached_reload() is
needed to see the L2's current value,
- if DPDES is still set the doorbell was not delivered, so keep
'doorbell_request' pending so that it is re-injected on the next
entry; otherwise clear it.
This keeps a pending doorbell visible to the L1 for as long as the L2 has
not consumed it, so vCPU wakeup and DPDES emulation on sibling vCPUs stay
consistent with the L2's actual state.
Fixes: 54ec2bd9e017 ("KVM: PPC: Book3S HV nestedv2: Fix doorbell emulation")
Signed-off-by: Vaibhav Jain <redacted>
Assisted-by: Claude:Opus-5
---
arch/powerpc/kvm/book3s_hv.c | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
@@ -4296,6 +4295,17 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,if(rc<0)return-EINVAL;+/* Check if privileged door bell was requested and handled */+if(vcpu->arch.vcore->dpdes){+kvmhv_nestedv2_cached_reload(vcpu,KVMPPC_GSID_DPDES);+if(vcpu->arch.vcore->dpdes)+vcpu->arch.doorbell_request|=vcpu->arch.vcore->dpdes;+else+cpu->arch.doorbell_request=0;+}else{+vcpu->arch.doorbell_request=0;+}+timer_rearm_host_dec(*tb);/* Record context switch and guest_run_time data */
Hi Vaibhav,
I have tested this patch and it is still giving me the issue I reported, so here is my analysis:
a) Without applying the patch :
1) Start the guest and run stress-ng as below for sometime
localhost:~ # stress-ng --cpu 4 --vm 2 --vm-bytes 1G --hdd 2 --hdd-bytes 1G --sched other --timeout 3600000s
stress-ng: info: [1464] setting to a 41 days, 16 hours, 0 secs run per stressor
stress-ng: info: [1464] dispatching hogs: 4 cpu, 2 vm, 2 hdd
2) Start the migration from H1 to H2:
ltc-lp7:~ # virsh migrate --live --domain sles16_anu qemu+ssh://10.xx.xx.xx/system --verbose --undefinesource --persistent --auto-converge --postcopy
(root@10.xx.xx.xx) Password:
Migration: [100.00 %]
3) Migration got completed but guest is not getting recovered from continuous softlockups
[ 1336.003836][ C1] watchdog: BUG: soft lockup - CPU#1 stuck for 977s! [htxd_monitor:1337]
[ 1336.006834][ C4] watchdog: BUG: soft lockup - CPU#4 stuck for 1002s! [rcu_exp_par_gp_:19]
[ 1346.015839][ C0] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 1090s!
[ 1346.016355][ C0] BUG: workqueue lockup - pool cpus=3 node=0 flags=0x0 nice=0 stuck for 1107s!
[ 1346.016874][ C0] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 1093s!
[ 1356.007835][ C6] watchdog: BUG: soft lockup - CPU#6 stuck for 912s! [systemd:1353]
[ 1356.008835][ C7] watchdog: BUG: soft lockup - CPU#7 stuck for 998s! [systemd-journal:570]
[ 1360.003836][ C1] watchdog: BUG: soft lockup - CPU#1 stuck for 999s! [htxd_monitor:1337]
[ 1360.006834][ C4] watchdog: BUG: soft lockup - CPU#4 stuck for 1024s! [rcu_exp_par_gp_:19]
[ 1368.933835][ C4] rcu: INFO: rcu_preempt self-detected stall on CPU
[ 1368.933973][ C4] rcu: 4-....: (1129830 ticks this GP) idle=afc4/1/0x4000000000000002 softirq=3694/428556 fqs=259639
[ 1368.934106][ C4] rcu: hardirqs softirqs csw/system
[ 1368.934188][ C4] rcu: number: 1 444039 0
[ 1368.934271][ C4] rcu: cputime: 3 8 1096165 ==> 1110021(ms)
[ 1368.934373][ C4] rcu: (t=1140022 jiffies g=6177 q=1684 ncpus=8)
[ 1376.224839][ C0] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 1120s!
[ 1376.225307][ C0] BUG: workqueue lockup - pool cpus=3 node=0 flags=0x0 nice=0 stuck for 1138s!
[ 1376.225428][ C0] BUG: workqueue lockup - pool cpus=5 node=0 flags=0x0 nice=0 stuck for 715s!
[ 1376.225548][ C0] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x0 nice=0 stuck for 1027s!
[ 1376.225667][ C0] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 1123s!
[ 1444.006835][ C4] watchdog: BUG: soft lockup - CPU#4 stuck for 1100s! [rcu_exp_par_gp_:19]
b) Even after applying the patch also it is giving same softlockup issue as mentioned above:
Though I have enough vcpus and memory on the guest (16 vcpus , 13Gi of memory) and ample amount of memory and cpus
present on host still these softlockups are happening after applying the patch too. I tried reducing stress also on the guest
but still this issue is seen.
stress-ng --cpu 4 --vm 2 --vm-bytes 1G --hdd 2 --hdd-bytes 1G --sched other --timeout 3600000s
If you are planning to send next version of this patch,
Please do add my reported-by:
Reported-by: Anushree Mathur <redacted>
Though I have enough vcpus and memory on the guest (16 vcpus , 13Gi of
memory) and ample amount of memory and cpus
present on host still these softlockups are happening after applying the
patch too. I tried reducing stress also on the guest
but still this issue is seen.
stress-ng --cpu 4 --vm 2 --vm-bytes 1G --hdd 2 --hdd-bytes 1G --sched other
--timeout 3600000s
If you are planning to send next version of this patch,
Please do add my reported-by:
Reported-by: Anushree Mathur <redacted>
Though I have enough vcpus and memory on the guest (16 vcpus , 13Gi of
memory) and ample amount of memory and cpus
present on host still these softlockups are happening after applying the
patch too. I tried reducing stress also on the guest
but still this issue is seen.
stress-ng --cpu 4 --vm 2 --vm-bytes 1G --hdd 2 --hdd-bytes 1G --sched other
--timeout 3600000s
If you are planning to send next version of this patch,
Please do add my reported-by:
Reported-by: Anushree Mathur <redacted>
Hi Amit,
Yes I have tested this patch too and I am still seeing the issue. Let me
reply back for this v2 patch also.
Thank you,
Anushree Mathur