Hi folks,
Changes since v1 [2]:
- Dropped the guard()/scoped_guard() conversion patches: standalone churn
on code this series does not otherwise rework. (Marc)
- Rebased onto kvmarm/next. The VGIC flush primitive now bounds used_lrs
using the cached hyp_gicv3_nr_lr instead of reading ICH_VTR_EL2 on every
entry. (Marc)
- Grouped the PKVM_HOST_STATE_DIRTY flag with the other iflags and
clarified its comment. (Marc)
- Sync PSTATE alongside PC on every non-protected exit, and sync+dirty
before host-side SError injection so the syndrome is not dropped. (sashiko)
- Various cleanups and tidying up. (Vincent)
Building on Will's pKVM infrastructure series [1], this series reworks
how pKVM moves vCPU state between the host and EL2, and stops copying a
non-protected guest's state on every world switch.
EL2 gains proper primitives for the state it transfers: vCPU lookup
helpers, and VGIC flush/sync that reduces how much host state EL2
dereferences. The series also moves some preparatory code (such as sys
reg access and PSCI helpers) to shared headers and HYP, and implements
lazy copying of a non-protected guest's register state back to the host
until the host actually needs it, instead of on every exit.
This is the first of two series moving pKVM vCPU state management to
EL2. The follow-up completes the job for protected VMs: state
isolation, PSCI handling at EL2, and the resulting API behaviour.
The series is structured as follows:
01-04: Preparatory refactoring (MPIDR, sys reg access, vCPU reset, PSCI
helpers) to shared headers and HYP.
05: Host and hypervisor vCPU lookup primitives.
06-07: VGIC: reduce EL2's exposure to host state, add flush/sync primitives.
08: Lazy state sync for non-protected guests.
Based on kvmarm/next.
[1] https://lore.kernel.org/all/20260105154939.11041-1-will@kernel.org/
[2] https://lore.kernel.org/all/20260612065925.755562-1-tabba@google.com/
Cheers,
/fuad
Fuad Tabba (5):
KVM: arm64: Extract MPIDR computation into a shared header
KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code
KVM: arm64: Factor out reusable vCPU reset helpers
KVM: arm64: Move PSCI helper functions to a shared header
KVM: arm64: Implement lazy vCPU state sync for non-protected guests
Marc Zyngier (3):
KVM: arm64: Add host and hypervisor vCPU lookup primitives
KVM: arm64: Minimise EL2's exposure of host VGIC state during world
switch
KVM: arm64: Add primitives to flush/sync the VGIC state at EL2
arch/arm64/include/asm/kvm_arm.h | 12 ++
arch/arm64/include/asm/kvm_asm.h | 1 +
arch/arm64/include/asm/kvm_emulate.h | 79 +++++++-
arch/arm64/include/asm/kvm_host.h | 2 +
arch/arm64/kvm/arm.c | 7 +
arch/arm64/kvm/handle_exit.c | 30 ++++
arch/arm64/kvm/hyp/exception.c | 34 +---
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 258 +++++++++++++++++++++++----
arch/arm64/kvm/psci.c | 30 +---
arch/arm64/kvm/reset.c | 60 +------
arch/arm64/kvm/sys_regs.c | 14 +-
arch/arm64/kvm/sys_regs.h | 19 ++
include/kvm/arm_psci.h | 27 +++
13 files changed, 410 insertions(+), 163 deletions(-)
--
2.55.0.rc0.738.g0c8ab3ebcc-goog
Extract the vCPU MPIDR computation embedded in reset_mpidr() into a
kvm_calculate_mpidr() inline in sys_regs.h, so it can be computed
without duplicating the logic. A follow-up series reuses it to reset
protected vCPUs at EL2.
No functional change intended.
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/kvm/sys_regs.c | 14 +-------------
arch/arm64/kvm/sys_regs.h | 19 +++++++++++++++++++
2 files changed, 20 insertions(+), 13 deletions(-)
The vcpu_{read,write}_sys_reg() accessors are host-only, so helpers
built on them such as kvm_vcpu_set_be()/kvm_vcpu_is_be() cannot be
shared with hyp code. exception.c already wraps them in
__vcpu_{read,write}_sys_reg(), which pick the host- or hyp-side accessor
via has_vhe() and so are valid in any context.
Move those wrappers to kvm_emulate.h as kvm_vcpu_{read,write}_sys_reg()
and switch the callers over, so a follow-up series can share that
emulation code at EL2.
No functional change intended.
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/include/asm/kvm_emulate.h | 22 +++++++++++++++---
arch/arm64/kvm/hyp/exception.c | 34 ++++++++--------------------
2 files changed, 28 insertions(+), 28 deletions(-)
Pull the reusable pieces out of kvm_reset_vcpu(): expose the reset
PSTATE values in kvm_arm.h, and split the core register reset and the
PSCI-driven reset into kvm_reset_vcpu_core() and kvm_reset_vcpu_psci().
A follow-up series reuses these to reset protected vCPUs at EL2.
No functional change intended.
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/include/asm/kvm_arm.h | 12 ++++++
arch/arm64/include/asm/kvm_emulate.h | 57 ++++++++++++++++++++++++++
arch/arm64/kvm/reset.c | 60 ++--------------------------
3 files changed, 72 insertions(+), 57 deletions(-)
Move kvm_psci_valid_affinity() and kvm_psci_narrow_to_32bit() from
psci.c to include/kvm/arm_psci.h, and move psci_affinity_mask() there
too, renaming it kvm_psci_affinity_mask() now that it is no longer
file-local. A follow-up series handles some protected-guest PSCI calls
at EL2 using these helpers.
No functional change intended.
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/kvm/psci.c | 30 +-----------------------------
include/kvm/arm_psci.h | 27 +++++++++++++++++++++++++++
2 files changed, 28 insertions(+), 29 deletions(-)
From: Marc Zyngier <maz@kernel.org>
The nVHE hypervisor repeatedly resolves a host vCPU into the EL2
address space and validates that the loaded hyp vCPU matches it, with
that logic open-coded in each handler.
Add __get_host_hyp_vcpus() and the get_host_hyp_vcpus() macro, which
translate the host vCPU into the hypervisor's address space and, when
pKVM is enabled, also return the loaded hyp vCPU if it matches. If pKVM
is enabled but the loaded hyp vCPU does not correspond to the requested
host vCPU, both the host and hyp vCPU are returned as NULL. Convert
handle___kvm_vcpu_run() to use it.
No functional change intended.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Co-developed-by: Fuad Tabba <redacted>
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 52 ++++++++++++++++++++++--------
1 file changed, 38 insertions(+), 14 deletions(-)
@@ -231,23 +262,16 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)gotoout;}-if(!hyp_vcpu){-ret=-EINVAL;-gotoout;-}-flush_hyp_vcpu(hyp_vcpu);ret=__kvm_vcpu_run(&hyp_vcpu->vcpu);sync_hyp_vcpu(hyp_vcpu);}else{-structkvm_vcpu*vcpu=kern_hyp_va(host_vcpu);-/* The host is fully trusted, run its vCPU directly. */-fpsimd_lazy_switch_to_guest(vcpu);-ret=__kvm_vcpu_run(vcpu);-fpsimd_lazy_switch_to_host(vcpu);+fpsimd_lazy_switch_to_guest(host_vcpu);+ret=__kvm_vcpu_run(host_vcpu);+fpsimd_lazy_switch_to_host(host_vcpu);}out:cpu_reg(host_ctxt,1)=ret;
From: Marc Zyngier <maz@kernel.org>
The host passes a vgic_v3_cpu_if pointer to the __vgic_v3_save_aprs and
__vgic_v3_restore_vmcr_aprs hypercalls, which EL2 dereferences
wholesale. That exposes the host's full VGIC emulation state to the
hypervisor, against pKVM's isolation goals.
Recover the host vCPU from the supplied cpu_if via container_of() and
copy only vgic_vmcr and the active priority registers between EL2's
hyp-side state and the host vCPU, so EL2 no longer dereferences the
host's vgic_v3_cpu_if directly.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Co-developed-by: Fuad Tabba <redacted>
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 67 ++++++++++++++++++++++++++++--
1 file changed, 63 insertions(+), 4 deletions(-)
@@ -506,16 +518,63 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt)staticvoidhandle___vgic_v3_save_aprs(structkvm_cpu_context*host_ctxt){-DECLARE_REG(structvgic_v3_cpu_if*,cpu_if,host_ctxt,1);+structpkvm_hyp_vcpu*hyp_vcpu;+structkvm_vcpu*host_vcpu;-__vgic_v3_save_aprs(kern_hyp_va(cpu_if));+host_vcpu=get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt,1,+&hyp_vcpu);+if(!host_vcpu)+return;++if(unlikely(hyp_vcpu)){+structvgic_v3_cpu_if*hyp_cpu_if,*host_cpu_if;+inti;++hyp_cpu_if=&hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;+__vgic_v3_save_aprs(hyp_cpu_if);++host_cpu_if=&host_vcpu->arch.vgic_cpu.vgic_v3;+host_cpu_if->vgic_vmcr=hyp_cpu_if->vgic_vmcr;+for(i=0;i<ARRAY_SIZE(host_cpu_if->vgic_ap0r);i++){+host_cpu_if->vgic_ap0r[i]=hyp_cpu_if->vgic_ap0r[i];+host_cpu_if->vgic_ap1r[i]=hyp_cpu_if->vgic_ap1r[i];+}+}else{+__vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);+}}staticvoidhandle___vgic_v3_restore_vmcr_aprs(structkvm_cpu_context*host_ctxt){-DECLARE_REG(structvgic_v3_cpu_if*,cpu_if,host_ctxt,1);+structpkvm_hyp_vcpu*hyp_vcpu;+structkvm_vcpu*host_vcpu;-__vgic_v3_restore_vmcr_aprs(kern_hyp_va(cpu_if));+host_vcpu=get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt,1,+&hyp_vcpu);+if(!host_vcpu)+return;++if(unlikely(hyp_vcpu)){+structvgic_v3_cpu_if*hyp_cpu_if,*host_cpu_if;+inti;++hyp_cpu_if=&hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;+host_cpu_if=&host_vcpu->arch.vgic_cpu.vgic_v3;++hyp_cpu_if->vgic_vmcr=host_cpu_if->vgic_vmcr;+/* Should be a one-off */+hyp_cpu_if->vgic_sre=(ICC_SRE_EL1_DIB|+ICC_SRE_EL1_DFB|+ICC_SRE_EL1_SRE);+for(i=0;i<ARRAY_SIZE(host_cpu_if->vgic_ap0r);i++){+hyp_cpu_if->vgic_ap0r[i]=host_cpu_if->vgic_ap0r[i];+hyp_cpu_if->vgic_ap1r[i]=host_cpu_if->vgic_ap1r[i];+}++__vgic_v3_restore_vmcr_aprs(hyp_cpu_if);+}else{+__vgic_v3_restore_vmcr_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);+}}staticvoidhandle___pkvm_init(structkvm_cpu_context*host_ctxt)
From: Marc Zyngier <maz@kernel.org>
pKVM performs its own world switch for protected VMs but has no
primitives to move the per-vCPU VGIC state between the host and
hypervisor vCPU contexts.
Add flush_hyp_vgic_state() and sync_hyp_vgic_state(). Flush copies
vgic_hcr, the in-use list registers and used_lrs from the host into the
hyp vCPU and pins vgic_sre to a fixed value; sync copies vgic_hcr,
vgic_vmcr and the in-use list registers back. The active priority
registers are handled separately by the save/restore-aprs path.
Bound used_lrs by hyp_gicv3_nr_lr, the cached implemented-LR count,
instead of reading ICH_VTR_EL2 on each entry. That clamps the
host-supplied value and avoids a per-entry sysreg read that is costly
under NV.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Co-developed-by: Fuad Tabba <redacted>
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 55 ++++++++++++++++++++++--------
1 file changed, 41 insertions(+), 14 deletions(-)
@@ -102,6 +102,45 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu)*host_data_ptr(fp_owner)=FP_STATE_HOST_OWNED;}+staticvoidflush_hyp_vgic_state(structpkvm_hyp_vcpu*hyp_vcpu)+{+structkvm_vcpu*host_vcpu=hyp_vcpu->host_vcpu;+structvgic_v3_cpu_if*host_cpu_if,*hyp_cpu_if;+unsignedintused_lrs,i;++host_cpu_if=&host_vcpu->arch.vgic_cpu.vgic_v3;+hyp_cpu_if=&hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;++used_lrs=host_cpu_if->used_lrs;+used_lrs=min(used_lrs,hyp_gicv3_nr_lr);++hyp_cpu_if->vgic_hcr=host_cpu_if->vgic_hcr;+/* Should be a one-off */+hyp_cpu_if->vgic_sre=(ICC_SRE_EL1_DIB|+ICC_SRE_EL1_DFB|+ICC_SRE_EL1_SRE);+hyp_cpu_if->used_lrs=used_lrs;++for(i=0;i<used_lrs;i++)+hyp_cpu_if->vgic_lr[i]=host_cpu_if->vgic_lr[i];+}++staticvoidsync_hyp_vgic_state(structpkvm_hyp_vcpu*hyp_vcpu)+{+structkvm_vcpu*host_vcpu=hyp_vcpu->host_vcpu;+structvgic_v3_cpu_if*host_cpu_if,*hyp_cpu_if;+unsignedinti;++host_cpu_if=&host_vcpu->arch.vgic_cpu.vgic_v3;+hyp_cpu_if=&hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;++host_cpu_if->vgic_hcr=hyp_cpu_if->vgic_hcr;+host_cpu_if->vgic_vmcr=hyp_cpu_if->vgic_vmcr;++for(i=0;i<hyp_cpu_if->used_lrs;i++)+host_cpu_if->vgic_lr[i]=hyp_cpu_if->vgic_lr[i];+}+staticvoidflush_debug_state(structpkvm_hyp_vcpu*hyp_vcpu){structkvm_vcpu*host_vcpu=hyp_vcpu->host_vcpu;
@@ -150,13 +189,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)hyp_vcpu->vcpu.arch.vsesr_el2=host_vcpu->arch.vsesr_el2;-hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3=host_vcpu->arch.vgic_cpu.vgic_v3;--/* Bound used_lrs by the number of implemented list registers. */-hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs=-min_t(unsignedint,-hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs,-hyp_gicv3_nr_lr);+flush_hyp_vgic_state(hyp_vcpu);hyp_vcpu->vcpu.arch.pid=host_vcpu->arch.pid;}
pKVM copies a non-protected guest's register context between the host
and the hypervisor on every world switch, even when the host never
inspects it. Defer the copy: on entry, flush the host context into the
hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on
exit, leave it in the hyp vCPU and copy it back only when the host needs
it, via a __pkvm_vcpu_sync_state hypercall on trap handling or at vcpu
put. A protected guest's context is copied as before, since lazy sync
only helps where the host is trusted to see the guest's registers.
PC and PSTATE are the exception: they are copied back on every exit so
the kvm_exit tracepoint reports the guest's real exit PC, and the run
loop's vcpu_mode_is_bad_32bit() and SError-masking checks evaluate the
guest's current PSTATE rather than the value left by the previous sync.
handle_exit_early() can also inject an SError, which writes the guest
context (ESR_EL1) outside the trap-handling path. For a non-protected
guest it therefore syncs the context from the hyp vCPU and marks it
dirty, as handle_trap_exceptions() does, so the injection reaches the
hyp vCPU on re-entry rather than being dropped.
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/include/asm/kvm_asm.h | 1 +
arch/arm64/include/asm/kvm_host.h | 2 +
arch/arm64/kvm/arm.c | 7 +++
arch/arm64/kvm/handle_exit.c | 30 +++++++++++
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 86 ++++++++++++++++++++++++++++--
5 files changed, 121 insertions(+), 5 deletions(-)
@@ -1068,6 +1068,8 @@ struct kvm_vcpu_arch {#define INCREMENT_PC __vcpu_single_flag(iflags, BIT(1))/* Target EL/MODE (not a single flag, but let's abuse the macro) */#define EXCEPT_MASK __vcpu_single_flag(iflags, GENMASK(3, 1))+/* Host-set: the hyp flushes the non-protected vCPU state in on entry */+#define PKVM_HOST_STATE_DIRTY __vcpu_single_flag(iflags, BIT(4))/* Helpers to encode exceptions with minimum fuss */#define __EXCEPT_MASK_VAL unpack_vcpu_flag(EXCEPT_MASK)
@@ -733,6 +733,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)if(is_protected_kvm_enabled()){kvm_call_hyp(__vgic_v3_save_aprs,&vcpu->arch.vgic_cpu.vgic_v3);kvm_call_hyp_nvhe(__pkvm_vcpu_put);++/* __pkvm_vcpu_put implies a sync of the state */+if(!kvm_vm_is_protected(vcpu->kvm))+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);}kvm_vcpu_put_debug(vcpu);
@@ -964,6 +968,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)returnret;if(is_protected_kvm_enabled()){+/* Start with the vcpu in a dirty state */+if(!kvm_vm_is_protected(vcpu->kvm))+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);ret=pkvm_create_hyp_vm(kvm);if(ret)returnret;
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu){inthandled;+/*+*Ifwerunanon-protectedVMwhenprotectionisenabled+*system-wide,resyncthestatefromthehypervisorandmark+*itasdirtyonthehostsideifitwasn'tdirtyalready+*(whichcouldhappenifpreemptionhastakenplace).+*/+if(is_protected_kvm_enabled()&&!kvm_vm_is_protected(vcpu->kvm)){+guard(preempt)();+if(!(vcpu_get_flag(vcpu,PKVM_HOST_STATE_DIRTY))){+kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);+}+}+/**SeeARMARMB1.14.1:"Hyp traps on instructions*thatfailtheirconditioncodecheck"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)/* For exit types that need handling before we can be preempted */voidhandle_exit_early(structkvm_vcpu*vcpu,intexception_index){+boolinject_serror=ARM_SERROR_PENDING(exception_index)||+ARM_EXCEPTION_CODE(exception_index)==ARM_EXCEPTION_EL1_SERROR;++/*+*AnSErrorinjectedbelowwritesthehostctxt;foranon-protected+*guest,syncfromthehypvCPUandkeepitdirtysoitisn'tdropped.+*/+if(is_protected_kvm_enabled()){+vcpu_clear_flag(vcpu,PKVM_HOST_STATE_DIRTY);++if(inject_serror&&!kvm_vm_is_protected(vcpu->kvm)){+kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);+}+}+if(ARM_SERROR_PENDING(exception_index)){if(this_cpu_has_cap(ARM64_HAS_RAS_EXTN)){u64disr=kvm_vcpu_get_disr(vcpu);
@@ -170,7 +212,17 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)fpsimd_sve_flush();flush_debug_state(hyp_vcpu);-hyp_vcpu->vcpu.arch.ctxt=host_vcpu->arch.ctxt;+/*+*Ifwedealwithanon-protectedguestandthestateispotentially+*dirty(fromahostperspective),copythestatebackintothehyp+*vcpu.+*/+if(!pkvm_hyp_vcpu_is_protected(hyp_vcpu)){+if(vcpu_get_flag(host_vcpu,PKVM_HOST_STATE_DIRTY))+flush_hyp_vcpu_state(hyp_vcpu);+}else{+hyp_vcpu->vcpu.arch.ctxt=host_vcpu->arch.ctxt;+}/* __hyp_running_vcpu must be NULL in a guest context. */hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu=NULL;
@@ -201,9 +253,13 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)fpsimd_sve_sync(&hyp_vcpu->vcpu);sync_debug_state(hyp_vcpu);-host_vcpu->arch.ctxt=hyp_vcpu->vcpu.arch.ctxt;--host_vcpu->arch.hcr_el2=hyp_vcpu->vcpu.arch.hcr_el2;+if(pkvm_hyp_vcpu_is_protected(hyp_vcpu)){+host_vcpu->arch.ctxt=hyp_vcpu->vcpu.arch.ctxt;+}else{+/* Keep PC (tracepoint) and PSTATE (vcpu_mode_is_bad_32bit) current. */+host_vcpu->arch.ctxt.regs.pc=hyp_vcpu->vcpu.arch.ctxt.regs.pc;+host_vcpu->arch.ctxt.regs.pstate=hyp_vcpu->vcpu.arch.ctxt.regs.pstate;+}host_vcpu->arch.fault=hyp_vcpu->vcpu.arch.fault;
From: Vincent Donnefort <hidden> Date: 2026-06-19 13:13:02
On Fri, Jun 19, 2026 at 08:07:19AM +0100, Fuad Tabba wrote:
quoted hunk
pKVM copies a non-protected guest's register context between the host
and the hypervisor on every world switch, even when the host never
inspects it. Defer the copy: on entry, flush the host context into the
hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on
exit, leave it in the hyp vCPU and copy it back only when the host needs
it, via a __pkvm_vcpu_sync_state hypercall on trap handling or at vcpu
put. A protected guest's context is copied as before, since lazy sync
only helps where the host is trusted to see the guest's registers.
PC and PSTATE are the exception: they are copied back on every exit so
the kvm_exit tracepoint reports the guest's real exit PC, and the run
loop's vcpu_mode_is_bad_32bit() and SError-masking checks evaluate the
guest's current PSTATE rather than the value left by the previous sync.
handle_exit_early() can also inject an SError, which writes the guest
context (ESR_EL1) outside the trap-handling path. For a non-protected
guest it therefore syncs the context from the hyp vCPU and marks it
dirty, as handle_trap_exceptions() does, so the injection reaches the
hyp vCPU on re-entry rather than being dropped.
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/include/asm/kvm_asm.h | 1 +
arch/arm64/include/asm/kvm_host.h | 2 +
arch/arm64/kvm/arm.c | 7 +++
arch/arm64/kvm/handle_exit.c | 30 +++++++++++
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 86 ++++++++++++++++++++++++++++--
5 files changed, 121 insertions(+), 5 deletions(-)
@@ -1068,6 +1068,8 @@ struct kvm_vcpu_arch {#define INCREMENT_PC __vcpu_single_flag(iflags, BIT(1))/* Target EL/MODE (not a single flag, but let's abuse the macro) */#define EXCEPT_MASK __vcpu_single_flag(iflags, GENMASK(3, 1))+/* Host-set: the hyp flushes the non-protected vCPU state in on entry */+#define PKVM_HOST_STATE_DIRTY __vcpu_single_flag(iflags, BIT(4))/* Helpers to encode exceptions with minimum fuss */#define __EXCEPT_MASK_VAL unpack_vcpu_flag(EXCEPT_MASK)
@@ -733,6 +733,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)if(is_protected_kvm_enabled()){kvm_call_hyp(__vgic_v3_save_aprs,&vcpu->arch.vgic_cpu.vgic_v3);kvm_call_hyp_nvhe(__pkvm_vcpu_put);++/* __pkvm_vcpu_put implies a sync of the state */+if(!kvm_vm_is_protected(vcpu->kvm))+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);}kvm_vcpu_put_debug(vcpu);
@@ -964,6 +968,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)returnret;if(is_protected_kvm_enabled()){+/* Start with the vcpu in a dirty state */+if(!kvm_vm_is_protected(vcpu->kvm))+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);ret=pkvm_create_hyp_vm(kvm);if(ret)returnret;
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu){inthandled;+/*+*Ifwerunanon-protectedVMwhenprotectionisenabled+*system-wide,resyncthestatefromthehypervisorandmark+*itasdirtyonthehostsideifitwasn'tdirtyalready+*(whichcouldhappenifpreemptionhastakenplace).+*/+if(is_protected_kvm_enabled()&&!kvm_vm_is_protected(vcpu->kvm)){+guard(preempt)();+if(!(vcpu_get_flag(vcpu,PKVM_HOST_STATE_DIRTY))){+kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);+}+}+
Could we remove this update here and let handle_exit_early() do the sync
regardless of the SError injection? One of the main point of handle_exit_early()
is to do things under !prempt().
quoted hunk
/*
* See ARM ARM B1.14.1: "Hyp traps on instructions
* that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index) /* For exit types that need handling before we can be preempted */ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index) {+ bool inject_serror = ARM_SERROR_PENDING(exception_index) ||+ ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;++ /*+ * An SError injected below writes the host ctxt; for a non-protected+ * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.+ */+ if (is_protected_kvm_enabled()) {
Should we test !kvm_vm_is_protected(vcpu->kvm) here, as the
PKVM_HOST_STATE_DIRTY is only updated for p-guests everywhere else?
+ vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+ if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+ kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+ vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+ }
+ }
+
if (ARM_SERROR_PENDING(exception_index)) {
if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
u64 disr = kvm_vcpu_get_disr(vcpu);
From: Vincent Donnefort <hidden> Date: 2026-06-19 13:25:05
On Fri, Jun 19, 2026 at 08:07:12AM +0100, Fuad Tabba wrote:
Extract the vCPU MPIDR computation embedded in reset_mpidr() into a
kvm_calculate_mpidr() inline in sys_regs.h, so it can be computed
without duplicating the logic. A follow-up series reuses it to reset
protected vCPUs at EL2.
No functional change intended.
Signed-off-by: Fuad Tabba <redacted>
From: Vincent Donnefort <hidden> Date: 2026-06-19 13:26:53
On Fri, Jun 19, 2026 at 08:07:13AM +0100, Fuad Tabba wrote:
The vcpu_{read,write}_sys_reg() accessors are host-only, so helpers
built on them such as kvm_vcpu_set_be()/kvm_vcpu_is_be() cannot be
shared with hyp code. exception.c already wraps them in
__vcpu_{read,write}_sys_reg(), which pick the host- or hyp-side accessor
via has_vhe() and so are valid in any context.
Move those wrappers to kvm_emulate.h as kvm_vcpu_{read,write}_sys_reg()
and switch the callers over, so a follow-up series can share that
emulation code at EL2.
No functional change intended.
Signed-off-by: Fuad Tabba <redacted>
From: Vincent Donnefort <hidden> Date: 2026-06-19 13:29:25
On Fri, Jun 19, 2026 at 08:07:14AM +0100, Fuad Tabba wrote:
Pull the reusable pieces out of kvm_reset_vcpu(): expose the reset
PSTATE values in kvm_arm.h, and split the core register reset and the
PSCI-driven reset into kvm_reset_vcpu_core() and kvm_reset_vcpu_psci().
A follow-up series reuses these to reset protected vCPUs at EL2.
No functional change intended.
Signed-off-by: Fuad Tabba <redacted>
From: Vincent Donnefort <hidden> Date: 2026-06-19 13:30:55
On Fri, Jun 19, 2026 at 08:07:15AM +0100, Fuad Tabba wrote:
Move kvm_psci_valid_affinity() and kvm_psci_narrow_to_32bit() from
psci.c to include/kvm/arm_psci.h, and move psci_affinity_mask() there
too, renaming it kvm_psci_affinity_mask() now that it is no longer
file-local. A follow-up series handles some protected-guest PSCI calls
at EL2 using these helpers.
No functional change intended.
Signed-off-by: Fuad Tabba <redacted>
From: Vincent Donnefort <hidden> Date: 2026-06-19 13:32:05
On Fri, Jun 19, 2026 at 08:07:16AM +0100, Fuad Tabba wrote:
From: Marc Zyngier <maz@kernel.org>
The nVHE hypervisor repeatedly resolves a host vCPU into the EL2
address space and validates that the loaded hyp vCPU matches it, with
that logic open-coded in each handler.
Add __get_host_hyp_vcpus() and the get_host_hyp_vcpus() macro, which
translate the host vCPU into the hypervisor's address space and, when
pKVM is enabled, also return the loaded hyp vCPU if it matches. If pKVM
is enabled but the loaded hyp vCPU does not correspond to the requested
host vCPU, both the host and hyp vCPU are returned as NULL. Convert
handle___kvm_vcpu_run() to use it.
No functional change intended.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Co-developed-by: Fuad Tabba <redacted>
Signed-off-by: Fuad Tabba <redacted>
@@ -231,23 +262,16 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)gotoout;}-if(!hyp_vcpu){-ret=-EINVAL;-gotoout;-}-flush_hyp_vcpu(hyp_vcpu);ret=__kvm_vcpu_run(&hyp_vcpu->vcpu);sync_hyp_vcpu(hyp_vcpu);}else{-structkvm_vcpu*vcpu=kern_hyp_va(host_vcpu);-/* The host is fully trusted, run its vCPU directly. */-fpsimd_lazy_switch_to_guest(vcpu);-ret=__kvm_vcpu_run(vcpu);-fpsimd_lazy_switch_to_host(vcpu);+fpsimd_lazy_switch_to_guest(host_vcpu);+ret=__kvm_vcpu_run(host_vcpu);+fpsimd_lazy_switch_to_host(host_vcpu);}out:cpu_reg(host_ctxt,1)=ret;
On Fri, 19 Jun 2026 at 14:13, Vincent Donnefort [off-list ref] wrote:
On Fri, Jun 19, 2026 at 08:07:19AM +0100, Fuad Tabba wrote:
quoted
pKVM copies a non-protected guest's register context between the host
and the hypervisor on every world switch, even when the host never
inspects it. Defer the copy: on entry, flush the host context into the
hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on
exit, leave it in the hyp vCPU and copy it back only when the host needs
it, via a __pkvm_vcpu_sync_state hypercall on trap handling or at vcpu
put. A protected guest's context is copied as before, since lazy sync
only helps where the host is trusted to see the guest's registers.
PC and PSTATE are the exception: they are copied back on every exit so
the kvm_exit tracepoint reports the guest's real exit PC, and the run
loop's vcpu_mode_is_bad_32bit() and SError-masking checks evaluate the
guest's current PSTATE rather than the value left by the previous sync.
handle_exit_early() can also inject an SError, which writes the guest
context (ESR_EL1) outside the trap-handling path. For a non-protected
guest it therefore syncs the context from the hyp vCPU and marks it
dirty, as handle_trap_exceptions() does, so the injection reaches the
hyp vCPU on re-entry rather than being dropped.
Signed-off-by: Fuad Tabba <redacted>
---
arch/arm64/include/asm/kvm_asm.h | 1 +
arch/arm64/include/asm/kvm_host.h | 2 +
arch/arm64/kvm/arm.c | 7 +++
arch/arm64/kvm/handle_exit.c | 30 +++++++++++
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 86 ++++++++++++++++++++++++++++--
5 files changed, 121 insertions(+), 5 deletions(-)
@@ -1068,6 +1068,8 @@ struct kvm_vcpu_arch {#define INCREMENT_PC __vcpu_single_flag(iflags, BIT(1))/* Target EL/MODE (not a single flag, but let's abuse the macro) */#define EXCEPT_MASK __vcpu_single_flag(iflags, GENMASK(3, 1))+/* Host-set: the hyp flushes the non-protected vCPU state in on entry */+#define PKVM_HOST_STATE_DIRTY __vcpu_single_flag(iflags, BIT(4))/* Helpers to encode exceptions with minimum fuss */#define __EXCEPT_MASK_VAL unpack_vcpu_flag(EXCEPT_MASK)
@@ -733,6 +733,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)if(is_protected_kvm_enabled()){kvm_call_hyp(__vgic_v3_save_aprs,&vcpu->arch.vgic_cpu.vgic_v3);kvm_call_hyp_nvhe(__pkvm_vcpu_put);++/* __pkvm_vcpu_put implies a sync of the state */+if(!kvm_vm_is_protected(vcpu->kvm))+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);}kvm_vcpu_put_debug(vcpu);
@@ -964,6 +968,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)returnret;if(is_protected_kvm_enabled()){+/* Start with the vcpu in a dirty state */+if(!kvm_vm_is_protected(vcpu->kvm))+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);ret=pkvm_create_hyp_vm(kvm);if(ret)returnret;
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu){inthandled;+/*+*Ifwerunanon-protectedVMwhenprotectionisenabled+*system-wide,resyncthestatefromthehypervisorandmark+*itasdirtyonthehostsideifitwasn'tdirtyalready+*(whichcouldhappenifpreemptionhastakenplace).+*/+if(is_protected_kvm_enabled()&&!kvm_vm_is_protected(vcpu->kvm)){+guard(preempt)();+if(!(vcpu_get_flag(vcpu,PKVM_HOST_STATE_DIRTY))){+kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);+}+}+
Could we remove this update here and let handle_exit_early() do the sync
regardless of the SError injection? One of the main point of handle_exit_early()
is to do things under !prempt().
Agreed on the move: handle_exit_early() is already preempt-off, so the
guard() goes away. Not on every exit though. handle_exit_early() runs
on every exit, and sync_hyp_vcpu() only copies PC/PSTATE/fault back
for a non-protected guest; the GPRs and sysregs cross solely via
__pkvm_vcpu_sync_state. Syncing unconditionally would pull the full
context back on plain IRQ exits, which is the copy this patch avoids.
So I will gate it on trap-or-SError and drop the
handle_trap_exceptions() block.
quoted
/*
* See ARM ARM B1.14.1: "Hyp traps on instructions
* that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index) /* For exit types that need handling before we can be preempted */ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index) {+ bool inject_serror = ARM_SERROR_PENDING(exception_index) ||+ ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;++ /*+ * An SError injected below writes the host ctxt; for a non-protected+ * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.+ */+ if (is_protected_kvm_enabled()) {
Should we test !kvm_vm_is_protected(vcpu->kvm) here, as the
PKVM_HOST_STATE_DIRTY is only updated for p-guests everywhere else?
Yes. The flag is only ever set for non-protected guests, so clearing it
for a protected one is a no-op, but gating it matches the invariant.
Both fold into one block in handle_exit_early():
if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
if (inject_serror ||
ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_TRAP) {
kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
} else {
vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
}
}
I will fold this into the next respin.
Thanks for the reviews!
/fuad
quoted
+ vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+ if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+ kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+ vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+ }
+ }
+
if (ARM_SERROR_PENDING(exception_index)) {
if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
u64 disr = kvm_vcpu_get_disr(vcpu);
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu){inthandled;+/*+*Ifwerunanon-protectedVMwhenprotectionisenabled+*system-wide,resyncthestatefromthehypervisorandmark+*itasdirtyonthehostsideifitwasn'tdirtyalready+*(whichcouldhappenifpreemptionhastakenplace).+*/+if(is_protected_kvm_enabled()&&!kvm_vm_is_protected(vcpu->kvm)){+guard(preempt)();+if(!(vcpu_get_flag(vcpu,PKVM_HOST_STATE_DIRTY))){+kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);+}+}+
Could we remove this update here and let handle_exit_early() do the sync
regardless of the SError injection? One of the main point of handle_exit_early()
is to do things under !prempt().
Agreed on the move: handle_exit_early() is already preempt-off, so the
guard() goes away. Not on every exit though. handle_exit_early() runs
on every exit, and sync_hyp_vcpu() only copies PC/PSTATE/fault back
for a non-protected guest; the GPRs and sysregs cross solely via
__pkvm_vcpu_sync_state. Syncing unconditionally would pull the full
context back on plain IRQ exits, which is the copy this patch avoids.
So I will gate it on trap-or-SError and drop the
handle_trap_exceptions() block.
quoted
quoted
/*
* See ARM ARM B1.14.1: "Hyp traps on instructions
* that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index) /* For exit types that need handling before we can be preempted */ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index) {+ bool inject_serror = ARM_SERROR_PENDING(exception_index) ||+ ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;++ /*+ * An SError injected below writes the host ctxt; for a non-protected+ * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.+ */+ if (is_protected_kvm_enabled()) {
Should we test !kvm_vm_is_protected(vcpu->kvm) here, as the
PKVM_HOST_STATE_DIRTY is only updated for p-guests everywhere else?
Yes. The flag is only ever set for non-protected guests, so clearing it
for a protected one is a no-op, but gating it matches the invariant.
Both fold into one block in handle_exit_early():
if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
if (inject_serror ||
ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_TRAP) {
kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
} else {
vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
}
}
I will fold this into the next respin.
Ah yes of course, I was hoping we could just have a switch here, just like
handle_exit() does, but that's not possible because of ARM_SERROR_PENDING().
Perhaps it would look cleaner if done in a separate function
handle_exit_pkvm_state()?
Thanks for the reviews!
/fuad
quoted
quoted
+ vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+ if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+ kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+ vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+ }
+ }
+
if (ARM_SERROR_PENDING(exception_index)) {
if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
u64 disr = kvm_vcpu_get_disr(vcpu);
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu){inthandled;+/*+*Ifwerunanon-protectedVMwhenprotectionisenabled+*system-wide,resyncthestatefromthehypervisorandmark+*itasdirtyonthehostsideifitwasn'tdirtyalready+*(whichcouldhappenifpreemptionhastakenplace).+*/+if(is_protected_kvm_enabled()&&!kvm_vm_is_protected(vcpu->kvm)){+guard(preempt)();+if(!(vcpu_get_flag(vcpu,PKVM_HOST_STATE_DIRTY))){+kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);+vcpu_set_flag(vcpu,PKVM_HOST_STATE_DIRTY);+}+}+
Could we remove this update here and let handle_exit_early() do the sync
regardless of the SError injection? One of the main point of handle_exit_early()
is to do things under !prempt().
Agreed on the move: handle_exit_early() is already preempt-off, so the
guard() goes away. Not on every exit though. handle_exit_early() runs
on every exit, and sync_hyp_vcpu() only copies PC/PSTATE/fault back
for a non-protected guest; the GPRs and sysregs cross solely via
__pkvm_vcpu_sync_state. Syncing unconditionally would pull the full
context back on plain IRQ exits, which is the copy this patch avoids.
So I will gate it on trap-or-SError and drop the
handle_trap_exceptions() block.
quoted
quoted
/*
* See ARM ARM B1.14.1: "Hyp traps on instructions
* that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index) /* For exit types that need handling before we can be preempted */ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index) {+ bool inject_serror = ARM_SERROR_PENDING(exception_index) ||+ ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;++ /*+ * An SError injected below writes the host ctxt; for a non-protected+ * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.+ */+ if (is_protected_kvm_enabled()) {
Should we test !kvm_vm_is_protected(vcpu->kvm) here, as the
PKVM_HOST_STATE_DIRTY is only updated for p-guests everywhere else?
Yes. The flag is only ever set for non-protected guests, so clearing it
for a protected one is a no-op, but gating it matches the invariant.
Both fold into one block in handle_exit_early():
if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
if (inject_serror ||
ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_TRAP) {
kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
} else {
vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
}
}
I will fold this into the next respin.
Ah yes of course, I was hoping we could just have a switch here, just like
handle_exit() does, but that's not possible because of ARM_SERROR_PENDING().
Perhaps it would look cleaner if done in a separate function
handle_exit_pkvm_state()?
Agreed, that reads better. Pulling it into handle_exit_pkvm_state()
also gets inject_serror out of handle_exit_early(), which only used it
for this. Done for the next respin.
Cheers,
/fuad
quoted
Thanks for the reviews!
/fuad
quoted
quoted
+ vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+ if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+ kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+ vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+ }
+ }
+
if (ARM_SERROR_PENDING(exception_index)) {
if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
u64 disr = kvm_vcpu_get_disr(vcpu);