From: Colton Lewis <hidden> Date: 2025-12-09 20:52:25
This series creates a new PMU scheme on ARM, a partitioned PMU that
allows reserving a subset of counters for more direct guest access,
significantly reducing overhead. More details, including performance
benchmarks, can be read in the v1 cover letter linked below.
An overview of what this series accomplishes was presented at KVM
Forum 2025. Slides [1] and video [2] are linked below.
The long duration between v4 and v5 is due to time spent on this
project being monopolized preparing this feature for internal
production. As a result, there are too many improvements to fully list
here, but I will cover the notable ones.
v5:
* Rebase onto v6.18-rc7. This required pulling some reorganization
patches from Anish and Sean that were dependencies from previous
versions based on kvm/queue but never made it to upstream.
* Ensure FGTs (fine-grained traps) are correctly programmed at vCPU
load using kvm_vcpu_load_fgt() and helpers introduced by Oliver
Upton.
* Cleanly separate concerns of whether the partitioned PMU is enabled
for the guest and whether FGT should be enabled. This allows that
the capability can be VM-scoped while the implementation detail of
whether FGT and context switching are in effect can remain
vCPU-scoped.
* Shrink the uAPI change. Instead of a cap and corresponding ioctl,
the feature can be controlled by just a cap with an argument. The
cap is now also VM-scoped and enforces ordering that it should be
decided before vCPUs are created. Whether the cap is enabled is now
tracked by the new flag KVM_ARCH_ARM_PARTITIONED_PMU_ENABLED.
* Improve log messages when partitioning in the PMUv3 driver.
* Introduce a global variable armv8pmu_hpmn_max in the PMUv3 driver so
KVM code can read if a value was set before the PMU is probed. This
is needed to properly test if we have the capability before vCPUs
are created.
* Make it possible for a VMM to filter the HPMN0 feature bit.
* Fix event filter problems with PMEVTYPER handling in
writethrough_pmevtyper() and kvm_pmu_apply_event_filter() by using
kvm_pmu_event_mask() in the right spots. And if an event is
filtered, write the physical register with the appropriate exclude
bits set but keep the virtual register exactly what the guest wrote.
* Fix register access problems with the PMU register fast path handler
by lifting some static PMU access checks from sys_regs.c to use them
in the fast path too and make bit masking more strict for better ARM
compliance.
* Fix the readability and logic of programming the MDCR_EL2 register
when entering the guest. Make sure to set the HPME bit to allow host
counters to count guest events. Set TPM and TPMCR by default and
clear them if partitioning is enabled rather than the previous
inverted logic of leaving them clear and setting them if
partitioning is not enabled. Make the HPMN field computation more
clear.
* As part of lazy context switching, do a load when the guest is
switching to physical access to ensure any previous writes that only
reached the virtual registers reach the physical ones as well and
are not clobbered by the next vcpu_put().
* Other fixes and improvements that are too small to mention or left
out from my personal notes.
v4:
https://lore.kernel.org/kvmarm/20250714225917.1396543-1-coltonlewis@google.com/
v3:
https://lore.kernel.org/kvm/20250626200459.1153955-1-coltonlewis@google.com/
v2:
https://lore.kernel.org/kvm/20250620221326.1261128-1-coltonlewis@google.com/
v1:
https://lore.kernel.org/kvm/20250602192702.2125115-1-coltonlewis@google.com/
[1] https://gitlab.com/qemu-project/kvm-forum/-/raw/main/_attachments/2025/Optimizing__itvHkhc.pdf
[2] https://www.youtube.com/watch?v=YRzZ8jMIA6M&list=PLW3ep1uCIRfxwmllXTOA2txfDWN6vUOHp&index=9
Anish Ghulati (1):
KVM: arm64: Move arm_{psci,hypercalls}.h to an internal KVM path
Colton Lewis (20):
arm64: cpufeature: Add cpucap for HPMN0
KVM: arm64: Reorganize PMU functions
perf: arm_pmuv3: Introduce method to partition the PMU
perf: arm_pmuv3: Generalize counter bitmasks
perf: arm_pmuv3: Keep out of guest counter partition
KVM: arm64: Set up FGT for Partitioned PMU
KVM: arm64: Writethrough trapped PMEVTYPER register
KVM: arm64: Use physical PMSELR for PMXEVTYPER if partitioned
KVM: arm64: Writethrough trapped PMOVS register
KVM: arm64: Write fast path PMU register handlers
KVM: arm64: Setup MDCR_EL2 to handle a partitioned PMU
KVM: arm64: Account for partitioning in PMCR_EL0 access
KVM: arm64: Context swap Partitioned PMU guest registers
KVM: arm64: Enforce PMU event filter at vcpu_load()
KVM: arm64: Implement lazy PMU context swaps
perf: arm_pmuv3: Handle IRQs for Partitioned PMU guest counters
KVM: arm64: Inject recorded guest interrupts
KVM: arm64: Add KVM_CAP to partition the PMU
KVM: selftests: Add find_bit to KVM library
KVM: arm64: selftests: Add test case for partitioned PMU
Marc Zyngier (1):
KVM: arm64: Reorganize PMU includes
Sean Christopherson (2):
KVM: arm64: Include KVM headers to get forward declarations
KVM: arm64: Move ARM specific headers in include/kvm to arch directory
Documentation/virt/kvm/api.rst | 24 +
arch/arm/include/asm/arm_pmuv3.h | 28 +
arch/arm64/include/asm/arm_pmuv3.h | 61 +-
.../arm64/include/asm/kvm_arch_timer.h | 2 +
arch/arm64/include/asm/kvm_host.h | 24 +-
.../arm64/include/asm/kvm_pmu.h | 142 ++++
arch/arm64/include/asm/kvm_types.h | 7 +-
.../arm64/include/asm/kvm_vgic.h | 0
arch/arm64/kernel/cpufeature.c | 8 +
arch/arm64/kvm/Makefile | 2 +-
arch/arm64/kvm/arch_timer.c | 5 +-
arch/arm64/kvm/arm.c | 23 +-
{include => arch/arm64}/kvm/arm_hypercalls.h | 0
{include => arch/arm64}/kvm/arm_psci.h | 0
arch/arm64/kvm/config.c | 34 +-
arch/arm64/kvm/debug.c | 31 +-
arch/arm64/kvm/guest.c | 2 +-
arch/arm64/kvm/handle_exit.c | 2 +-
arch/arm64/kvm/hyp/Makefile | 6 +-
arch/arm64/kvm/hyp/include/hyp/switch.h | 211 ++++-
arch/arm64/kvm/hyp/nvhe/switch.c | 4 +-
arch/arm64/kvm/hyp/vhe/switch.c | 4 +-
arch/arm64/kvm/hypercalls.c | 4 +-
arch/arm64/kvm/pmu-direct.c | 464 +++++++++++
arch/arm64/kvm/pmu-emul.c | 678 +---------------
arch/arm64/kvm/pmu.c | 726 ++++++++++++++++++
arch/arm64/kvm/psci.c | 4 +-
arch/arm64/kvm/pvtime.c | 2 +-
arch/arm64/kvm/reset.c | 3 +-
arch/arm64/kvm/sys_regs.c | 110 +--
arch/arm64/kvm/trace_arm.h | 2 +-
arch/arm64/kvm/trng.c | 2 +-
arch/arm64/kvm/vgic/vgic-debug.c | 2 +-
arch/arm64/kvm/vgic/vgic-init.c | 2 +-
arch/arm64/kvm/vgic/vgic-irqfd.c | 2 +-
arch/arm64/kvm/vgic/vgic-kvm-device.c | 2 +-
arch/arm64/kvm/vgic/vgic-mmio-v2.c | 2 +-
arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +-
arch/arm64/kvm/vgic/vgic-mmio.c | 4 +-
arch/arm64/kvm/vgic/vgic-v2.c | 2 +-
arch/arm64/kvm/vgic/vgic-v3-nested.c | 3 +-
arch/arm64/kvm/vgic/vgic-v3.c | 2 +-
arch/arm64/kvm/vgic/vgic-v5.c | 2 +-
arch/arm64/tools/cpucaps | 1 +
arch/arm64/tools/sysreg | 6 +-
drivers/perf/arm_pmuv3.c | 137 +++-
include/linux/perf/arm_pmu.h | 1 +
include/linux/perf/arm_pmuv3.h | 14 +-
include/uapi/linux/kvm.h | 1 +
tools/include/uapi/linux/kvm.h | 1 +
tools/testing/selftests/kvm/Makefile.kvm | 1 +
.../selftests/kvm/arm64/vpmu_counter_access.c | 77 +-
tools/testing/selftests/kvm/lib/find_bit.c | 1 +
53 files changed, 2049 insertions(+), 831 deletions(-)
rename include/kvm/arm_arch_timer.h => arch/arm64/include/asm/kvm_arch_timer.h (98%)
rename include/kvm/arm_pmu.h => arch/arm64/include/asm/kvm_pmu.h (61%)
rename include/kvm/arm_vgic.h => arch/arm64/include/asm/kvm_vgic.h (100%)
rename {include => arch/arm64}/kvm/arm_hypercalls.h (100%)
rename {include => arch/arm64}/kvm/arm_psci.h (100%)
create mode 100644 arch/arm64/kvm/pmu-direct.c
create mode 100644 tools/testing/selftests/kvm/lib/find_bit.c
base-commit: ac3fd01e4c1efce8f2c054cdeb2ddd2fc0fb150d
--
2.52.0.239.gd5f0c6e74e-goog
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:27
Add a capability for FEAT_HPMN0, whether MDCR_EL2.HPMN can specify 0
counters reserved for the guest.
This required changing HPMN0 to an UnsignedEnum in tools/sysreg
because otherwise not all the appropriate macros are generated to add
it to arm64_cpu_capabilities_arm64_features.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/kernel/cpufeature.c | 8 ++++++++
arch/arm64/kvm/sys_regs.c | 3 ++-
arch/arm64/tools/cpucaps | 1 +
arch/arm64/tools/sysreg | 6 +++---
4 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/include/kvm/arm_hypercalls.h b/arch/arm64/kvm/arm_hypercalls.hsimilarity index 100%rename from include/kvm/arm_hypercalls.hrename to arch/arm64/kvm/arm_hypercalls.hdiff --git a/include/kvm/arm_psci.h b/arch/arm64/kvm/arm_psci.hsimilarity index 100%rename from include/kvm/arm_psci.hrename to arch/arm64/kvm/arm_psci.hdiff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.cindex 1c87699fd886e..863b351ae1221 100644--- a/arch/arm64/kvm/guest.c+++ b/arch/arm64/kvm/guest.c
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:29
From: Sean Christopherson <seanjc@google.com>
Include include/uapi/linux/kvm.h and include/linux/kvm_types.h in ARM's
public arm_arch_timer.h and arm_pmu.h headers to get forward declarations
of things like "struct kvm_vcpu" and "struct kvm_device_attr", which are
referenced but never declared (neither file includes *any* KVM headers).
The missing includes don't currently cause problems because of the order
of includes in parent files, but that order is largely arbitrary and is
subject to change, e.g. a future commit will move the ARM specific headers
to arch/arm64/include/asm and reorder parent includes to maintain
alphabetic ordering.
Reported-by: kernel test robot <redacted>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: [off-list ref]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
include/kvm/arm_arch_timer.h | 2 ++
include/kvm/arm_pmu.h | 2 ++
2 files changed, 4 insertions(+)
diff --git a/include/kvm/arm_pmu.h b/arch/arm64/include/asm/kvm_pmu.hsimilarity index 100%rename from include/kvm/arm_pmu.hrename to arch/arm64/include/asm/kvm_pmu.hdiff --git a/include/kvm/arm_vgic.h b/arch/arm64/include/asm/kvm_vgic.hsimilarity index 100%rename from include/kvm/arm_vgic.hrename to arch/arm64/include/asm/kvm_vgic.hdiff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.cindex 3f675875abea2..ce62a12cf0e5c 100644--- a/arch/arm64/kvm/arch_timer.c+++ b/arch/arm64/kvm/arch_timer.c
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:32
From: Marc Zyngier <maz@kernel.org>
Including *all* of asm/kvm_host.h in asm/arm_pmuv3.h is a bad idea
because that is much more than arm_pmuv3.h logically needs and creates
a circular dependency that makes it easy to introduce compiler errors
when editing this code.
asm/kvm_host.h includes asm/kvm_pmu.h includes perf/arm_pmuv3.h
includes asm/arm_pmuv3.h includes asm/kvm_host.h
Reorganize the PMU includes to be more sane. In particular:
* Remove the circular dependency by removing the kvm_host.h include
from asm/arm_pmuv3.h since 99% of it isn't needed.
* Move the remaining tiny bit of KVM/PMU interface from kvm_host.h
into kvm_pmu.h
* Conditionally on ARM64, include the more targeted kvm_pmu.h directly
in the arm_pmuv3.c driver.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/arm_pmuv3.h | 2 --
arch/arm64/include/asm/kvm_host.h | 14 --------------
arch/arm64/include/asm/kvm_pmu.h | 15 +++++++++++++++
drivers/perf/arm_pmuv3.c | 5 +++++
4 files changed, 20 insertions(+), 16 deletions(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:32
A lot of functions in pmu-emul.c aren't specific to the emulated PMU
implementation. Move them to the more appropriate pmu.c file where
shared PMU functions should live.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/kvm_pmu.h | 3 +
arch/arm64/kvm/pmu-emul.c | 672 +-----------------------------
arch/arm64/kvm/pmu.c | 675 +++++++++++++++++++++++++++++++
3 files changed, 679 insertions(+), 671 deletions(-)
@@ -40,46 +31,6 @@ static struct kvm_pmc *kvm_vcpu_idx_to_pmc(struct kvm_vcpu *vcpu, int cnt_idx)return&vcpu->arch.pmu.pmc[cnt_idx];}-staticu32__kvm_pmu_event_mask(unsignedintpmuver)-{-switch(pmuver){-caseID_AA64DFR0_EL1_PMUVer_IMP:-returnGENMASK(9,0);-caseID_AA64DFR0_EL1_PMUVer_V3P1:-caseID_AA64DFR0_EL1_PMUVer_V3P4:-caseID_AA64DFR0_EL1_PMUVer_V3P5:-caseID_AA64DFR0_EL1_PMUVer_V3P7:-returnGENMASK(15,0);-default:/* Shouldn't be here, just for sanity */-WARN_ONCE(1,"Unknown PMU version %d\n",pmuver);-return0;-}-}--staticu32kvm_pmu_event_mask(structkvm*kvm)-{-u64dfr0=kvm_read_vm_id_reg(kvm,SYS_ID_AA64DFR0_EL1);-u8pmuver=SYS_FIELD_GET(ID_AA64DFR0_EL1,PMUVer,dfr0);--return__kvm_pmu_event_mask(pmuver);-}--u64kvm_pmu_evtyper_mask(structkvm*kvm)-{-u64mask=ARMV8_PMU_EXCLUDE_EL1|ARMV8_PMU_EXCLUDE_EL0|-kvm_pmu_event_mask(kvm);--if(kvm_has_feat(kvm,ID_AA64PFR0_EL1,EL2,IMP))-mask|=ARMV8_PMU_INCLUDE_EL2;--if(kvm_has_feat(kvm,ID_AA64PFR0_EL1,EL3,IMP))-mask|=ARMV8_PMU_EXCLUDE_NS_EL0|-ARMV8_PMU_EXCLUDE_NS_EL1|-ARMV8_PMU_EXCLUDE_EL3;--returnmask;-}-/***kvm_pmc_is_64bit-determineifcounteris64bit*@pmc:countercontext
@@ -784,132 +627,6 @@ void kvm_pmu_set_counter_event_type(struct kvm_vcpu *vcpu, u64 data,kvm_pmu_create_perf_event(pmc);}-voidkvm_host_pmu_init(structarm_pmu*pmu)-{-structarm_pmu_entry*entry;--/*-*CheckthesanitisedPMUversionforthesystem,asKVMdoesnot-*supportimplementationswherePMUv3existsonasubsetofCPUs.-*/-if(!pmuv3_implemented(kvm_arm_pmu_get_pmuver_limit()))-return;--guard(mutex)(&arm_pmus_lock);--entry=kmalloc(sizeof(*entry),GFP_KERNEL);-if(!entry)-return;--entry->arm_pmu=pmu;-list_add_tail(&entry->entry,&arm_pmus);-}--staticstructarm_pmu*kvm_pmu_probe_armpmu(void)-{-structarm_pmu_entry*entry;-structarm_pmu*pmu;-intcpu;--guard(mutex)(&arm_pmus_lock);--/*-*ItissafetouseastalecputoiteratethelistofPMUssolongas-*thesamevalueisusedfortheentiretyoftheloop.Giventhis,and-*thefactthatnopercpudataisusedforthelookupthereisnoneed-*todisablepreemption.-*-*Itisstillnecessarytogetavalidcpu,though,toprobeforthe-*defaultPMUinstanceasuserspaceisnotrequiredtospecifyaPMU-*type.InordertoupholdthepreexistingbehaviorKVMselectsthe-*PMUinstanceforthecoreduringvcpuinit.Adependentuse-*casewouldbeauserwithdisdainofallthingsbig.LITTLEthat-*affinestheVMMtoaparticularclusterofcores.-*-*Inanycase,userspaceshouldjustdothesanethingandusetheUAPI-*toselectaPMUtypedirectly.But,bewaryofthebaggagebeing-*carriedhere.-*/-cpu=raw_smp_processor_id();-list_for_each_entry(entry,&arm_pmus,entry){-pmu=entry->arm_pmu;--if(cpumask_test_cpu(cpu,&pmu->supported_cpus))-returnpmu;-}--returnNULL;-}--staticu64__compute_pmceid(structarm_pmu*pmu,boolpmceid1)-{-u32hi[2],lo[2];--bitmap_to_arr32(lo,pmu->pmceid_bitmap,ARMV8_PMUV3_MAX_COMMON_EVENTS);-bitmap_to_arr32(hi,pmu->pmceid_ext_bitmap,ARMV8_PMUV3_MAX_COMMON_EVENTS);--return((u64)hi[pmceid1]<<32)|lo[pmceid1];-}--staticu64compute_pmceid0(structarm_pmu*pmu)-{-u64val=__compute_pmceid(pmu,0);--/* always support SW_INCR */-val|=BIT(ARMV8_PMUV3_PERFCTR_SW_INCR);-/* always support CHAIN */-val|=BIT(ARMV8_PMUV3_PERFCTR_CHAIN);-returnval;-}--staticu64compute_pmceid1(structarm_pmu*pmu)-{-u64val=__compute_pmceid(pmu,1);--/*-*Don'tadvertiseSTALL_SLOT*,asPMMIR_EL0ishandled-*asRAZ-*/-val&=~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT-32)|-BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND-32)|-BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND-32));-returnval;-}--u64kvm_pmu_get_pmceid(structkvm_vcpu*vcpu,boolpmceid1)-{-structarm_pmu*cpu_pmu=vcpu->kvm->arch.arm_pmu;-unsignedlong*bmap=vcpu->kvm->arch.pmu_filter;-u64val,mask=0;-intbase,i,nr_events;--if(!pmceid1){-val=compute_pmceid0(cpu_pmu);-base=0;-}else{-val=compute_pmceid1(cpu_pmu);-base=32;-}--if(!bmap)-returnval;--nr_events=kvm_pmu_event_mask(vcpu->kvm)+1;--for(i=0;i<32;i+=8){-u64byte;--byte=bitmap_get_value8(bmap,base+i);-mask|=byte<<i;-if(nr_events>=(0x4000+base+32)){-byte=bitmap_get_value8(bmap,0x4000+base+i);-mask|=byte<<(32+i);-}-}--returnval&mask;-}-voidkvm_vcpu_reload_pmu(structkvm_vcpu*vcpu){u64mask=kvm_pmu_implemented_counter_mask(vcpu);
@@ -921,393 +638,6 @@ void kvm_vcpu_reload_pmu(struct kvm_vcpu *vcpu)kvm_pmu_reprogram_counter_mask(vcpu,mask);}-intkvm_arm_pmu_v3_enable(structkvm_vcpu*vcpu)-{-if(!vcpu->arch.pmu.created)-return-EINVAL;--/*-*AvalidinterruptconfigurationforthePMUiseithertohavea-*properlyconfiguredinterruptnumberandusinganin-kernel-*irqchip,ortonothaveanin-kernelGICandnotsetanIRQ.-*/-if(irqchip_in_kernel(vcpu->kvm)){-intirq=vcpu->arch.pmu.irq_num;-/*-*Ifweareusinganin-kernelvgic,atthispointweknow-*thevgicwillbeinitialized,sowecancheckthePMUirq-*numberagainstthedimensionsofthevgicandmakesure-*it'svalid.-*/-if(!irq_is_ppi(irq)&&!vgic_valid_spi(vcpu->kvm,irq))-return-EINVAL;-}elseif(kvm_arm_pmu_irq_initialized(vcpu)){-return-EINVAL;-}--return0;-}--staticintkvm_arm_pmu_v3_init(structkvm_vcpu*vcpu)-{-if(irqchip_in_kernel(vcpu->kvm)){-intret;--/*-*IfusingthePMUwithanin-kernelvirtualGIC-*implementation,werequiretheGICtobealready-*initializedwheninitializingthePMU.-*/-if(!vgic_initialized(vcpu->kvm))-return-ENODEV;--if(!kvm_arm_pmu_irq_initialized(vcpu))-return-ENXIO;--ret=kvm_vgic_set_owner(vcpu,vcpu->arch.pmu.irq_num,-&vcpu->arch.pmu);-if(ret)-returnret;-}--init_irq_work(&vcpu->arch.pmu.overflow_work,-kvm_pmu_perf_overflow_notify_vcpu);--vcpu->arch.pmu.created=true;-return0;-}--/*-*ForoneVMtheinterrupttypemustbesameforeachvcpu.-*AsaPPI,theinterruptnumberisthesameforallvcpus,-*whileasanSPIitmustbeaseparatenumberpervcpu.-*/-staticboolpmu_irq_is_valid(structkvm*kvm,intirq)-{-unsignedlongi;-structkvm_vcpu*vcpu;--kvm_for_each_vcpu(i,vcpu,kvm){-if(!kvm_arm_pmu_irq_initialized(vcpu))-continue;--if(irq_is_ppi(irq)){-if(vcpu->arch.pmu.irq_num!=irq)-returnfalse;-}else{-if(vcpu->arch.pmu.irq_num==irq)-returnfalse;-}-}--returntrue;-}--/**-*kvm_arm_pmu_get_max_counters-ReturnthemaxnumberofPMUcounters.-*@kvm:Thekvmpointer-*/-u8kvm_arm_pmu_get_max_counters(structkvm*kvm)-{-structarm_pmu*arm_pmu=kvm->arch.arm_pmu;--/*-*PMUv3requiresthatalleventcountersarecapableofcountingany-*event,thoughthesamemaynotbetrueofnon-PMUv3hardware.-*/-if(cpus_have_final_cap(ARM64_WORKAROUND_PMUV3_IMPDEF_TRAPS))-return1;--/*-*Thearm_pmu->cntr_maskconsidersthefixedcounter(s)aswell.-*Ignorethoseandreturnonlythegeneral-purposecounters.-*/-returnbitmap_weight(arm_pmu->cntr_mask,ARMV8_PMU_MAX_GENERAL_COUNTERS);-}--staticvoidkvm_arm_set_nr_counters(structkvm*kvm,unsignedintnr)-{-kvm->arch.nr_pmu_counters=nr;--/* Reset MDCR_EL2.HPMN behind the vcpus' back... */-if(test_bit(KVM_ARM_VCPU_HAS_EL2,kvm->arch.vcpu_features)){-structkvm_vcpu*vcpu;-unsignedlongi;--kvm_for_each_vcpu(i,vcpu,kvm){-u64val=__vcpu_sys_reg(vcpu,MDCR_EL2);-val&=~MDCR_EL2_HPMN;-val|=FIELD_PREP(MDCR_EL2_HPMN,kvm->arch.nr_pmu_counters);-__vcpu_assign_sys_reg(vcpu,MDCR_EL2,val);-}-}-}--staticvoidkvm_arm_set_pmu(structkvm*kvm,structarm_pmu*arm_pmu)-{-lockdep_assert_held(&kvm->arch.config_lock);--kvm->arch.arm_pmu=arm_pmu;-kvm_arm_set_nr_counters(kvm,kvm_arm_pmu_get_max_counters(kvm));-}--/**-*kvm_arm_set_default_pmu-NoPMUset,getthedefaultone.-*@kvm:Thekvmpointer-*-*Theobservantamongyouwillnoticethatthesupported_cpus-*maskdoesnotgetupdatedforthedefaultPMUeventhoughit-*isquitepossibletheselectedinstancesupportsonlya-*subsetofcoresinthesystem.Thisisintentional,and-*upholdsthepreexistingbehavioronheterogeneoussystems-*wherevCPUscanbescheduledonanycorebuttheguest-*counterscouldstopworking.-*/-intkvm_arm_set_default_pmu(structkvm*kvm)-{-structarm_pmu*arm_pmu=kvm_pmu_probe_armpmu();--if(!arm_pmu)-return-ENODEV;--kvm_arm_set_pmu(kvm,arm_pmu);-return0;-}--staticintkvm_arm_pmu_v3_set_pmu(structkvm_vcpu*vcpu,intpmu_id)-{-structkvm*kvm=vcpu->kvm;-structarm_pmu_entry*entry;-structarm_pmu*arm_pmu;-intret=-ENXIO;--lockdep_assert_held(&kvm->arch.config_lock);-mutex_lock(&arm_pmus_lock);--list_for_each_entry(entry,&arm_pmus,entry){-arm_pmu=entry->arm_pmu;-if(arm_pmu->pmu.type==pmu_id){-if(kvm_vm_has_ran_once(kvm)||-(kvm->arch.pmu_filter&&kvm->arch.arm_pmu!=arm_pmu)){-ret=-EBUSY;-break;-}--kvm_arm_set_pmu(kvm,arm_pmu);-cpumask_copy(kvm->arch.supported_cpus,&arm_pmu->supported_cpus);-ret=0;-break;-}-}--mutex_unlock(&arm_pmus_lock);-returnret;-}--staticintkvm_arm_pmu_v3_set_nr_counters(structkvm_vcpu*vcpu,unsignedintn)-{-structkvm*kvm=vcpu->kvm;--if(!kvm->arch.arm_pmu)-return-EINVAL;--if(n>kvm_arm_pmu_get_max_counters(kvm))-return-EINVAL;--kvm_arm_set_nr_counters(kvm,n);-return0;-}--intkvm_arm_pmu_v3_set_attr(structkvm_vcpu*vcpu,structkvm_device_attr*attr)-{-structkvm*kvm=vcpu->kvm;--lockdep_assert_held(&kvm->arch.config_lock);--if(!kvm_vcpu_has_pmu(vcpu))-return-ENODEV;--if(vcpu->arch.pmu.created)-return-EBUSY;--switch(attr->attr){-caseKVM_ARM_VCPU_PMU_V3_IRQ:{-int__user*uaddr=(int__user*)(long)attr->addr;-intirq;--if(!irqchip_in_kernel(kvm))-return-EINVAL;--if(get_user(irq,uaddr))-return-EFAULT;--/* The PMU overflow interrupt can be a PPI or a valid SPI. */-if(!(irq_is_ppi(irq)||irq_is_spi(irq)))-return-EINVAL;--if(!pmu_irq_is_valid(kvm,irq))-return-EINVAL;--if(kvm_arm_pmu_irq_initialized(vcpu))-return-EBUSY;--kvm_debug("Set kvm ARM PMU irq: %d\n",irq);-vcpu->arch.pmu.irq_num=irq;-return0;-}-caseKVM_ARM_VCPU_PMU_V3_FILTER:{-u8pmuver=kvm_arm_pmu_get_pmuver_limit();-structkvm_pmu_event_filter__user*uaddr;-structkvm_pmu_event_filterfilter;-intnr_events;--/*-*Allowuserspacetospecifyaneventfilterfortheentire-*eventrangesupportedbyPMUVerofthehardware,rather-*thantheguest'sPMUVerforKVMbackwardcompatibility.-*/-nr_events=__kvm_pmu_event_mask(pmuver)+1;--uaddr=(structkvm_pmu_event_filter__user*)(long)attr->addr;--if(copy_from_user(&filter,uaddr,sizeof(filter)))-return-EFAULT;--if(((u32)filter.base_event+filter.nevents)>nr_events||-(filter.action!=KVM_PMU_EVENT_ALLOW&&-filter.action!=KVM_PMU_EVENT_DENY))-return-EINVAL;--if(kvm_vm_has_ran_once(kvm))-return-EBUSY;--if(!kvm->arch.pmu_filter){-kvm->arch.pmu_filter=bitmap_alloc(nr_events,GFP_KERNEL_ACCOUNT);-if(!kvm->arch.pmu_filter)-return-ENOMEM;--/*-*Thedefaultdependsonthefirstappliedfilter.-*Ifitallowsevents,thedefaultistodeny.-*Conversely,ifthefirstfilterdeniesasetof-*events,thedefaultistoallow.-*/-if(filter.action==KVM_PMU_EVENT_ALLOW)-bitmap_zero(kvm->arch.pmu_filter,nr_events);-else-bitmap_fill(kvm->arch.pmu_filter,nr_events);-}--if(filter.action==KVM_PMU_EVENT_ALLOW)-bitmap_set(kvm->arch.pmu_filter,filter.base_event,filter.nevents);-else-bitmap_clear(kvm->arch.pmu_filter,filter.base_event,filter.nevents);--return0;-}-caseKVM_ARM_VCPU_PMU_V3_SET_PMU:{-int__user*uaddr=(int__user*)(long)attr->addr;-intpmu_id;--if(get_user(pmu_id,uaddr))-return-EFAULT;--returnkvm_arm_pmu_v3_set_pmu(vcpu,pmu_id);-}-caseKVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS:{-unsignedint__user*uaddr=(unsignedint__user*)(long)attr->addr;-unsignedintn;--if(get_user(n,uaddr))-return-EFAULT;--returnkvm_arm_pmu_v3_set_nr_counters(vcpu,n);-}-caseKVM_ARM_VCPU_PMU_V3_INIT:-returnkvm_arm_pmu_v3_init(vcpu);-}--return-ENXIO;-}--intkvm_arm_pmu_v3_get_attr(structkvm_vcpu*vcpu,structkvm_device_attr*attr)-{-switch(attr->attr){-caseKVM_ARM_VCPU_PMU_V3_IRQ:{-int__user*uaddr=(int__user*)(long)attr->addr;-intirq;--if(!irqchip_in_kernel(vcpu->kvm))-return-EINVAL;--if(!kvm_vcpu_has_pmu(vcpu))-return-ENODEV;--if(!kvm_arm_pmu_irq_initialized(vcpu))-return-ENXIO;--irq=vcpu->arch.pmu.irq_num;-returnput_user(irq,uaddr);-}-}--return-ENXIO;-}--intkvm_arm_pmu_v3_has_attr(structkvm_vcpu*vcpu,structkvm_device_attr*attr)-{-switch(attr->attr){-caseKVM_ARM_VCPU_PMU_V3_IRQ:-caseKVM_ARM_VCPU_PMU_V3_INIT:-caseKVM_ARM_VCPU_PMU_V3_FILTER:-caseKVM_ARM_VCPU_PMU_V3_SET_PMU:-caseKVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS:-if(kvm_vcpu_has_pmu(vcpu))-return0;-}--return-ENXIO;-}--u8kvm_arm_pmu_get_pmuver_limit(void)-{-unsignedintpmuver;--pmuver=SYS_FIELD_GET(ID_AA64DFR0_EL1,PMUVer,-read_sanitised_ftr_reg(SYS_ID_AA64DFR0_EL1));--/*-*SpoofabarebonesPMUv3implementationifthesystemsupportsIMPDEF-*trapsofthePMUv3sysregs-*/-if(cpus_have_final_cap(ARM64_WORKAROUND_PMUV3_IMPDEF_TRAPS))-returnID_AA64DFR0_EL1_PMUVer_IMP;--/*-*Otherwise,treatIMPLEMENTATIONDEFINEDfunctionalityas-*unimplemented-*/-if(pmuver==ID_AA64DFR0_EL1_PMUVer_IMP_DEF)-return0;--returnmin(pmuver,ID_AA64DFR0_EL1_PMUVer_V3P5);-}--/**-*kvm_vcpu_read_pmcr-ReadPMCR_EL0registerforthevCPU-*@vcpu:Thevcpupointer-*/-u64kvm_vcpu_read_pmcr(structkvm_vcpu*vcpu)-{-u64pmcr=__vcpu_sys_reg(vcpu,PMCR_EL0);-u64n=vcpu->kvm->arch.nr_pmu_counters;--if(vcpu_has_nv(vcpu)&&!vcpu_is_el2(vcpu))-n=FIELD_GET(MDCR_EL2_HPMN,__vcpu_sys_reg(vcpu,MDCR_EL2));--returnu64_replace_bits(pmcr,n,ARMV8_PMU_PMCR_N);-}-voidkvm_pmu_nested_transition(structkvm_vcpu*vcpu){boolreprogrammed=false;
@@ -209,3 +222,665 @@ void kvm_vcpu_pmu_resync_el0(void)kvm_make_request(KVM_REQ_RESYNC_PMU_EL0,vcpu);}++voidkvm_host_pmu_init(structarm_pmu*pmu)+{+structarm_pmu_entry*entry;++/*+*CheckthesanitisedPMUversionforthesystem,asKVMdoesnot+*supportimplementationswherePMUv3existsonasubsetofCPUs.+*/+if(!pmuv3_implemented(kvm_arm_pmu_get_pmuver_limit()))+return;++guard(mutex)(&arm_pmus_lock);++entry=kmalloc(sizeof(*entry),GFP_KERNEL);+if(!entry)+return;++entry->arm_pmu=pmu;+list_add_tail(&entry->entry,&arm_pmus);+}++staticstructarm_pmu*kvm_pmu_probe_armpmu(void)+{+structarm_pmu_entry*entry;+structarm_pmu*pmu;+intcpu;++guard(mutex)(&arm_pmus_lock);++/*+*ItissafetouseastalecputoiteratethelistofPMUssolongas+*thesamevalueisusedfortheentiretyoftheloop.Giventhis,and+*thefactthatnopercpudataisusedforthelookupthereisnoneed+*todisablepreemption.+*+*Itisstillnecessarytogetavalidcpu,though,toprobeforthe+*defaultPMUinstanceasuserspaceisnotrequiredtospecifyaPMU+*type.InordertoupholdthepreexistingbehaviorKVMselectsthe+*PMUinstanceforthecoreduringvcpuinit.Adependentuse+*casewouldbeauserwithdisdainofallthingsbig.LITTLEthat+*affinestheVMMtoaparticularclusterofcores.+*+*Inanycase,userspaceshouldjustdothesanethingandusetheUAPI+*toselectaPMUtypedirectly.But,bewaryofthebaggagebeing+*carriedhere.+*/+cpu=raw_smp_processor_id();+list_for_each_entry(entry,&arm_pmus,entry){+pmu=entry->arm_pmu;++if(cpumask_test_cpu(cpu,&pmu->supported_cpus))+returnpmu;+}++returnNULL;+}++staticu64__compute_pmceid(structarm_pmu*pmu,boolpmceid1)+{+u32hi[2],lo[2];++bitmap_to_arr32(lo,pmu->pmceid_bitmap,ARMV8_PMUV3_MAX_COMMON_EVENTS);+bitmap_to_arr32(hi,pmu->pmceid_ext_bitmap,ARMV8_PMUV3_MAX_COMMON_EVENTS);++return((u64)hi[pmceid1]<<32)|lo[pmceid1];+}++staticu64compute_pmceid0(structarm_pmu*pmu)+{+u64val=__compute_pmceid(pmu,0);++/* always support SW_INCR */+val|=BIT(ARMV8_PMUV3_PERFCTR_SW_INCR);+/* always support CHAIN */+val|=BIT(ARMV8_PMUV3_PERFCTR_CHAIN);+returnval;+}++staticu64compute_pmceid1(structarm_pmu*pmu)+{+u64val=__compute_pmceid(pmu,1);++/*+*Don'tadvertiseSTALL_SLOT*,asPMMIR_EL0ishandled+*asRAZ+*/+val&=~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT-32)|+BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND-32)|+BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND-32));+returnval;+}++u64kvm_pmu_get_pmceid(structkvm_vcpu*vcpu,boolpmceid1)+{+structarm_pmu*cpu_pmu=vcpu->kvm->arch.arm_pmu;+unsignedlong*bmap=vcpu->kvm->arch.pmu_filter;+u64val,mask=0;+intbase,i,nr_events;++if(!pmceid1){+val=compute_pmceid0(cpu_pmu);+base=0;+}else{+val=compute_pmceid1(cpu_pmu);+base=32;+}++if(!bmap)+returnval;++nr_events=kvm_pmu_event_mask(vcpu->kvm)+1;++for(i=0;i<32;i+=8){+u64byte;++byte=bitmap_get_value8(bmap,base+i);+mask|=byte<<i;+if(nr_events>=(0x4000+base+32)){+byte=bitmap_get_value8(bmap,0x4000+base+i);+mask|=byte<<(32+i);+}+}++returnval&mask;+}++/*+*WhenperfinterruptisanNMI,wecannotsafelynotifythevcpucorresponding+*totheevent.+*ThisiswhyweneedacallbacktodoitonceoutsideoftheNMIcontext.+*/+staticvoidkvm_pmu_perf_overflow_notify_vcpu(structirq_work*work)+{+structkvm_vcpu*vcpu;++vcpu=container_of(work,structkvm_vcpu,arch.pmu.overflow_work);+kvm_vcpu_kick(vcpu);+}++staticu32__kvm_pmu_event_mask(unsignedintpmuver)+{+switch(pmuver){+caseID_AA64DFR0_EL1_PMUVer_IMP:+returnGENMASK(9,0);+caseID_AA64DFR0_EL1_PMUVer_V3P1:+caseID_AA64DFR0_EL1_PMUVer_V3P4:+caseID_AA64DFR0_EL1_PMUVer_V3P5:+caseID_AA64DFR0_EL1_PMUVer_V3P7:+returnGENMASK(15,0);+default:/* Shouldn't be here, just for sanity */+WARN_ONCE(1,"Unknown PMU version %d\n",pmuver);+return0;+}+}++u32kvm_pmu_event_mask(structkvm*kvm)+{+u64dfr0=kvm_read_vm_id_reg(kvm,SYS_ID_AA64DFR0_EL1);+u8pmuver=SYS_FIELD_GET(ID_AA64DFR0_EL1,PMUVer,dfr0);++return__kvm_pmu_event_mask(pmuver);+}++u64kvm_pmu_evtyper_mask(structkvm*kvm)+{+u64mask=ARMV8_PMU_EXCLUDE_EL1|ARMV8_PMU_EXCLUDE_EL0|+kvm_pmu_event_mask(kvm);++if(kvm_has_feat(kvm,ID_AA64PFR0_EL1,EL2,IMP))+mask|=ARMV8_PMU_INCLUDE_EL2;++if(kvm_has_feat(kvm,ID_AA64PFR0_EL1,EL3,IMP))+mask|=ARMV8_PMU_EXCLUDE_NS_EL0|+ARMV8_PMU_EXCLUDE_NS_EL1|+ARMV8_PMU_EXCLUDE_EL3;++returnmask;+}++staticvoidkvm_pmu_update_state(structkvm_vcpu*vcpu)+{+structkvm_pmu*pmu=&vcpu->arch.pmu;+booloverflow;++overflow=kvm_pmu_overflow_status(vcpu);+if(pmu->irq_level==overflow)+return;++pmu->irq_level=overflow;++if(likely(irqchip_in_kernel(vcpu->kvm))){+intret=kvm_vgic_inject_irq(vcpu->kvm,vcpu,+pmu->irq_num,overflow,pmu);+WARN_ON(ret);+}+}++/**+*kvm_pmu_flush_hwstate-flushpmustatetocpu+*@vcpu:Thevcpupointer+*+*CheckifthePMUhasoverflowedwhilewewererunninginthehost,andinject+*aninterruptifthatwasthecase.+*/+voidkvm_pmu_flush_hwstate(structkvm_vcpu*vcpu)+{+kvm_pmu_update_state(vcpu);+}++/**+*kvm_pmu_sync_hwstate-syncpmustatefromcpu+*@vcpu:Thevcpupointer+*+*CheckifthePMUhasoverflowedwhilewewererunningintheguest,and+*injectaninterruptifthatwasthecase.+*/+voidkvm_pmu_sync_hwstate(structkvm_vcpu*vcpu)+{+kvm_pmu_update_state(vcpu);+}++intkvm_arm_pmu_v3_enable(structkvm_vcpu*vcpu)+{+if(!vcpu->arch.pmu.created)+return-EINVAL;++/*+*AvalidinterruptconfigurationforthePMUiseithertohavea+*properlyconfiguredinterruptnumberandusinganin-kernel+*irqchip,ortonothaveanin-kernelGICandnotsetanIRQ.+*/+if(irqchip_in_kernel(vcpu->kvm)){+intirq=vcpu->arch.pmu.irq_num;+/*+*Ifweareusinganin-kernelvgic,atthispointweknow+*thevgicwillbeinitialized,sowecancheckthePMUirq+*numberagainstthedimensionsofthevgicandmakesure+*it'svalid.+*/+if(!irq_is_ppi(irq)&&!vgic_valid_spi(vcpu->kvm,irq))+return-EINVAL;+}elseif(kvm_arm_pmu_irq_initialized(vcpu)){+return-EINVAL;+}++return0;+}++staticintkvm_arm_pmu_v3_init(structkvm_vcpu*vcpu)+{+if(irqchip_in_kernel(vcpu->kvm)){+intret;++/*+*IfusingthePMUwithanin-kernelvirtualGIC+*implementation,werequiretheGICtobealready+*initializedwheninitializingthePMU.+*/+if(!vgic_initialized(vcpu->kvm))+return-ENODEV;++if(!kvm_arm_pmu_irq_initialized(vcpu))+return-ENXIO;++ret=kvm_vgic_set_owner(vcpu,vcpu->arch.pmu.irq_num,+&vcpu->arch.pmu);+if(ret)+returnret;+}++init_irq_work(&vcpu->arch.pmu.overflow_work,+kvm_pmu_perf_overflow_notify_vcpu);++vcpu->arch.pmu.created=true;+return0;+}++/*+*ForoneVMtheinterrupttypemustbesameforeachvcpu.+*AsaPPI,theinterruptnumberisthesameforallvcpus,+*whileasanSPIitmustbeaseparatenumberpervcpu.+*/+staticboolpmu_irq_is_valid(structkvm*kvm,intirq)+{+unsignedlongi;+structkvm_vcpu*vcpu;++kvm_for_each_vcpu(i,vcpu,kvm){+if(!kvm_arm_pmu_irq_initialized(vcpu))+continue;++if(irq_is_ppi(irq)){+if(vcpu->arch.pmu.irq_num!=irq)+returnfalse;+}else{+if(vcpu->arch.pmu.irq_num==irq)+returnfalse;+}+}++returntrue;+}++/**+*kvm_arm_pmu_get_max_counters-ReturnthemaxnumberofPMUcounters.+*@kvm:Thekvmpointer+*/+u8kvm_arm_pmu_get_max_counters(structkvm*kvm)+{+structarm_pmu*arm_pmu=kvm->arch.arm_pmu;++/*+*PMUv3requiresthatalleventcountersarecapableofcountingany+*event,thoughthesamemaynotbetrueofnon-PMUv3hardware.+*/+if(cpus_have_final_cap(ARM64_WORKAROUND_PMUV3_IMPDEF_TRAPS))+return1;++/*+*Thearm_pmu->cntr_maskconsidersthefixedcounter(s)aswell.+*Ignorethoseandreturnonlythegeneral-purposecounters.+*/+returnbitmap_weight(arm_pmu->cntr_mask,ARMV8_PMU_MAX_GENERAL_COUNTERS);+}++staticvoidkvm_arm_set_nr_counters(structkvm*kvm,unsignedintnr)+{+kvm->arch.nr_pmu_counters=nr;++/* Reset MDCR_EL2.HPMN behind the vcpus' back... */+if(test_bit(KVM_ARM_VCPU_HAS_EL2,kvm->arch.vcpu_features)){+structkvm_vcpu*vcpu;+unsignedlongi;++kvm_for_each_vcpu(i,vcpu,kvm){+u64val=__vcpu_sys_reg(vcpu,MDCR_EL2);++val&=~MDCR_EL2_HPMN;+val|=FIELD_PREP(MDCR_EL2_HPMN,kvm->arch.nr_pmu_counters);+__vcpu_assign_sys_reg(vcpu,MDCR_EL2,val);+}+}+}++staticvoidkvm_arm_set_pmu(structkvm*kvm,structarm_pmu*arm_pmu)+{+lockdep_assert_held(&kvm->arch.config_lock);++kvm->arch.arm_pmu=arm_pmu;+kvm_arm_set_nr_counters(kvm,kvm_arm_pmu_get_max_counters(kvm));+}++/**+*kvm_arm_set_default_pmu-NoPMUset,getthedefaultone.+*@kvm:Thekvmpointer+*+*Theobservantamongyouwillnoticethatthesupported_cpus+*maskdoesnotgetupdatedforthedefaultPMUeventhoughit+*isquitepossibletheselectedinstancesupportsonlya+*subsetofcoresinthesystem.Thisisintentional,and+*upholdsthepreexistingbehavioronheterogeneoussystems+*wherevCPUscanbescheduledonanycorebuttheguest+*counterscouldstopworking.+*/+intkvm_arm_set_default_pmu(structkvm*kvm)+{+structarm_pmu*arm_pmu=kvm_pmu_probe_armpmu();++if(!arm_pmu)+return-ENODEV;++kvm_arm_set_pmu(kvm,arm_pmu);+return0;+}++staticintkvm_arm_pmu_v3_set_pmu(structkvm_vcpu*vcpu,intpmu_id)+{+structkvm*kvm=vcpu->kvm;+structarm_pmu_entry*entry;+structarm_pmu*arm_pmu;+intret=-ENXIO;++lockdep_assert_held(&kvm->arch.config_lock);+mutex_lock(&arm_pmus_lock);++list_for_each_entry(entry,&arm_pmus,entry){+arm_pmu=entry->arm_pmu;+if(arm_pmu->pmu.type==pmu_id){+if(kvm_vm_has_ran_once(kvm)||+(kvm->arch.pmu_filter&&kvm->arch.arm_pmu!=arm_pmu)){+ret=-EBUSY;+break;+}++kvm_arm_set_pmu(kvm,arm_pmu);+cpumask_copy(kvm->arch.supported_cpus,&arm_pmu->supported_cpus);+ret=0;+break;+}+}++mutex_unlock(&arm_pmus_lock);+returnret;+}++staticintkvm_arm_pmu_v3_set_nr_counters(structkvm_vcpu*vcpu,unsignedintn)+{+structkvm*kvm=vcpu->kvm;++if(!kvm->arch.arm_pmu)+return-EINVAL;++if(n>kvm_arm_pmu_get_max_counters(kvm))+return-EINVAL;++kvm_arm_set_nr_counters(kvm,n);+return0;+}++intkvm_arm_pmu_v3_set_attr(structkvm_vcpu*vcpu,structkvm_device_attr*attr)+{+structkvm*kvm=vcpu->kvm;++lockdep_assert_held(&kvm->arch.config_lock);++if(!kvm_vcpu_has_pmu(vcpu))+return-ENODEV;++if(vcpu->arch.pmu.created)+return-EBUSY;++switch(attr->attr){+caseKVM_ARM_VCPU_PMU_V3_IRQ:{+int__user*uaddr=(int__user*)(long)attr->addr;+intirq;++if(!irqchip_in_kernel(kvm))+return-EINVAL;++if(get_user(irq,uaddr))+return-EFAULT;++/* The PMU overflow interrupt can be a PPI or a valid SPI. */+if(!(irq_is_ppi(irq)||irq_is_spi(irq)))+return-EINVAL;++if(!pmu_irq_is_valid(kvm,irq))+return-EINVAL;++if(kvm_arm_pmu_irq_initialized(vcpu))+return-EBUSY;++kvm_debug("Set kvm ARM PMU irq: %d\n",irq);+vcpu->arch.pmu.irq_num=irq;+return0;+}+caseKVM_ARM_VCPU_PMU_V3_FILTER:{+u8pmuver=kvm_arm_pmu_get_pmuver_limit();+structkvm_pmu_event_filter__user*uaddr;+structkvm_pmu_event_filterfilter;+intnr_events;++/*+*Allowuserspacetospecifyaneventfilterfortheentire+*eventrangesupportedbyPMUVerofthehardware,rather+*thantheguest'sPMUVerforKVMbackwardcompatibility.+*/+nr_events=__kvm_pmu_event_mask(pmuver)+1;++uaddr=(structkvm_pmu_event_filter__user*)(long)attr->addr;++if(copy_from_user(&filter,uaddr,sizeof(filter)))+return-EFAULT;++if(((u32)filter.base_event+filter.nevents)>nr_events||+(filter.action!=KVM_PMU_EVENT_ALLOW&&+filter.action!=KVM_PMU_EVENT_DENY))+return-EINVAL;++if(kvm_vm_has_ran_once(kvm))+return-EBUSY;++if(!kvm->arch.pmu_filter){+kvm->arch.pmu_filter=bitmap_alloc(nr_events,GFP_KERNEL_ACCOUNT);+if(!kvm->arch.pmu_filter)+return-ENOMEM;++/*+*Thedefaultdependsonthefirstappliedfilter.+*Ifitallowsevents,thedefaultistodeny.+*Conversely,ifthefirstfilterdeniesasetof+*events,thedefaultistoallow.+*/+if(filter.action==KVM_PMU_EVENT_ALLOW)+bitmap_zero(kvm->arch.pmu_filter,nr_events);+else+bitmap_fill(kvm->arch.pmu_filter,nr_events);+}++if(filter.action==KVM_PMU_EVENT_ALLOW)+bitmap_set(kvm->arch.pmu_filter,filter.base_event,filter.nevents);+else+bitmap_clear(kvm->arch.pmu_filter,filter.base_event,filter.nevents);++return0;+}+caseKVM_ARM_VCPU_PMU_V3_SET_PMU:{+int__user*uaddr=(int__user*)(long)attr->addr;+intpmu_id;++if(get_user(pmu_id,uaddr))+return-EFAULT;++returnkvm_arm_pmu_v3_set_pmu(vcpu,pmu_id);+}+caseKVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS:{+unsignedint__user*uaddr=(unsignedint__user*)(long)attr->addr;+unsignedintn;++if(get_user(n,uaddr))+return-EFAULT;++returnkvm_arm_pmu_v3_set_nr_counters(vcpu,n);+}+caseKVM_ARM_VCPU_PMU_V3_INIT:+returnkvm_arm_pmu_v3_init(vcpu);+}++return-ENXIO;+}++intkvm_arm_pmu_v3_get_attr(structkvm_vcpu*vcpu,structkvm_device_attr*attr)+{+switch(attr->attr){+caseKVM_ARM_VCPU_PMU_V3_IRQ:{+int__user*uaddr=(int__user*)(long)attr->addr;+intirq;++if(!irqchip_in_kernel(vcpu->kvm))+return-EINVAL;++if(!kvm_vcpu_has_pmu(vcpu))+return-ENODEV;++if(!kvm_arm_pmu_irq_initialized(vcpu))+return-ENXIO;++irq=vcpu->arch.pmu.irq_num;+returnput_user(irq,uaddr);+}+}++return-ENXIO;+}++intkvm_arm_pmu_v3_has_attr(structkvm_vcpu*vcpu,structkvm_device_attr*attr)+{+switch(attr->attr){+caseKVM_ARM_VCPU_PMU_V3_IRQ:+caseKVM_ARM_VCPU_PMU_V3_INIT:+caseKVM_ARM_VCPU_PMU_V3_FILTER:+caseKVM_ARM_VCPU_PMU_V3_SET_PMU:+caseKVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS:+if(kvm_vcpu_has_pmu(vcpu))+return0;+}++return-ENXIO;+}++u8kvm_arm_pmu_get_pmuver_limit(void)+{+unsignedintpmuver;++pmuver=SYS_FIELD_GET(ID_AA64DFR0_EL1,PMUVer,+read_sanitised_ftr_reg(SYS_ID_AA64DFR0_EL1));++/*+*SpoofabarebonesPMUv3implementationifthesystemsupportsIMPDEF+*trapsofthePMUv3sysregs+*/+if(cpus_have_final_cap(ARM64_WORKAROUND_PMUV3_IMPDEF_TRAPS))+returnID_AA64DFR0_EL1_PMUVer_IMP;++/*+*Otherwise,treatIMPLEMENTATIONDEFINEDfunctionalityas+*unimplemented+*/+if(pmuver==ID_AA64DFR0_EL1_PMUVer_IMP_DEF)+return0;++returnmin(pmuver,ID_AA64DFR0_EL1_PMUVer_V3P5);+}++u64kvm_pmu_implemented_counter_mask(structkvm_vcpu*vcpu)+{+u64val=FIELD_GET(ARMV8_PMU_PMCR_N,kvm_vcpu_read_pmcr(vcpu));++if(val==0)+returnBIT(ARMV8_PMU_CYCLE_IDX);+else+returnGENMASK(val-1,0)|BIT(ARMV8_PMU_CYCLE_IDX);+}++u64kvm_pmu_hyp_counter_mask(structkvm_vcpu*vcpu)+{+unsignedinthpmn,n;++if(!vcpu_has_nv(vcpu))+return0;++hpmn=SYS_FIELD_GET(MDCR_EL2,HPMN,__vcpu_sys_reg(vcpu,MDCR_EL2));+n=vcpu->kvm->arch.nr_pmu_counters;++/*+*ProgrammingHPMNtoavaluegreaterthanPMCR_EL0.Nis+*CONSTRAINEDUNPREDICTABLE.Maketheimplementationchoicethatan+*UNKNOWNnumberofcounters(inourcase,zero)arereservedforEL2.+*/+if(hpmn>=n)+return0;++/*+*ProgrammingHPMN=0isCONSTRAINEDUNPREDICTABLEifFEAT_HPMN0isn't+*implemented.SinceKVM'sabilitytoemulateHPMN=0doesnotdirectly+*dependonhardware(allPMUregistersaretrapped),makethe+*implementationchoicethatallcountersareincludedinthesecond+*rangereservedforEL2/EL3.+*/+returnGENMASK(n-1,hpmn);+}++boolkvm_pmu_counter_is_hyp(structkvm_vcpu*vcpu,unsignedintidx)+{+returnkvm_pmu_hyp_counter_mask(vcpu)&BIT(idx);+}++u64kvm_pmu_accessible_counter_mask(structkvm_vcpu*vcpu)+{+u64mask=kvm_pmu_implemented_counter_mask(vcpu);++if(!vcpu_has_nv(vcpu)||vcpu_is_el2(vcpu))+returnmask;++returnmask&~kvm_pmu_hyp_counter_mask(vcpu);+}++/**+*kvm_vcpu_read_pmcr-ReadPMCR_EL0registerforthevCPU+*@vcpu:Thevcpupointer+*/+u64kvm_vcpu_read_pmcr(structkvm_vcpu*vcpu)+{+u64pmcr=__vcpu_sys_reg(vcpu,PMCR_EL0);+u64n=vcpu->kvm->arch.nr_pmu_counters;++if(vcpu_has_nv(vcpu)&&!vcpu_is_el2(vcpu))+n=FIELD_GET(MDCR_EL2_HPMN,__vcpu_sys_reg(vcpu,MDCR_EL2));++returnu64_replace_bits(pmcr,n,ARMV8_PMU_PMCR_N);+}
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:34
For PMUv3, the register field MDCR_EL2.HPMN partitiones the PMU
counters into two ranges where counters 0..HPMN-1 are accessible by
EL1 and, if allowed, EL0 while counters HPMN..N are only accessible by
EL2.
Create module parameter reserved_host_counters to reserve a number of
counters for the host. This number is set at boot because the perf
subsystem assumes the number of counters will not change after the PMU
is probed.
Introduce the function armv8pmu_partition() to modify the PMU driver's
cntr_mask of available counters to exclude the counters being reserved
for the guest and record reserved_guest_counters as the maximum
allowable value for HPMN.
Due to the difficulty this feature would create for the driver running
in nVHE mode, partitioning is only allowed in VHE mode. In order to
support a partitioning on nVHE we'd need to explicitly disable guest
counters on every exit and reset HPMN to place all counters in the
first range.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm/include/asm/arm_pmuv3.h | 4 ++
arch/arm64/include/asm/arm_pmuv3.h | 5 ++
arch/arm64/include/asm/kvm_pmu.h | 8 +++
arch/arm64/kvm/Makefile | 2 +-
arch/arm64/kvm/pmu-direct.c | 22 +++++++++
drivers/perf/arm_pmuv3.c | 78 +++++++++++++++++++++++++++++-
include/linux/perf/arm_pmu.h | 1 +
7 files changed, 117 insertions(+), 3 deletions(-)
create mode 100644 arch/arm64/kvm/pmu-direct.c
@@ -1299,6 +1311,61 @@ struct armv8pmu_probe_info {boolpresent;};+/**+*armv8pmu_reservation_is_valid()-Determineifreservationisallowed+*@host_counters:Numberofhostcounterstoreserve+*+*Determineifthenumberofhostcountersintheargumentisan+*allowedreservation,0toNR_COUNTERSinclusive.+*+*Return:Trueifreservationallowed,falseotherwise+*/+staticboolarmv8pmu_reservation_is_valid(inthost_counters)+{+returnhost_counters>=0&&+host_counters<=armv8pmu_pmcr_n_read();+}++/**+*armv8pmu_partition()-PartitionthePMU+*@pmu:Pointertopmubeingpartitioned+*@host_counters:Numberofhostcounterstoreserve+*+*PartitionthegivenPMUbytakinganumberofhostcountersto+*reserveand,ifitisavalidreservation,recordingthe+*correspondingHPMNvalueinthehpmn_maxfieldofthePMUand+*clearingtheguest-reservedcountersfromthecountermask.+*+*Return:0onsuccess,-ERRORotherwise+*/+staticintarmv8pmu_partition(structarm_pmu*pmu,inthost_counters)+{+u8nr_counters;+u8hpmn;++if(!armv8pmu_reservation_is_valid(host_counters)){+pr_err("PMU partition reservation of %d host counters is not valid",host_counters);+return-EINVAL;+}++nr_counters=armv8pmu_pmcr_n_read();+hpmn=nr_counters-host_counters;++pmu->hpmn_max=hpmn;+armv8pmu_hpmn_max=hpmn;++bitmap_clear(pmu->cntr_mask,0,hpmn);+bitmap_set(pmu->cntr_mask,hpmn,host_counters);+clear_bit(ARMV8_PMU_CYCLE_IDX,pmu->cntr_mask);++if(pmuv3_has_icntr())+clear_bit(ARMV8_PMU_INSTR_IDX,pmu->cntr_mask);++pr_info("Partitioned PMU with %d host counters -> %u guest counters",host_counters,hpmn);++return0;+}+staticvoid__armv8pmu_probe_pmu(void*info){structarmv8pmu_probe_info*probe=info;
@@ -1313,10 +1380,10 @@ static void __armv8pmu_probe_pmu(void *info)cpu_pmu->pmuver=pmuver;probe->present=true;+cpu_pmu->hpmn_max=-1;/* Read the nb of CNTx counters supported from PMNC */-bitmap_set(cpu_pmu->cntr_mask,-0,FIELD_GET(ARMV8_PMU_PMCR_N,armv8pmu_pmcr_read()));+bitmap_set(cpu_pmu->cntr_mask,0,armv8pmu_pmcr_n_read());/* Add the CPU cycles counter */set_bit(ARMV8_PMU_CYCLE_IDX,cpu_pmu->cntr_mask);
@@ -1325,6 +1392,13 @@ static void __armv8pmu_probe_pmu(void *info)if(pmuv3_has_icntr())set_bit(ARMV8_PMU_INSTR_IDX,cpu_pmu->cntr_mask);+if(reserved_host_counters>=0){+if(kvm_pmu_partition_supported())+armv8pmu_partition(cpu_pmu,reserved_host_counters);+else+pr_err("PMU partition is not supported");+}+pmceid[0]=pmceid_raw[0]=read_pmceid0();pmceid[1]=pmceid_raw[1]=read_pmceid1();
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:35
The OVSR bitmasks are valid for enable and interrupt registers as well as
overflow registers. Generalize the names.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Colton Lewis <redacted>
---
drivers/perf/arm_pmuv3.c | 4 ++--
include/linux/perf/arm_pmuv3.h | 14 +++++++-------
2 files changed, 9 insertions(+), 9 deletions(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:36
If the PMU is partitioned, keep the driver out of the guest counter
partition and only use the host counter partition.
Define some functions that determine whether the PMU is partitioned
and construct mutually exclusive bitmaps for testing which partition a
particular counter is in. Note that despite their separate position in
the bitmap, the cycle and instruction counters are always in the guest
partition.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm/include/asm/arm_pmuv3.h | 18 +++++++
arch/arm64/include/asm/kvm_pmu.h | 24 +++++++++
arch/arm64/kvm/pmu-direct.c | 86 ++++++++++++++++++++++++++++++++
drivers/perf/arm_pmuv3.c | 41 +++++++++++++--
4 files changed, 165 insertions(+), 4 deletions(-)
@@ -1044,6 +1057,7 @@ static int armv8pmu_get_event_idx(struct pmu_hw_events *cpuc,*maynotknowhowtohandleit.*/if((evtype==ARMV8_PMUV3_PERFCTR_INST_RETIRED)&&+!kvm_pmu_is_partitioned(cpu_pmu)&&!armv8pmu_event_get_threshold(&event->attr)&&test_bit(ARMV8_PMU_INSTR_IDX,cpu_pmu->cntr_mask)&&!armv8pmu_event_want_user_access(event)){
@@ -1055,7 +1069,7 @@ static int armv8pmu_get_event_idx(struct pmu_hw_events *cpuc,*Otherwiseuseeventscounters*/if(armv8pmu_event_is_chained(event))-returnarmv8pmu_get_chain_idx(cpuc,cpu_pmu);+returnarmv8pmu_get_chain_idx(cpuc,cpu_pmu);elsereturnarmv8pmu_get_single_idx(cpuc,cpu_pmu);}
@@ -1167,6 +1181,14 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,return0;}+staticvoidarmv8pmu_reset_host_counters(structarm_pmu*cpu_pmu)+{+intidx;++for_each_set_bit(idx,cpu_pmu->cntr_mask,ARMV8_PMU_MAX_GENERAL_COUNTERS)+armv8pmu_write_evcntr(idx,0);+}+staticvoidarmv8pmu_reset(void*info){structarm_pmu*cpu_pmu=(structarm_pmu*)info;
@@ -1174,6 +1196,9 @@ static void armv8pmu_reset(void *info)bitmap_to_arr64(&mask,cpu_pmu->cntr_mask,ARMPMU_MAX_HWEVENTS);+if(kvm_pmu_is_partitioned(cpu_pmu))+mask&=kvm_pmu_host_counter_mask(cpu_pmu);+/* The counter and interrupt enable registers are unknown at reset. */armv8pmu_disable_counter(mask);armv8pmu_disable_intens(mask);
@@ -1186,11 +1211,19 @@ static void armv8pmu_reset(void *info)brbe_invalidate();}+pmcr=ARMV8_PMU_PMCR_LC;+/*-*Initialize&ResetPMNC.Requestoverflowinterruptfor-*64bitcyclecounterbutcheatinarmv8pmu_write_counter().+*Initialize&ResetPMNC.Requestoverflowinterruptfor64+*bitcyclecounterbutcheatinarmv8pmu_write_counter().+*+*Whenpartitioned,thereisnosinglebittoresetonlythe+*hostcounters.soresetthemindividually.*/-pmcr=ARMV8_PMU_PMCR_P|ARMV8_PMU_PMCR_C|ARMV8_PMU_PMCR_LC;+if(kvm_pmu_is_partitioned(cpu_pmu))+armv8pmu_reset_host_counters(cpu_pmu);+else+pmcr=ARMV8_PMU_PMCR_P|ARMV8_PMU_PMCR_C;/* Enable long event counter support where available */if(armv8pmu_has_long_event(cpu_pmu))
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:39
In order to gain the best performance benefit from partitioning the
PMU, utilize fine grain traps (FEAT_FGT and FEAT_FGT2) to avoid
trapping common PMU register accesses by the guest to remove that
overhead.
Untrapped:
* PMCR_EL0
* PMUSERENR_EL0
* PMSELR_EL0
* PMCCNTR_EL0
* PMCNTEN_EL0
* PMINTEN_EL1
* PMEVCNTRn_EL0
These are safe to untrap because writing MDCR_EL2.HPMN as this series
will do limits the effect of writes to any of these registers to the
partition of counters 0..HPMN-1. Reads from these registers will not
leak information from between guests as all these registers are
context swapped by a later patch in this series. Reads from these
registers also do not leak any information about the host's hardware
beyond what is promised by PMUv3.
Trapped:
* PMOVS_EL0
* PMEVTYPERn_EL0
* PMCCFILTR_EL0
* PMICNTR_EL0
* PMICFILTR_EL0
* PMCEIDn_EL0
* PMMIR_EL1
PMOVS remains trapped so KVM can track overflow IRQs that will need to
be injected into the guest.
PMICNTR and PMIFILTR remain trapped because KVM is not handling them
yet.
PMEVTYPERn remains trapped so KVM can limit which events guests can
count, such as disallowing counting at EL2. PMCCFILTR and PMCIFILTR
are special cases of the same.
PMCEIDn and PMMIR remain trapped because they can leak information
specific to the host hardware implementation.
NOTE: This patch temporarily forces kvm_vcpu_pmu_is_partitioned() to
be false to prevent partial feature activation for easier debugging.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/kvm_pmu.h | 33 ++++++++++++++++++++++
arch/arm64/kvm/config.c | 34 ++++++++++++++++++++--
arch/arm64/kvm/pmu-direct.c | 48 ++++++++++++++++++++++++++++++++
3 files changed, 112 insertions(+), 3 deletions(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:40
Because PMOVS remains trapped, it needs to be written through when
partitioned to affect PMU hardware when expected.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/arm_pmuv3.h | 10 ++++++++++
arch/arm64/kvm/sys_regs.c | 17 ++++++++++++++++-
2 files changed, 26 insertions(+), 1 deletion(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:40
With FGT in place, the remaining trapped registers need to be written
through to the underlying physical registers as well as the virtual
ones. Failing to do this means guest writes will not take effect when
expected.
For the PMEVTYPER register, take care to enforce KVM's PMU event
filter. Do that by setting the bits to exclude EL1 and EL0 when an
event is not present in the filter and clearing the bit to include EL2
always.
Note the virtual register is always assigned the value specified by
the guest to hide the setting of those bits.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/kvm/sys_regs.c | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:40
Because PMXEVTYPER is trapped and PMSELR is not, it is not appropriate
to use the virtual PMSELR register when it could be outdated and lead
to an invalid write. Use the physical register when partitioned.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/arm_pmuv3.h | 7 ++++++-
arch/arm64/kvm/sys_regs.c | 9 +++++++--
2 files changed, 13 insertions(+), 3 deletions(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:43
We may want a partitioned PMU but not have FEAT_FGT to untrap the
specific registers that would normally be untrapped. Add a handler for
those registers in the fast path so we can still get a performance
boost from partitioning.
The idea is to handle traps for all the PMU registers quickly by
writing through to the hardware as possible instead of hooking into
the emulated vPMU as the standard handlers in sys_regs.c do. Since
context switching will not happen without FGT, make sure to write both
physical and virtual registers so they stay in sync.
To assist with that, fill of the gaps in arm_pmuv3.h for helper
functions to read and write PMU registers and lift some of the access
checking functions from sys_regs.c to pmu.c
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/arm_pmuv3.h | 37 ++++-
arch/arm64/include/asm/kvm_pmu.h | 25 +++
arch/arm64/kvm/hyp/include/hyp/switch.h | 201 ++++++++++++++++++++++++
arch/arm64/kvm/pmu.c | 40 +++++
arch/arm64/kvm/sys_regs.c | 41 +----
5 files changed, 303 insertions(+), 41 deletions(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:44
Setup MDCR_EL2 to handle a partitioned PMU. That means calculate an
appropriate value for HPMN instead of the default maximum setting the
host allows (which implies no partition) so hardware enforces that a
guest will only see the counters in the guest partition.
Setting HPMN to a non default value means the global enable bit for
the host counters is now MDCR_EL2.HPME instead of the usual
PMCR_EL0.E. Enable the HPME bit to allow the host to count guest
events. Since HPME only has an effect when HPMN is set which we only
do for the guest, it is correct to enable it unconditionally here.
Unset the TPM and TPMCR bits, which trap all PMU accesses, if
FGT (fine grain trapping) is being used.
If available, set the filtering bits HPMD and HCCD to be extra sure
nothing in the guest counts at EL2.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/kvm_pmu.h | 11 ++++++
arch/arm64/kvm/debug.c | 29 ++++++++++++--
arch/arm64/kvm/pmu-direct.c | 65 ++++++++++++++++++++++++++++++++
3 files changed, 102 insertions(+), 3 deletions(-)
@@ -36,20 +36,43 @@ static int cpu_has_spe(u64 dfr0)*/staticvoidkvm_arm_setup_mdcr_el2(structkvm_vcpu*vcpu){+inthpmn=kvm_pmu_hpmn(vcpu);+preempt_disable();/**ThisalsoclearsMDCR_EL2_E2PB_MASKandMDCR_EL2_E2TB_MASK*todisableguestaccesstotheprofilingandtracebuffers*/-vcpu->arch.mdcr_el2=FIELD_PREP(MDCR_EL2_HPMN,-*host_data_ptr(nr_event_counters));++vcpu->arch.mdcr_el2=FIELD_PREP(MDCR_EL2_HPMN,hpmn);vcpu->arch.mdcr_el2|=(MDCR_EL2_TPM|MDCR_EL2_TPMS|MDCR_EL2_TTRF|MDCR_EL2_TPMCR|MDCR_EL2_TDRA|-MDCR_EL2_TDOSA);+MDCR_EL2_TDOSA|+MDCR_EL2_HPME);++if(kvm_vcpu_pmu_is_partitioned(vcpu)){+/*+*Filteringtheseshouldberedundantbecausewetrap+*alltheTYPERandFILTRregistersanywayandensure+*theyfilterEL2,butsetthebitsiftheyarehere.+*/+if(is_pmuv3p1(read_pmuver()))+vcpu->arch.mdcr_el2|=MDCR_EL2_HPMD;+if(is_pmuv3p5(read_pmuver()))+vcpu->arch.mdcr_el2|=MDCR_EL2_HCCD;++/*+*Takeoutthecoarsegraintrapsifweareusing+*finegraintraps.+*/+if(kvm_vcpu_pmu_use_fgt(vcpu))+vcpu->arch.mdcr_el2&=~(MDCR_EL2_TPM|MDCR_EL2_TPMCR);++}/* Is the VM being debugged by userspace? */if(vcpu->guest_debug)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:45
Make sure reads and writes to PMCR_EL0 conform to additional
constraints imposed when the PMU is partitioned.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/kvm/pmu.c | 2 +-
arch/arm64/kvm/sys_regs.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:47
The KVM API for event filtering says that counters do not count when
blocked by the event filter. To enforce that, the event filter must be
rechecked on every load since it might have changed since the last
time the guest wrote a value. If the event is filtered, exclude
counting at all exception levels before writing the hardware.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/kvm/pmu-direct.c | 44 +++++++++++++++++++++++++++++++++++++
1 file changed, 44 insertions(+)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:47
Save and restore newly untrapped registers that can be directly
accessed by the guest when the PMU is partitioned.
* PMEVCNTRn_EL0
* PMCCNTR_EL0
* PMICNTR_EL0
* PMUSERENR_EL0
* PMSELR_EL0
* PMCR_EL0
* PMCNTEN_EL0
* PMINTEN_EL1
If we know we are not using FGT (that is, trapping everything), then
return immediately. Either the PMU is not partitioned, or it is but
all register writes are being written through the VCPU fields to
hardware, so all values are fresh.
Since we are taking over context switching, avoid the writes to
PMSELR_EL0 and PMUSERENR_EL0 that would normally occur in
__{,de}activate_traps_common()
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/kvm_pmu.h | 4 +
arch/arm64/kvm/arm.c | 2 +
arch/arm64/kvm/hyp/include/hyp/switch.h | 4 +-
arch/arm64/kvm/pmu-direct.c | 112 ++++++++++++++++++++++++
4 files changed, 120 insertions(+), 2 deletions(-)
@@ -219,3 +220,114 @@ u8 kvm_pmu_hpmn(struct kvm_vcpu *vcpu)returnnr_host_cnt_max;}++/**+*kvm_pmu_load()-LoaduntrappedPMUregisters+*@vcpu:Pointertostructkvm_vcpu+*+*LoadalluntrappedPMUregistersfromtheVCPUintothePCPU.Mask+*toonlybitsbelongingtoguest-reservedcountersandleave+*host-reservedcountersaloneinbitmaskregisters.+*/+voidkvm_pmu_load(structkvm_vcpu*vcpu)+{+structarm_pmu*pmu;+u64mask;+u8i;+u64val;++/*+*Ifwearen'tusingFGTthenwearetrappingeverything+*anyway,sononeedtobotherwiththeswap.+*/+if(!kvm_vcpu_pmu_use_fgt(vcpu))+return;++pmu=vcpu->kvm->arch.arm_pmu;++for(i=0;i<pmu->hpmn_max;i++){+val=__vcpu_sys_reg(vcpu,PMEVCNTR0_EL0+i);+write_pmevcntrn(i,val);+}++val=__vcpu_sys_reg(vcpu,PMCCNTR_EL0);+write_pmccntr(val);++val=__vcpu_sys_reg(vcpu,PMUSERENR_EL0);+write_pmuserenr(val);++val=__vcpu_sys_reg(vcpu,PMSELR_EL0);+write_pmselr(val);++/* Save only the stateful writable bits. */+val=__vcpu_sys_reg(vcpu,PMCR_EL0);+mask=ARMV8_PMU_PMCR_MASK&+~(ARMV8_PMU_PMCR_P|ARMV8_PMU_PMCR_C);+write_pmcr(val&mask);++/*+*Whenhandlingthese:+*1.Applyonlythebitsforguestcounters(indicatedbymask)+*2.Usethedifferentregistersforsetandclear+*/+mask=kvm_pmu_guest_counter_mask(pmu);++val=__vcpu_sys_reg(vcpu,PMCNTENSET_EL0);+write_pmcntenset(val&mask);+write_pmcntenclr(~val&mask);++val=__vcpu_sys_reg(vcpu,PMINTENSET_EL1);+write_pmintenset(val&mask);+write_pmintenclr(~val&mask);+}++/**+*kvm_pmu_put()-PutuntrappedPMUregisters+*@vcpu:Pointertostructkvm_vcpu+*+*PutalluntrappedPMUregistersfromtheVCPUintothePCPU.Mask+*toonlybitsbelongingtoguest-reservedcountersandleave+*host-reservedcountersaloneinbitmaskregisters.+*/+voidkvm_pmu_put(structkvm_vcpu*vcpu)+{+structarm_pmu*pmu;+u64mask;+u8i;+u64val;++/*+*Ifwearen'tusingFGTthenwearetrappingeverything+*anyway,sononeedtobotherwiththeswap.+*/+if(!kvm_vcpu_pmu_use_fgt(vcpu))+return;++pmu=vcpu->kvm->arch.arm_pmu;++for(i=0;i<pmu->hpmn_max;i++){+val=read_pmevcntrn(i);+__vcpu_assign_sys_reg(vcpu,PMEVCNTR0_EL0+i,val);+}++val=read_pmccntr();+__vcpu_assign_sys_reg(vcpu,PMCCNTR_EL0,val);++val=read_pmuserenr();+__vcpu_assign_sys_reg(vcpu,PMUSERENR_EL0,val);++val=read_pmselr();+__vcpu_assign_sys_reg(vcpu,PMSELR_EL0,val);++val=read_pmcr();+__vcpu_assign_sys_reg(vcpu,PMCR_EL0,val);++/* Mask these to only save the guest relevant bits. */+mask=kvm_pmu_guest_counter_mask(pmu);++val=read_pmcntenset();+__vcpu_assign_sys_reg(vcpu,PMCNTENSET_EL0,val&mask);++val=read_pmintenset();+__vcpu_assign_sys_reg(vcpu,PMINTENSET_EL1,val&mask);+}
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:49
Since many guests will never touch the PMU, they need not pay the cost
of context swapping those registers.
Use an enum to implement a simple state machine for PMU register
access. The PMU either accesses registers virtually or physically.
Virtual access implies all PMU registers are trapped coarsely by
MDCR_EL2.TPM and therefore do not need to be context swapped. Physical
access implies some registers are untrapped through FGT and do need to
be context swapped. All vCPUs do virtual access by default and
transition to physical if the PMU is partitioned and the guest
actually tries a PMU access.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/kvm_host.h | 1 +
arch/arm64/include/asm/kvm_pmu.h | 4 ++++
arch/arm64/include/asm/kvm_types.h | 7 ++++++-
arch/arm64/kvm/debug.c | 2 +-
arch/arm64/kvm/hyp/include/hyp/switch.h | 2 ++
arch/arm64/kvm/pmu-direct.c | 21 +++++++++++++++++++++
arch/arm64/kvm/pmu.c | 7 +++++++
arch/arm64/kvm/sys_regs.c | 4 ++++
8 files changed, 46 insertions(+), 2 deletions(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:49
Because ARM hardware is not yet capable of direct interrupt injection
into guests, guest counters will still trigger interrupts that need to
be handled by the host PMU interrupt handler. Clear the overflow flags
in hardware to handle the interrupt as normal, but record which guest
overflow flags were set in the virtual overflow register for later
injecting the interrupt into the guest.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm/include/asm/arm_pmuv3.h | 6 ++++++
arch/arm64/include/asm/kvm_pmu.h | 2 ++
arch/arm64/kvm/pmu-direct.c | 17 +++++++++++++++++
drivers/perf/arm_pmuv3.c | 9 +++++++++
4 files changed, 34 insertions(+)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:52
Add KVM_CAP_ARM_PARTITION_PMU to enable the partitioned PMU for a
given VM. This capability is allowed where PMUv3 and VHE are supported
and the host driver was configured with
arm_pmuv3.reserved_host_counters.
Ordering is enforced such that this must be called before the vCPUs
are created to avoid the possibility the guest is already using one
PMU configuration that gets switched out from under it.
The enabled capability is tracked by the new flag
KVM_ARCH_FLAG_PARTITIONED_PMU_ENABLED.
Signed-off-by: Colton Lewis <redacted>
---
Documentation/virt/kvm/api.rst | 24 +++++++++++++++++++++
arch/arm64/include/asm/kvm_host.h | 2 ++
arch/arm64/include/asm/kvm_pmu.h | 9 ++++++++
arch/arm64/kvm/arm.c | 15 +++++++++++++
arch/arm64/kvm/pmu-direct.c | 35 ++++++++++++++++++++++++++++---
include/uapi/linux/kvm.h | 1 +
6 files changed, 83 insertions(+), 3 deletions(-)
@@ -8703,6 +8703,30 @@ This capability indicate to the userspace whether a PFNMAP memory region can be safely mapped as cacheable. This relies on the presence of force write back (FWB) feature support on the hardware.++7.245 KVM_CAP_ARM_PARTITION_PMU+-------------------------------------++:Architectures: arm64+:Target: VM+:Parameters: arg[0] is a boolean that enables or disables the capability+:Returns: 0 on success+ -EPERM if host doesn't support+ -EBUSY if vPMU was already created++This API controls the PMU implementation used for VMs. The capability+is only available if the host PMUv3 driver was configured for+partitioning via the module parameter+``arm-pmuv3.reserved_guest_counters=[0-$NR_COUNTERS]``. When enabled,+VMs are configured to have direct hardware access to the most+frequently used registers for the counters configured by the+aforementioned module parameters, bypassing the KVM traps in the+standard emulated PMU implementation and reducing overhead of any+guest software that uses PMU capabilities such as ``perf``.++If the host driver was configured for partitioning but the partitioned+PMU is disabled through this interface, the VM will use the legacy PMU+8. Other capabilities. ======================
@@ -132,6 +134,16 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,}mutex_unlock(&kvm->lock);break;+caseKVM_CAP_ARM_PARTITION_PMU:+if(kvm->created_vcpus){+r=-EBUSY;+}elseif(!kvm_pmu_partition_ready()){+r=-EPERM;+}else{+r=0;+kvm_pmu_partition_enable(kvm,cap->args[0]);+}+break;default:break;}
@@ -388,6 +400,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)caseKVM_CAP_ARM_PMU_V3:r=kvm_supports_guest_pmuv3();break;+caseKVM_CAP_ARM_PARTITION_PMU:+r=kvm_pmu_partition_ready();+break;caseKVM_CAP_ARM_INJECT_SERROR_ESR:r=cpus_have_final_cap(ARM64_HAS_RAS_EXTN);break;
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:54
Some selftests have a dependency on find_bit and weren't compiling
separately without it, so I've added it to the KVM library here using
the same method as files like rbtree.c.
Signed-off-by: Colton Lewis <redacted>
---
tools/testing/selftests/kvm/Makefile.kvm | 1 +
tools/testing/selftests/kvm/lib/find_bit.c | 1 +
2 files changed, 2 insertions(+)
create mode 100644 tools/testing/selftests/kvm/lib/find_bit.c
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:55
When we re-enter the VM after handling a PMU interrupt, calculate
whether it was any of the guest counters that overflowed and inject an
interrupt into the guest if so.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/kvm_pmu.h | 2 ++
arch/arm64/kvm/pmu-direct.c | 20 ++++++++++++++++++++
arch/arm64/kvm/pmu-emul.c | 4 ++--
arch/arm64/kvm/pmu.c | 6 +++++-
4 files changed, 29 insertions(+), 3 deletions(-)
From: Colton Lewis <hidden> Date: 2025-12-09 20:52:55
Rerun all tests for a partitioned PMU in vpmu_counter_access.
Create an enum specifying whether we are testing the emulated or
partitioned PMU and all the test functions are modified to take the
implementation as an argument and make the difference in setup
appropriately.
Signed-off-by: Colton Lewis <redacted>
---
tools/include/uapi/linux/kvm.h | 1 +
.../selftests/kvm/arm64/vpmu_counter_access.c | 77 ++++++++++++++-----
2 files changed, 57 insertions(+), 21 deletions(-)
@@ -25,9 +25,20 @@/* The cycle counter bit position that's common among the PMU registers */#define ARMV8_PMU_CYCLE_IDX 31+enumpmu_impl{+EMULATED,+PARTITIONED+};++constchar*pmu_impl_str[]={+"Emulated",+"Partitioned"+};+structvpmu_vm{structkvm_vm*vm;structkvm_vcpu*vcpu;+boolpmu_partitioned;};staticstructvpmu_vmvpmu_vm;
@@ -399,7 +410,7 @@ static void guest_code(uint64_t expected_pmcr_n)}/* Create a VM that has one vCPU with PMUv3 configured. */-staticvoidcreate_vpmu_vm(void*guest_code)+staticvoidcreate_vpmu_vm(void*guest_code,enumpmu_implimpl){structkvm_vcpu_initinit;uint8_tpmuver,ec;
@@ -409,6 +420,11 @@ static void create_vpmu_vm(void *guest_code).attr=KVM_ARM_VCPU_PMU_V3_IRQ,.addr=(uint64_t)&irq,};+boolpartition=(impl==PARTITIONED);+structkvm_enable_cappartition_cap={+.cap=KVM_CAP_ARM_PARTITION_PMU,+.args[0]=partition,+};/* The test creates the vpmu_vm multiple times. Ensure a clean state */memset(&vpmu_vm,0,sizeof(vpmu_vm));
@@ -489,7 +512,7 @@ static void test_create_vpmu_vm_with_nr_counters(unsigned int nr_counters, bool*CreateaguestwithonevCPU,setthePMCR_EL0.NforthevCPUto@pmcr_n,*andrunthetest.*/-staticvoidrun_access_test(uint64_tpmcr_n)+staticvoidrun_access_test(uint64_tpmcr_n,enumpmu_implimpl){uint64_tsp;structkvm_vcpu*vcpu;
@@ -497,7 +520,7 @@ static void run_access_test(uint64_t pmcr_n)pr_debug("Test with pmcr_n %lu\n",pmcr_n);-test_create_vpmu_vm_with_nr_counters(pmcr_n,false);+test_create_vpmu_vm_with_nr_counters(pmcr_n,impl,false);vcpu=vpmu_vm.vcpu;/* Save the initial sp to restore them later to run the guest again */
@@ -588,11 +611,11 @@ static void run_pmregs_validity_test(uint64_t pmcr_n)*thevCPUto@pmcr_n,whichislargerthanthehostvalue.*Theattemptshouldfailas@pmcr_nistoobigtosetforthevCPU.*/-staticvoidrun_error_test(uint64_tpmcr_n)+staticvoidrun_error_test(uint64_tpmcr_n,enumpmu_implimpl){-pr_debug("Error test with pmcr_n %lu (larger than the host)\n",pmcr_n);+pr_debug("Error test with pmcr_n %lu (larger than the host allows)\n",pmcr_n);-test_create_vpmu_vm_with_nr_counters(pmcr_n,true);+test_create_vpmu_vm_with_nr_counters(pmcr_n,impl,true);destroy_vpmu_vm();}
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 21:08:15
Hi Colton,
On Tue, Dec 09, 2025 at 08:51:07PM +0000, Colton Lewis wrote:
quoted hunk
In order to gain the best performance benefit from partitioning the
PMU, utilize fine grain traps (FEAT_FGT and FEAT_FGT2) to avoid
trapping common PMU register accesses by the guest to remove that
overhead.
Untrapped:
* PMCR_EL0
* PMUSERENR_EL0
* PMSELR_EL0
* PMCCNTR_EL0
* PMCNTEN_EL0
* PMINTEN_EL1
* PMEVCNTRn_EL0
These are safe to untrap because writing MDCR_EL2.HPMN as this series
will do limits the effect of writes to any of these registers to the
partition of counters 0..HPMN-1. Reads from these registers will not
leak information from between guests as all these registers are
context swapped by a later patch in this series. Reads from these
registers also do not leak any information about the host's hardware
beyond what is promised by PMUv3.
Trapped:
* PMOVS_EL0
* PMEVTYPERn_EL0
* PMCCFILTR_EL0
* PMICNTR_EL0
* PMICFILTR_EL0
* PMCEIDn_EL0
* PMMIR_EL1
PMOVS remains trapped so KVM can track overflow IRQs that will need to
be injected into the guest.
PMICNTR and PMIFILTR remain trapped because KVM is not handling them
yet.
PMEVTYPERn remains trapped so KVM can limit which events guests can
count, such as disallowing counting at EL2. PMCCFILTR and PMCIFILTR
are special cases of the same.
PMCEIDn and PMMIR remain trapped because they can leak information
specific to the host hardware implementation.
NOTE: This patch temporarily forces kvm_vcpu_pmu_is_partitioned() to
be false to prevent partial feature activation for easier debugging.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/kvm_pmu.h | 33 ++++++++++++++++++++++
arch/arm64/kvm/config.c | 34 ++++++++++++++++++++--
arch/arm64/kvm/pmu-direct.c | 48 ++++++++++++++++++++++++++++++++
3 files changed, 112 insertions(+), 3 deletions(-)
Couple of suggestions. I'd rather see this conditioned on
kvm_vcpu_pmu_is_partitioned() and get rid of the FGT predicate. After
all, kvm_vcpu_load_fgt() already checks for the presence of FEAT_FGT
first.
Additionally, I'd prefer that the trap configuration is inline instead
of done in a helper in some other file. Centralizing the FGT
configuration here was very much intentional.
The other reason for doing this is kvm_pmu_fgt_bits() assumes a
'positive' trap polarity, even though there are several cases where FGTs
have a 'negative' priority (i.e. 0 => trap).
Thanks,
Oliver
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 21:14:04
On Tue, Dec 09, 2025 at 08:51:09PM +0000, Colton Lewis wrote:
quoted hunk
Because PMXEVTYPER is trapped and PMSELR is not, it is not appropriate
to use the virtual PMSELR register when it could be outdated and lead
to an invalid write. Use the physical register when partitioned.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/arm_pmuv3.h | 7 ++++++-
arch/arm64/kvm/sys_regs.c | 9 +++++++--
2 files changed, 13 insertions(+), 3 deletions(-)
This isn't preemption safe. Nor should the "if (partitioned) do X else do Y" get
open-coded throughout the shop.
I would rather this be handled with a prepatory patch that provides
generic PMU register accessors to the rest of KVM (e.g.
vcpu_read_pmu_reg() / vcpu_write_pmu_reg()). Internally those helpers
can locate the vCPU's PMU registers (emulated, partitioned in-memory,
partitioned in-CPU).
Thanks,
Oliver
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 21:19:10
On Tue, Dec 09, 2025 at 08:51:10PM +0000, Colton Lewis wrote:
quoted hunk
Because PMOVS remains trapped, it needs to be written through when
partitioned to affect PMU hardware when expected.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/arm_pmuv3.h | 10 ++++++++++
arch/arm64/kvm/sys_regs.c | 17 ++++++++++++++++-
2 files changed, 26 insertions(+), 1 deletion(-)
There's only ever a single canonical guest view of a register. Either it has
been loaded onto the CPU or it is in memory, writing the value to two
different locations is odd. What guarantees the guest context is on the
CPU currently? And what about preemption?
Thanks,
Oliver
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 21:33:17
On Tue, Dec 09, 2025 at 08:51:12PM +0000, Colton Lewis wrote:
quoted hunk
Setup MDCR_EL2 to handle a partitioned PMU. That means calculate an
appropriate value for HPMN instead of the default maximum setting the
host allows (which implies no partition) so hardware enforces that a
guest will only see the counters in the guest partition.
Setting HPMN to a non default value means the global enable bit for
the host counters is now MDCR_EL2.HPME instead of the usual
PMCR_EL0.E. Enable the HPME bit to allow the host to count guest
events. Since HPME only has an effect when HPMN is set which we only
do for the guest, it is correct to enable it unconditionally here.
Unset the TPM and TPMCR bits, which trap all PMU accesses, if
FGT (fine grain trapping) is being used.
If available, set the filtering bits HPMD and HCCD to be extra sure
nothing in the guest counts at EL2.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/include/asm/kvm_pmu.h | 11 ++++++
arch/arm64/kvm/debug.c | 29 ++++++++++++--
arch/arm64/kvm/pmu-direct.c | 65 ++++++++++++++++++++++++++++++++
3 files changed, 102 insertions(+), 3 deletions(-)
@@ -36,20 +36,43 @@ static int cpu_has_spe(u64 dfr0)*/staticvoidkvm_arm_setup_mdcr_el2(structkvm_vcpu*vcpu){+inthpmn=kvm_pmu_hpmn(vcpu);+preempt_disable();/**ThisalsoclearsMDCR_EL2_E2PB_MASKandMDCR_EL2_E2TB_MASK*todisableguestaccesstotheprofilingandtracebuffers*/-vcpu->arch.mdcr_el2=FIELD_PREP(MDCR_EL2_HPMN,-*host_data_ptr(nr_event_counters));++vcpu->arch.mdcr_el2=FIELD_PREP(MDCR_EL2_HPMN,hpmn);vcpu->arch.mdcr_el2|=(MDCR_EL2_TPM|MDCR_EL2_TPMS|MDCR_EL2_TTRF|MDCR_EL2_TPMCR|MDCR_EL2_TDRA|-MDCR_EL2_TDOSA);+MDCR_EL2_TDOSA|+MDCR_EL2_HPME);++if(kvm_vcpu_pmu_is_partitioned(vcpu)){+/*+*Filteringtheseshouldberedundantbecausewetrap+*alltheTYPERandFILTRregistersanywayandensure+*theyfilterEL2,butsetthebitsiftheyarehere.+*/+if(is_pmuv3p1(read_pmuver()))+vcpu->arch.mdcr_el2|=MDCR_EL2_HPMD;+if(is_pmuv3p5(read_pmuver()))+vcpu->arch.mdcr_el2|=MDCR_EL2_HCCD;++/*+*Takeoutthecoarsegraintrapsifweareusing+*finegraintraps.+*/+if(kvm_vcpu_pmu_use_fgt(vcpu))+vcpu->arch.mdcr_el2&=~(MDCR_EL2_TPM|MDCR_EL2_TPMCR);++}/* Is the VM being debugged by userspace? */if(vcpu->guest_debug)
+/**+ * kvm_pmu_guest_num_counters() - Number of counters to show to guest+ * @vcpu: Pointer to struct kvm_vcpu+ *+ * Calculate the number of counters to show to the guest via+ * PMCR_EL0.N, making sure to respect the maximum the host allows,+ * which is hpmn_max if partitioned and host_max otherwise.+ *+ * Return: Valid value for PMCR_EL0.N+ */+u8 kvm_pmu_guest_num_counters(struct kvm_vcpu *vcpu)+{+ u8 nr_cnt = vcpu->kvm->arch.nr_pmu_counters;+ int hpmn_max = armv8pmu_hpmn_max;+ u8 host_max = *host_data_ptr(nr_event_counters);++ if (vcpu->kvm->arch.arm_pmu)+ hpmn_max = vcpu->kvm->arch.arm_pmu->hpmn_max;++ if (kvm_vcpu_pmu_is_partitioned(vcpu)) {+ if (nr_cnt <= hpmn_max && nr_cnt <= host_max)+ return nr_cnt;+ if (hpmn_max <= host_max)+ return hpmn_max;+ }++ if (nr_cnt <= host_max)+ return nr_cnt;++ return host_max;+}++/**+ * kvm_pmu_hpmn() - Calculate HPMN field value+ * @vcpu: Pointer to struct kvm_vcpu+ *+ * Calculate the appropriate value to set for MDCR_EL2.HPMN. If+ * partitioned, this is the number of counters set for the guest if+ * supported, falling back to hpmn_max if needed. If we are not+ * partitioned or can't set the implied HPMN value, fall back to the+ * host value.+ *+ * Return: A valid HPMN value+ */+u8 kvm_pmu_hpmn(struct kvm_vcpu *vcpu)+{+ u8 nr_guest_cnt = kvm_pmu_guest_num_counters(vcpu);+ int nr_guest_cnt_max = armv8pmu_hpmn_max;+ u8 nr_host_cnt_max = *host_data_ptr(nr_event_counters);++ if (vcpu->kvm->arch.arm_pmu)+ nr_guest_cnt_max = vcpu->kvm->arch.arm_pmu->hpmn_max;++ if (kvm_vcpu_pmu_is_partitioned(vcpu)) {+ if (cpus_have_final_cap(ARM64_HAS_HPMN0))+ return nr_guest_cnt;+ else if (nr_guest_cnt > 0)+ return nr_guest_cnt;+ else if (nr_guest_cnt_max > 0)+ return nr_guest_cnt_max;+ }++ return nr_host_cnt_max;+}
</snip>
I find all of this rather confusing. It seems like you're dealing with
sanitizing kvm->arch.nr_pmu_counters vs. the underlying implementation.
I'm not sure why you need to do that, I would expect that we reject
unsupported values at the time of the ioctl.
The only thing you do need to handle is if the vCPU has migrated to an
"unsupported" CPU, for which we already have supporting helpers. I'm too
lazy to fetch the Arm ARM and cite architecture but I'm pretty sure an
out-of-range HPMN has UNPREDICTABLE behavior.
I think you just need to move the vcpu_set_unsupported_cpu() call earlier in
kvm_arch_vcpu_load().
Taking all that into consideration:
u8 kvm_mdcr_hpmn(struct kvm_vcpu *vcpu)
{
u8 nr_counters = *host_data_ptr(nr_event_counters);
if (!kvm_vcpu_pmu_is_partitioned(vcpu) || vcpu_on_unsupported_cpu(vcpu))
return nr_counters;
return vcpu->kvm->arch.nr_pmu_counters;
}
Thanks,
Oliver
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 21:37:53
On Tue, Dec 09, 2025 at 08:51:13PM +0000, Colton Lewis wrote:
quoted hunk
Make sure reads and writes to PMCR_EL0 conform to additional
constraints imposed when the PMU is partitioned.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/kvm/pmu.c | 2 +-
arch/arm64/kvm/sys_regs.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
This is the legacy UAPI for setting the number of PMU counters by
writing to PMCR_EL0.N.
The 'partitioned' implementation should take a dependency on the
SET_NR_COUNTERS attribute and reject attempts to change the value of
PMCR_EL0.N. Just like nested.
Thanks,
Oliver
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 21:56:00
On Tue, Dec 09, 2025 at 08:51:14PM +0000, Colton Lewis wrote:
quoted hunk
+/**+ * kvm_pmu_load() - Load untrapped PMU registers+ * @vcpu: Pointer to struct kvm_vcpu+ *+ * Load all untrapped PMU registers from the VCPU into the PCPU. Mask+ * to only bits belonging to guest-reserved counters and leave+ * host-reserved counters alone in bitmask registers.+ */+void kvm_pmu_load(struct kvm_vcpu *vcpu)+{+ struct arm_pmu *pmu;+ u64 mask;+ u8 i;+ u64 val;+
Assert that preemption is disabled.
quoted hunk
+ /*+ * If we aren't using FGT then we are trapping everything+ * anyway, so no need to bother with the swap.+ */+ if (!kvm_vcpu_pmu_use_fgt(vcpu))+ return;
Uhh... Then how do events count in this case?
The absence of FEAT_FGT shouldn't affect the residence of the guest PMU
context. We just need to handle the extra traps, ideally by reading the
PMU registers directly as a fast path exit handler.
quoted hunk
+ pmu = vcpu->kvm->arch.arm_pmu;++ for (i = 0; i < pmu->hpmn_max; i++) {+ val = __vcpu_sys_reg(vcpu, PMEVCNTR0_EL0 + i);+ write_pmevcntrn(i, val);+ }++ val = __vcpu_sys_reg(vcpu, PMCCNTR_EL0);+ write_pmccntr(val);++ val = __vcpu_sys_reg(vcpu, PMUSERENR_EL0);+ write_pmuserenr(val);
What about the host's value for PMUSERENR?
+ val = __vcpu_sys_reg(vcpu, PMSELR_EL0);
+ write_pmselr(val);
PMSELR_EL0 needs to be switched late, e.g. at sysreg_restore_guest_state_vhe().
Even though the host doesn't currently use the selector-based accessor,
I'd prefer we not load things that'd affect the host context until we're
about to enter the guest.
quoted hunk
+ /* Save only the stateful writable bits. */+ val = __vcpu_sys_reg(vcpu, PMCR_EL0);+ mask = ARMV8_PMU_PMCR_MASK &+ ~(ARMV8_PMU_PMCR_P | ARMV8_PMU_PMCR_C);+ write_pmcr(val & mask);++ /*+ * When handling these:+ * 1. Apply only the bits for guest counters (indicated by mask)+ * 2. Use the different registers for set and clear+ */+ mask = kvm_pmu_guest_counter_mask(pmu);++ val = __vcpu_sys_reg(vcpu, PMCNTENSET_EL0);+ write_pmcntenset(val & mask);+ write_pmcntenclr(~val & mask);++ val = __vcpu_sys_reg(vcpu, PMINTENSET_EL1);+ write_pmintenset(val & mask);+ write_pmintenclr(~val & mask);
Is this safe? What happens if we put the PMU into an overflow condition?
quoted hunk
+}++/**+ * kvm_pmu_put() - Put untrapped PMU registers+ * @vcpu: Pointer to struct kvm_vcpu+ *+ * Put all untrapped PMU registers from the VCPU into the PCPU. Mask+ * to only bits belonging to guest-reserved counters and leave+ * host-reserved counters alone in bitmask registers.+ */+void kvm_pmu_put(struct kvm_vcpu *vcpu)+{+ struct arm_pmu *pmu;+ u64 mask;+ u8 i;+ u64 val;++ /*+ * If we aren't using FGT then we are trapping everything+ * anyway, so no need to bother with the swap.+ */+ if (!kvm_vcpu_pmu_use_fgt(vcpu))+ return;++ pmu = vcpu->kvm->arch.arm_pmu;++ for (i = 0; i < pmu->hpmn_max; i++) {+ val = read_pmevcntrn(i);+ __vcpu_assign_sys_reg(vcpu, PMEVCNTR0_EL0 + i, val);+ }++ val = read_pmccntr();+ __vcpu_assign_sys_reg(vcpu, PMCCNTR_EL0, val);++ val = read_pmuserenr();+ __vcpu_assign_sys_reg(vcpu, PMUSERENR_EL0, val);++ val = read_pmselr();+ __vcpu_assign_sys_reg(vcpu, PMSELR_EL0, val);++ val = read_pmcr();+ __vcpu_assign_sys_reg(vcpu, PMCR_EL0, val);++ /* Mask these to only save the guest relevant bits. */+ mask = kvm_pmu_guest_counter_mask(pmu);++ val = read_pmcntenset();+ __vcpu_assign_sys_reg(vcpu, PMCNTENSET_EL0, val & mask);++ val = read_pmintenset();+ __vcpu_assign_sys_reg(vcpu, PMINTENSET_EL1, val & mask);
What if the PMU is in an overflow state at this point?
Thanks,
Oliver
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 22:00:49
On Tue, Dec 09, 2025 at 08:51:15PM +0000, Colton Lewis wrote:
quoted hunk
The KVM API for event filtering says that counters do not count when
blocked by the event filter. To enforce that, the event filter must be
rechecked on every load since it might have changed since the last
time the guest wrote a value. If the event is filtered, exclude
counting at all exception levels before writing the hardware.
Signed-off-by: Colton Lewis <redacted>
---
arch/arm64/kvm/pmu-direct.c | 44 +++++++++++++++++++++++++++++++++++++
1 file changed, 44 insertions(+)
This doesn't work for nested. I agree that the hardware value of
PMEVTYPERn_EL0 needs to be under KVM control, but depending on whether
or not we're in a hyp context the meaning of the EL1 filtering bit
changes. Have a look at kvm_pmu_create_perf_event().
Thanks,
Oliver
This is confusing. Even when the guest is accessing registers directly
on the CPU I'd still call that "hardware assisted virtualization" and
not "physical".
It isn't immediately obvious how this guards against preemption.
Also, the general approach for these context-loading situations is to do
a full load/put on the vCPU rather than a directed load.
Aren't there a ton of other registers the guest may access before
these two? Having generic PMU register accessors would allow you to
manage residence of PMU registers from a single spot.
Thanks,
Oliver
The state ownership of this whole interaction is very odd. I would much
rather that KVM have full ownership of the range of counters while the
guest is loaded. By that I mean the PMUv3 driver only clears overflows
on PMCs that it owns and KVM will do the same on the back of the IRQ.
Similarly, KVM should be leaving the "guest" range of counters in a
non-overflow condition at vcpu_put().
Thanks,
Oliver
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 22:52:03
In no situation should KVM be injecting a "recorded" IRQ. The overflow
condition of the PMU is well defined in the architecture and we should
implement *exactly* that.
On Tue, Dec 09, 2025 at 08:51:18PM +0000, Colton Lewis wrote:
quoted hunk
+/**+ * kvm_pmu_part_overflow_status() - Determine if any guest counters have overflowed+ * @vcpu: Ponter to struct kvm_vcpu+ *+ * Determine if any guest counters have overflowed and therefore an+ * IRQ needs to be injected into the guest.+ *+ * Return: True if there was an overflow, false otherwise+ */+bool kvm_pmu_part_overflow_status(struct kvm_vcpu *vcpu)+{+ struct arm_pmu *pmu = vcpu->kvm->arch.arm_pmu;+ u64 mask = kvm_pmu_guest_counter_mask(pmu);+ u64 pmovs = __vcpu_sys_reg(vcpu, PMOVSSET_EL0);+ u64 pmint = read_pmintenset();+ u64 pmcr = read_pmcr();
How do we know that the vPMU has been loaded on the CPU at this point?
I'd rather reuse kvm_pmu_overflow_status(), relying on the accessors to
abstract away the implementation / location of the guest PMU context.
Thanks,
Oliver
Why can't this be a vCPU attribute similar to the other vPMU controls?
Making the UAPI consistent will make it easier for userspace to reason
about it.
Better yet, we could make the UAPI such that userspace selects a PMU
implementation and the partitioned-ness of the PMU at the same time.
quoted hunk
@@ -132,6 +134,16 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, } mutex_unlock(&kvm->lock); break;+ case KVM_CAP_ARM_PARTITION_PMU:+ if (kvm->created_vcpus) {+ r = -EBUSY;+ } else if (!kvm_pmu_partition_ready()) {+ r = -EPERM;+ } else {+ r = 0;+ kvm_pmu_partition_enable(kvm, cap->args[0]);+ }+ break; default: break; }
@@ -388,6 +400,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_ARM_PMU_V3: r = kvm_supports_guest_pmuv3(); break;+ case KVM_CAP_ARM_PARTITION_PMU:+ r = kvm_pmu_partition_ready();
"ready" is very confusing in this context, as KVM will never be ready to
support the feature on a system w/o the prerequisites.
Thanks,
Oliver
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-09 23:01:02
On Tue, Dec 09, 2025 at 08:50:57PM +0000, Colton Lewis wrote:
This series creates a new PMU scheme on ARM, a partitioned PMU that
allows reserving a subset of counters for more direct guest access,
significantly reducing overhead. More details, including performance
benchmarks, can be read in the v1 cover letter linked below.
An overview of what this series accomplishes was presented at KVM
Forum 2025. Slides [1] and video [2] are linked below.
The long duration between v4 and v5 is due to time spent on this
project being monopolized preparing this feature for internal
production. As a result, there are too many improvements to fully list
here, but I will cover the notable ones.
Thanks for reposting. I think there's still quite a bit of ground to
cover on the KVM side of this, but I would definitely appreciate it if
someone with more context on the perf side of things could chime in.
Will, IIRC you had some thoughts around counter allocation, right?
Best,
Oliver
From: Suzuki K Poulose <suzuki.poulose@arm.com> Date: 2025-12-10 10:54:22
On 09/12/2025 20:50, Colton Lewis wrote:
Add a capability for FEAT_HPMN0, whether MDCR_EL2.HPMN can specify 0
counters reserved for the guest.
This required changing HPMN0 to an UnsignedEnum in tools/sysreg
because otherwise not all the appropriate macros are generated to add
it to arm64_cpu_capabilities_arm64_features.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Colton Lewis <redacted>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
arch/arm64/kvm/sys_regs.c:1183:34: error: implicit declaration of function 'kvm_pmu_event_mask'; did you mean 'kvm_pmu_evtyper_mask'? [-Wimplicit-function-declaration]
I wouldn't bother with adding accessors to the PMUv3 driver header, this
gets a bit confusing when the 32-bit counterpart lacks matching definitions.
Additionally, the part of KVM you're modifying is very much an
"internal" part meant to run in a not-quite-kernel context.
Considering this, I'd prefer KVM directly accessed the PMU registers to
avoid the possibility of taking some instrumented codepath in the
future.
+ if (!kvm_vcpu_pmu_is_partitioned(vcpu)
+ || pmu_access_el0_disabled(vcpu))
Always put operators on the preceding line for line continuations.
Also, don't call pmu_access_el0_disabled() from here. It can potentially
do a full emulated exception entry even though the vCPU is in an
extremely inconsistent state (i.e. haven't returned to kernel context
yet). Isn't the current value for PMUSERENR_EL0 on the CPU instead of in
the vCPU's saved context anyway?
The fast-path accessors really need to be *just* accessors, reading
state from the CPU in the unfortunate situation that a TPM trap has been
forced upon us.
quoted hunk
+ case SYS_PMXEVCNTR_EL0:+ idx = FIELD_GET(PMSELR_EL0_SEL, read_pmselr());++ if (pmu_access_event_counter_el0_disabled(vcpu))+ return false;++ if (!pmu_counter_idx_valid(vcpu, idx))+ return false;++ ret = handle_pmu_reg(vcpu, &p, PMEVCNTR0_EL0 + idx, rt, val,+ &read_pmxevcntr, &write_pmxevcntr);+ break;
It is a bit odd to handle the muxing for finding the in-memory value yet
using the selector-based register for hardware.
quoted hunk
+ case SYS_PMEVCNTRn_EL0(0) ... SYS_PMEVCNTRn_EL0(30):+ idx = ((p.CRm & 3) << 3) | (p.Op2 & 7);++ if (pmu_access_event_counter_el0_disabled(vcpu))+ return false;++ if (!pmu_counter_idx_valid(vcpu, idx))+ return false;++ if (p.is_write) {+ write_pmevcntrn(idx, val);+ __vcpu_assign_sys_reg(vcpu, PMEVCNTR0_EL0 + idx, val);+ } else {+ vcpu_set_reg(vcpu, rt, read_pmevcntrn(idx));+ }++ ret = true;+ break;
Can't both of these cases share a helper once you've worked out the
index? Same for PMEVTYPERn_EL0.
quoted hunk
+ default:+ ret = false;+ }++ if (ret)+ __kvm_skip_instr(vcpu);++ return ret;+}+ static inline bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code) { if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) &&
@@ -785,6 +983,9 @@ static inline bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code) if (kvm_handle_cntxct(vcpu)) return true;+ if (kvm_hyp_handle_pmu_regs(vcpu))+ return true;+
Since the whole partitioned PMU feature is constrained to VHE-only you
should call this from kvm_hyp_handle_sysreg_vhe().
From: Oliver Upton <oupton@kernel.org> Date: 2025-12-17 00:57:21
Re-reading this patch...
On Tue, Dec 09, 2025 at 08:51:15PM +0000, Colton Lewis wrote:
The KVM API for event filtering says that counters do not count when
blocked by the event filter. To enforce that, the event filter must be
rechecked on every load since it might have changed since the last
time the guest wrote a value.
Just directly state that this is guarding against userspace programming
an unsupported event ID.
quoted hunk
+static void kvm_pmu_apply_event_filter(struct kvm_vcpu *vcpu)+{+ struct arm_pmu *pmu = vcpu->kvm->arch.arm_pmu;+ u64 evtyper_set = ARMV8_PMU_EXCLUDE_EL0 |+ ARMV8_PMU_EXCLUDE_EL1;+ u64 evtyper_clr = ARMV8_PMU_INCLUDE_EL2;+ u8 i;+ u64 val;+ u64 evsel;++ if (!pmu)+ return;++ for (i = 0; i < pmu->hpmn_max; i++) {
Iterate the bitmask of counters and you'll handle the cycle counter 'for
free'.
<snip>
quoted hunk
+ val = __vcpu_sys_reg(vcpu, PMEVTYPER0_EL0 + i);+ evsel = val & kvm_pmu_event_mask(vcpu->kvm);++ if (vcpu->kvm->arch.pmu_filter &&+ !test_bit(evsel, vcpu->kvm->arch.pmu_filter))+ val |= evtyper_set;++ val &= ~evtyper_clr;+ write_pmevtypern(i, val);
</snip>
This all needs to be shared with writethrough_pmevtyper() instead of
open-coding the same thing.
Thanks,
Oliver
From: Will Deacon <will@kernel.org> Date: 2026-01-15 13:03:01
On Tue, Dec 09, 2025 at 03:00:59PM -0800, Oliver Upton wrote:
On Tue, Dec 09, 2025 at 08:50:57PM +0000, Colton Lewis wrote:
quoted
This series creates a new PMU scheme on ARM, a partitioned PMU that
allows reserving a subset of counters for more direct guest access,
significantly reducing overhead. More details, including performance
benchmarks, can be read in the v1 cover letter linked below.
An overview of what this series accomplishes was presented at KVM
Forum 2025. Slides [1] and video [2] are linked below.
The long duration between v4 and v5 is due to time spent on this
project being monopolized preparing this feature for internal
production. As a result, there are too many improvements to fully list
here, but I will cover the notable ones.
Thanks for reposting. I think there's still quite a bit of ground to
cover on the KVM side of this, but I would definitely appreciate it if
someone with more context on the perf side of things could chime in.
Will, IIRC you had some thoughts around counter allocation, right?
Right, I was hoping that the host counter reservation could be more
dynamic than a cmdline option. Perf already has support for pinning
events to a CPU, so the concept of some counters being unavailable
shouldn't be too much for the driver to handle. You might just need to
create some fake pinned events so that perf code understands what is
happening.
Will