From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:51:20
This reduces radix guest full entry/exit latency on POWER9 and POWER10
by almost 2x.
Nested HV guests should see smaller improvements in their L1 entry/exit,
but this is also combined with most L0 speedups also applying to nested
entry. nginx localhost throughput test in a SMP nested guest is improved
about 10% (in a direct guest it doesn't change much because it uses XIVE
for IPIs) when L0 and L1 are patched.
It does this in several main ways:
- Rearrange code to optimise SPR accesses. Mainly, avoid scoreboard
stalls.
- Test SPR values to avoid mtSPRs where possible. mtSPRs are expensive.
- Reduce mftb. mftb is expensive.
- Demand fault certain facilities to avoid saving and/or restoring them
(at the cost of fault when they are used, but this is mitigated over
a number of entries, like the facilities when context switching
processes). PM, TM, and EBB so far.
- Defer some sequences that are made just in case a guest is interrupted
in the middle of a critical section to the case where the guest is
scheduled on a different CPU, rather than every time (at the cost of
an extra IPI in this case). Namely the tlbsync sequence for radix with
GTSE, which is very expensive.
This also adds the 2nd round patches to the series, which improve
performance mostly by reducing locking, barriers, and atomics related
to the vcpus-per-vcore > 1 handling that the P9 path does not require.
Some of the numbers quoted in changelogs may have changed a bit with
patches being updated, reordered, etc. They give a bit of a guide, but
I might remove them from the final submission because they're too much
to maintain.
Changes since RFC:
- Rebased with Fabiano's HV sanitising patches at the front.
- Several demand faulting bug fixes mostly relating to nested guests.
- Removed facility demand-faulting from L0 nested entry/exit handler.
Demand faulting is still done in the L1, but not the L0. The reason
is to reduce complexity (although it's only a small amount of
complexity), reduce demand faulting overhead that may require several
interrupts, and allow better testing of the L1 demand faulting,
because we may run on hypervisors that do not implement L0 demand
faulting. In future, depending on performance and such, we could add
demand faulting to L0 nested entry handling and/or remove it from the
L1.
- Fixed a timebase problem with the HMI subcore patch.
Fabiano Rosas (2):
KVM: PPC: Book3S HV: Sanitise vcpu registers in nested path
KVM: PPC: Book3S HV: Stop forwarding all HFUs to L1
Nicholas Piggin (53):
KVM: PPC: Book3S HV: Remove TM emulation from POWER7/8 path
KVM: PPC: Book3S HV P9: Fixes for TM softpatch interrupt
KVM: PPC: Book3S HV Nested: Reflect guest PMU in-use to L0 when guest
SPRs are live
powerpc/64s: Remove WORT SPR from POWER9/10
KMV: PPC: Book3S HV P9: Use set_dec to set decrementer to host
KVM: PPC: Book3S HV P9: Use host timer accounting to avoid decrementer
read
KVM: PPC: Book3S HV P9: Use large decrementer for HDEC
KVM: PPC: Book3S HV P9: Reduce mftb per guest entry/exit
powerpc/time: add API for KVM to re-arm the host timer/decrementer
KVM: PPC: Book3S HV: POWER10 enable HAIL when running radix guests
powerpc/64s: Keep AMOR SPR a constant ~0 at runtime
KVM: PPC: Book3S HV: Don't always save PMU for guest capable of
nesting
powerpc/64s: Always set PMU control registers to frozen/disabled when
not in use
powerpc/64s: Implement PMU override command line option
KVM: PPC: Book3S HV P9: Implement PMU save/restore in C
KVM: PPC: Book3S HV P9: Factor PMU save/load into context switch
functions
KVM: PPC: Book3S HV P9: Demand fault PMU SPRs when marked not inuse
KVM: PPC: Book3S HV P9: Factor out yield_count increment
KVM: PPC: Book3S HV: CTRL SPR does not require read-modify-write
KVM: PPC: Book3S HV P9: Move SPRG restore to restore_p9_host_os_sprs
KVM: PPC: Book3S HV P9: Reduce mtmsrd instructions required to save
host SPRs
KVM: PPC: Book3S HV P9: Improve mtmsrd scheduling by delaying MSR[EE]
disable
KVM: PPC: Book3S HV P9: Add kvmppc_stop_thread to match
kvmppc_start_thread
KVM: PPC: Book3S HV: Change dec_expires to be relative to guest
timebase
KVM: PPC: Book3S HV P9: Move TB updates
KVM: PPC: Book3S HV P9: Optimise timebase reads
KVM: PPC: Book3S HV P9: Avoid SPR scoreboard stalls
KVM: PPC: Book3S HV P9: Only execute mtSPR if the value changed
KVM: PPC: Book3S HV P9: Juggle SPR switching around
KVM: PPC: Book3S HV P9: Move vcpu register save/restore into functions
KVM: PPC: Book3S HV P9: Move host OS save/restore functions to
built-in
KVM: PPC: Book3S HV P9: Move nested guest entry into its own function
KVM: PPC: Book3S HV P9: Move remaining SPR and MSR access into low
level entry
KVM: PPC: Book3S HV P9: Implement TM fastpath for guest entry/exit
KVM: PPC: Book3S HV P9: Switch PMU to guest as late as possible
KVM: PPC: Book3S HV P9: Restrict DSISR canary workaround to processors
that require it
KVM: PPC: Book3S HV P9: More SPR speed improvements
KVM: PPC: Book3S HV P9: Demand fault EBB facility registers
KVM: PPC: Book3S HV P9: Demand fault TM facility registers
KVM: PPC: Book3S HV P9: Use Linux SPR save/restore to manage some host
SPRs
KVM: PPC: Book3S HV P9: Comment and fix MMU context switching code
KVM: PPC: Book3S HV P9: Test dawr_enabled() before saving host DAWR
SPRs
KVM: PPC: Book3S HV P9: Don't restore PSSCR if not needed
KVM: PPC: Book3S HV P9: Avoid tlbsync sequence on radix guest exit
KVM: PPC: Book3S HV Nested: Avoid extra mftb() in nested entry
KVM: PPC: Book3S HV P9: Improve mfmsr performance on entry
KVM: PPC: Book3S HV P9: Optimise hash guest SLB saving
KVM: PPC: Book3S HV P9: Add unlikely annotation for !mmu_ready
KVM: PPC: Book3S HV P9: Avoid cpu_in_guest atomics on entry and exit
KVM: PPC: Book3S HV P9: Remove most of the vcore logic
KVM: PPC: Book3S HV P9: Tidy kvmppc_create_dtl_entry
KVM: PPC: Book3S HV P9: Stop using vc->dpdes
KVM: PPC: Book3S HV P9: Remove subcore HMI handling
.../admin-guide/kernel-parameters.txt | 7 +
arch/powerpc/include/asm/asm-prototypes.h | 5 -
arch/powerpc/include/asm/kvm_asm.h | 1 +
arch/powerpc/include/asm/kvm_book3s.h | 6 +
arch/powerpc/include/asm/kvm_book3s_64.h | 6 +-
arch/powerpc/include/asm/kvm_host.h | 6 +-
arch/powerpc/include/asm/pmc.h | 7 +
arch/powerpc/include/asm/reg.h | 3 +-
arch/powerpc/include/asm/switch_to.h | 2 +
arch/powerpc/include/asm/time.h | 19 +-
arch/powerpc/kernel/cpu_setup_power.c | 12 +-
arch/powerpc/kernel/dt_cpu_ftrs.c | 8 +-
arch/powerpc/kernel/process.c | 30 +
arch/powerpc/kernel/time.c | 54 +-
arch/powerpc/kvm/book3s_64_mmu_radix.c | 4 +
arch/powerpc/kvm/book3s_hv.c | 857 ++++++++++--------
arch/powerpc/kvm/book3s_hv.h | 36 +
arch/powerpc/kvm/book3s_hv_builtin.c | 2 +
arch/powerpc/kvm/book3s_hv_hmi.c | 7 +-
arch/powerpc/kvm/book3s_hv_interrupts.S | 13 +-
arch/powerpc/kvm/book3s_hv_nested.c | 131 +--
arch/powerpc/kvm/book3s_hv_p9_entry.c | 793 ++++++++++++++--
arch/powerpc/kvm/book3s_hv_ras.c | 4 +
arch/powerpc/kvm/book3s_hv_rmhandlers.S | 116 +--
arch/powerpc/kvm/book3s_hv_tm.c | 57 +-
arch/powerpc/mm/book3s64/radix_pgtable.c | 15 -
arch/powerpc/perf/core-book3s.c | 35 +
arch/powerpc/platforms/powernv/idle.c | 10 +-
28 files changed, 1508 insertions(+), 738 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_hv.h
--
2.23.0
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:51:45
TM fake-suspend emulation is only used by POWER9. Remove it from the old
code path.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv_rmhandlers.S | 42 -------------------------
1 file changed, 42 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:52:11
The softpatch interrupt sets HSRR0 to the faulting instruction +4, so
it should subtract 4 for the faulting instruction address. Also have it
emulate and deliver HFAC interrupts correctly, which is important for
nested HV and facility demand-faulting in future.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/reg.h | 3 +-
arch/powerpc/kvm/book3s_hv.c | 35 ++++++++++++--------
arch/powerpc/kvm/book3s_hv_tm.c | 57 +++++++++++++++++++++------------
3 files changed, 61 insertions(+), 34 deletions(-)
@@ -1814,9 +1817,15 @@ static int kvmppc_handle_nested_exit(struct kvm_vcpu *vcpu)*modeandwastransitioningtotransactionalstate.*/r=kvmhv_p9_tm_emulation(vcpu);-break;+if(r!=-1)+break;+fallthrough;/* go to facility unavailable handler */#endif+caseBOOK3S_INTERRUPT_H_FAC_UNAVAIL:+r=RESUME_HOST;+break;+caseBOOK3S_INTERRUPT_HV_RM_HARD:vcpu->arch.trap=0;r=RESUME_GUEST;
@@ -74,19 +74,23 @@ int kvmhv_p9_tm_emulation(struct kvm_vcpu *vcpu)casePPC_INST_RFEBB:if((msr&MSR_PR)&&(vcpu->arch.vcore->pcr&PCR_ARCH_206)){/* generate an illegal instruction interrupt */+vcpu->arch.regs.nip-=4;kvmppc_core_queue_program(vcpu,SRR1_PROGILL);returnRESUME_GUEST;}/* check EBB facility is available */if(!(vcpu->arch.hfscr&HFSCR_EBB)){-/* generate an illegal instruction interrupt */-kvmppc_core_queue_program(vcpu,SRR1_PROGILL);-returnRESUME_GUEST;+vcpu->arch.regs.nip-=4;+vcpu->arch.hfscr&=~HFSCR_INTR_CAUSE;+vcpu->arch.hfscr|=(u64)FSCR_EBB_LG<<56;+vcpu->arch.trap=BOOK3S_INTERRUPT_H_FAC_UNAVAIL;+return-1;/* rerun host interrupt handler */}if((msr&MSR_PR)&&!(vcpu->arch.fscr&FSCR_EBB)){/* generate a facility unavailable interrupt */-vcpu->arch.fscr=(vcpu->arch.fscr&~(0xffull<<56))|-((u64)FSCR_EBB_LG<<56);+vcpu->arch.regs.nip-=4;+vcpu->arch.fscr&=~FSCR_INTR_CAUSE;+vcpu->arch.fscr|=(u64)FSCR_EBB_LG<<56;kvmppc_book3s_queue_irqprio(vcpu,BOOK3S_INTERRUPT_FAC_UNAVAIL);returnRESUME_GUEST;}
@@ -123,19 +127,23 @@ int kvmhv_p9_tm_emulation(struct kvm_vcpu *vcpu)/* check for PR=1 and arch 2.06 bit set in PCR */if((msr&MSR_PR)&&(vcpu->arch.vcore->pcr&PCR_ARCH_206)){/* generate an illegal instruction interrupt */+vcpu->arch.regs.nip-=4;kvmppc_core_queue_program(vcpu,SRR1_PROGILL);returnRESUME_GUEST;}/* check for TM disabled in the HFSCR or MSR */if(!(vcpu->arch.hfscr&HFSCR_TM)){-/* generate an illegal instruction interrupt */-kvmppc_core_queue_program(vcpu,SRR1_PROGILL);-returnRESUME_GUEST;+vcpu->arch.regs.nip-=4;+vcpu->arch.hfscr&=~HFSCR_INTR_CAUSE;+vcpu->arch.hfscr|=(u64)FSCR_TM_LG<<56;+vcpu->arch.trap=BOOK3S_INTERRUPT_H_FAC_UNAVAIL;+return-1;/* rerun host interrupt handler */}if(!(msr&MSR_TM)){/* generate a facility unavailable interrupt */-vcpu->arch.fscr=(vcpu->arch.fscr&~(0xffull<<56))|-((u64)FSCR_TM_LG<<56);+vcpu->arch.regs.nip-=4;+vcpu->arch.fscr&=~FSCR_INTR_CAUSE;+vcpu->arch.fscr|=(u64)FSCR_TM_LG<<56;kvmppc_book3s_queue_irqprio(vcpu,BOOK3S_INTERRUPT_FAC_UNAVAIL);returnRESUME_GUEST;
@@ -158,20 +166,24 @@ int kvmhv_p9_tm_emulation(struct kvm_vcpu *vcpu)case(PPC_INST_TRECLAIM&PO_XOP_OPCODE_MASK):/* check for TM disabled in the HFSCR or MSR */if(!(vcpu->arch.hfscr&HFSCR_TM)){-/* generate an illegal instruction interrupt */-kvmppc_core_queue_program(vcpu,SRR1_PROGILL);-returnRESUME_GUEST;+vcpu->arch.regs.nip-=4;+vcpu->arch.hfscr&=~HFSCR_INTR_CAUSE;+vcpu->arch.hfscr|=(u64)FSCR_TM_LG<<56;+vcpu->arch.trap=BOOK3S_INTERRUPT_H_FAC_UNAVAIL;+return-1;/* rerun host interrupt handler */}if(!(msr&MSR_TM)){/* generate a facility unavailable interrupt */-vcpu->arch.fscr=(vcpu->arch.fscr&~(0xffull<<56))|-((u64)FSCR_TM_LG<<56);+vcpu->arch.regs.nip-=4;+vcpu->arch.fscr&=~FSCR_INTR_CAUSE;+vcpu->arch.fscr|=(u64)FSCR_TM_LG<<56;kvmppc_book3s_queue_irqprio(vcpu,BOOK3S_INTERRUPT_FAC_UNAVAIL);returnRESUME_GUEST;}/* If no transaction active, generate TM bad thing */if(!MSR_TM_ACTIVE(msr)){+vcpu->arch.regs.nip-=4;kvmppc_core_queue_program(vcpu,SRR1_PROGTM);returnRESUME_GUEST;}
@@ -196,20 +208,24 @@ int kvmhv_p9_tm_emulation(struct kvm_vcpu *vcpu)/* XXX do we need to check for PR=0 here? *//* check for TM disabled in the HFSCR or MSR */if(!(vcpu->arch.hfscr&HFSCR_TM)){-/* generate an illegal instruction interrupt */-kvmppc_core_queue_program(vcpu,SRR1_PROGILL);-returnRESUME_GUEST;+vcpu->arch.regs.nip-=4;+vcpu->arch.hfscr&=~HFSCR_INTR_CAUSE;+vcpu->arch.hfscr|=(u64)FSCR_TM_LG<<56;+vcpu->arch.trap=BOOK3S_INTERRUPT_H_FAC_UNAVAIL;+return-1;/* rerun host interrupt handler */}if(!(msr&MSR_TM)){/* generate a facility unavailable interrupt */-vcpu->arch.fscr=(vcpu->arch.fscr&~(0xffull<<56))|-((u64)FSCR_TM_LG<<56);+vcpu->arch.regs.nip-=4;+vcpu->arch.fscr&=~FSCR_INTR_CAUSE;+vcpu->arch.fscr|=(u64)FSCR_TM_LG<<56;kvmppc_book3s_queue_irqprio(vcpu,BOOK3S_INTERRUPT_FAC_UNAVAIL);returnRESUME_GUEST;}/* If transaction active or TEXASR[FS] = 0, bad thing */if(MSR_TM_ACTIVE(msr)||!(vcpu->arch.texasr&TEXASR_FS)){+vcpu->arch.regs.nip-=4;kvmppc_core_queue_program(vcpu,SRR1_PROGTM);returnRESUME_GUEST;}
@@ -224,6 +240,7 @@ int kvmhv_p9_tm_emulation(struct kvm_vcpu *vcpu)}/* What should we do here? We didn't recognize the instruction */+vcpu->arch.regs.nip-=4;kvmppc_core_queue_program(vcpu,SRR1_PROGILL);pr_warn_ratelimited("Unrecognized TM-related instruction %#x for emulation",instr);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:52:36
From: Fabiano Rosas <redacted>
As one of the arguments of the H_ENTER_NESTED hypercall, the nested
hypervisor (L1) prepares a structure containing the values of various
hypervisor-privileged registers with which it wants the nested guest
(L2) to run. Since the nested HV runs in supervisor mode it needs the
host to write to these registers.
To stop a nested HV manipulating this mechanism and using a nested
guest as a proxy to access a facility that has been made unavailable
to it, we have a routine that sanitises the values of the HV registers
before copying them into the nested guest's vcpu struct.
However, when coming out of the guest the values are copied as they
were back into L1 memory, which means that any sanitisation we did
during guest entry will be exposed to L1 after H_ENTER_NESTED returns.
This patch alters this sanitisation to have effect on the vcpu->arch
registers directly before entering and after exiting the guest,
leaving the structure that is copied back into L1 unchanged (except
when we really want L1 to access the value, e.g the Cause bits of
HFSCR).
Signed-off-by: Fabiano Rosas <redacted>
---
arch/powerpc/kvm/book3s_hv_nested.c | 100 +++++++++++++++-------------
1 file changed, 52 insertions(+), 48 deletions(-)
@@ -134,49 +143,7 @@ static void save_hv_return_state(struct kvm_vcpu *vcpu, int trap,}}-/*-*ThiscanresultinsomeL0HVregisterstatebeingleakedtoanL1-*hypervisorwhenthehv_guest_stateiscopiedbacktotheguestafter-*beingmodifiedhere.-*-*Thereisnoknownproblemwithsuchaleak,andinmanycasesthese-*registersettingscouldbederivedbytheguestbyobservingbehaviour-*andtiming,interrupts,etc.,butitisanissuetoconsider.-*/-staticvoidsanitise_hv_regs(structkvm_vcpu*vcpu,structhv_guest_state*hr)-{-structkvmppc_vcore*vc=vcpu->arch.vcore;-u64mask;--/*-*Don'tletL1changeLPCRbitsfortheL2exceptthese:-*/-mask=LPCR_DPFD|LPCR_ILE|LPCR_TC|LPCR_AIL|LPCR_LD|-LPCR_LPES|LPCR_MER;--/*-*Additionalfilteringisrequireddependingonhardware-*andconfiguration.-*/-hr->lpcr=kvmppc_filter_lpcr_hv(vcpu->kvm,-(vc->lpcr&~mask)|(hr->lpcr&mask));--/*-*Don'tletL1enablefeaturesforL2whichwe'vedisabledforL1,-*butpreservetheinterruptcausefield.-*/-hr->hfscr&=(HFSCR_INTR_CAUSE|vcpu->arch.hfscr);--/* Don't let data address watchpoint match in hypervisor state */-hr->dawrx0&=~DAWRX_HYP;-hr->dawrx1&=~DAWRX_HYP;--/* Don't let completed instruction address breakpt match in HV state */-if((hr->ciabr&CIABR_PRIV)==CIABR_PRIV_HYPER)-hr->ciabr&=~CIABR_PRIV;-}--staticvoidrestore_hv_regs(structkvm_vcpu*vcpu,structhv_guest_state*hr)+staticvoidrestore_hv_regs(structkvm_vcpu*vcpu,conststructhv_guest_state*hr){structkvmppc_vcore*vc=vcpu->arch.vcore;
@@ -288,6 +255,43 @@ static int kvmhv_write_guest_state_and_regs(struct kvm_vcpu *vcpu,sizeof(structpt_regs));}+staticvoidload_l2_hv_regs(structkvm_vcpu*vcpu,+conststructhv_guest_state*l2_hv,+conststructhv_guest_state*l1_hv,u64*lpcr)+{+structkvmppc_vcore*vc=vcpu->arch.vcore;+u64mask;++restore_hv_regs(vcpu,l2_hv);++/*+*Don'tletL1changeLPCRbitsfortheL2exceptthese:+*/+mask=LPCR_DPFD|LPCR_ILE|LPCR_TC|LPCR_AIL|LPCR_LD|+LPCR_LPES|LPCR_MER;++/*+*Additionalfilteringisrequireddependingonhardware+*andconfiguration.+*/+*lpcr=kvmppc_filter_lpcr_hv(vcpu->kvm,+(vc->lpcr&~mask)|(*lpcr&mask));++/*+*Don'tletL1enablefeaturesforL2whichwe'vedisabledforL1,+*butpreservetheinterruptcausefield.+*/+vcpu->arch.hfscr=l2_hv->hfscr&(HFSCR_INTR_CAUSE|l1_hv->hfscr);++/* Don't let data address watchpoint match in hypervisor state */+vcpu->arch.dawrx0=l2_hv->dawrx0&~DAWRX_HYP;+vcpu->arch.dawrx1=l2_hv->dawrx1&~DAWRX_HYP;++/* Don't let completed instruction address breakpt match in HV state */+if((l2_hv->ciabr&CIABR_PRIV)==CIABR_PRIV_HYPER)+vcpu->arch.ciabr=l2_hv->ciabr&~CIABR_PRIV;+}+longkvmhv_enter_nested_guest(structkvm_vcpu*vcpu){longinterr,r;
@@ -296,7 +300,7 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)structhv_guest_statel2_hv={0},saved_l1_hv;structkvmppc_vcore*vc=vcpu->arch.vcore;u64hv_ptr,regs_ptr;-u64hdec_exp;+u64hdec_exp,lpcr;s64delta_purr,delta_spurr,delta_ic,delta_vtb;if(vcpu->kvm->arch.l1_ptcr==0)
@@ -369,8 +373,8 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)/* Guest must always run with ME enabled, HV disabled. */vcpu->arch.shregs.msr=(vcpu->arch.regs.msr|MSR_ME)&~MSR_HV;-sanitise_hv_regs(vcpu,&l2_hv);-restore_hv_regs(vcpu,&l2_hv);+lpcr=l2_hv.lpcr;+load_l2_hv_regs(vcpu,&l2_hv,&saved_l1_hv,&lpcr);vcpu->arch.ret=RESUME_GUEST;vcpu->arch.trap=0;
@@ -380,7 +384,7 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)r=RESUME_HOST;break;}-r=kvmhv_run_single_vcpu(vcpu,hdec_exp,l2_hv.lpcr);+r=kvmhv_run_single_vcpu(vcpu,hdec_exp,lpcr);}while(is_kvmppc_resume_guest(r));/* save L2 state for return */
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:53:06
From: Fabiano Rosas <redacted>
If the nested hypervisor has no access to a facility because it has
been disabled by the host, it should also not be able to see the
Hypervisor Facility Unavailable that arises from one of its guests
trying to access the facility.
This patch turns a HFU that happened in L2 into a Hypervisor Emulation
Assistance interrupt and forwards it to L1 for handling. The ones that
happened because L1 explicitly disabled the facility for L2 are still
let through, along with the corresponding Cause bits in the HFSCR.
Signed-off-by: Fabiano Rosas <redacted>
---
arch/powerpc/kvm/book3s_hv_nested.c | 27 ++++++++++++++++++++++++---
1 file changed, 24 insertions(+), 3 deletions(-)
@@ -137,6 +137,27 @@ static void save_hv_return_state(struct kvm_vcpu *vcpu, int trap,caseBOOK3S_INTERRUPT_H_INST_STORAGE:hr->asdr=vcpu->arch.fault_gpa;break;+caseBOOK3S_INTERRUPT_H_FAC_UNAVAIL:+{+u8cause=vcpu->arch.hfscr>>56;++WARN_ON_ONCE(cause>=BITS_PER_LONG);++if(!(hr->hfscr&(1UL<<cause)))+break;++/*+*Wehavedisabledthisfacility,soitdoesnot+*existfromL1'sperspective.TurnitintoaHEAI.+*/+vcpu->arch.trap=BOOK3S_INTERRUPT_H_EMUL_ASSIST;+kvmppc_load_last_inst(vcpu,INST_GENERIC,&vcpu->arch.emul_inst);++/* Don't leak the cause field */+hr->hfscr&=~HFSCR_INTR_CAUSE;++fallthrough;+}caseBOOK3S_INTERRUPT_H_EMUL_ASSIST:hr->heir=vcpu->arch.emul_inst;break;
@@ -394,7 +415,7 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)delta_spurr=vcpu->arch.spurr-l2_hv.spurr;delta_ic=vcpu->arch.ic-l2_hv.ic;delta_vtb=vc->vtb-l2_hv.vtb;-save_hv_return_state(vcpu,vcpu->arch.trap,&l2_hv);+save_hv_return_state(vcpu,&l2_hv);/* restore L1 state */vcpu->arch.nested=NULL;
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:53:29
After the L1 saves its PMU SPRs but before loading the L2's PMU SPRs,
switch the pmcregs_in_use field in the L1 lppaca to the value advertised
by the L2 in its VPA. On the way out of the L2, set it back after saving
the L2 PMU registers (if they were in-use).
This transfers the PMU liveness indication between the L1 and L2 at the
points where the registers are not live.
This fixes the nested HV bug for which a workaround was added to the L0
HV by commit 63279eeb7f93a ("KVM: PPC: Book3S HV: Always save guest pmu
for guest capable of nesting"), which explains the problem in detail.
That workaround is no longer required for guests that include this bug
fix.
Fixes: 360cae313702 ("KVM: PPC: Book3S HV: Nested guest entry via hypercall")
Reviewed-by: Fabiano Rosas <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/pmc.h | 7 +++++++
arch/powerpc/kvm/book3s_hv.c | 20 ++++++++++++++++++++
2 files changed, 27 insertions(+)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:53:51
This register is not architected and not implemented in POWER9 or 10,
it just reads back zeroes for compatibility.
-78 cycles (9255) cycles POWER9 virt-mode NULL hcall
Reviewed-by: Fabiano Rosas <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 3 ---
arch/powerpc/platforms/powernv/idle.c | 2 --
2 files changed, 5 deletions(-)
@@ -667,7 +667,6 @@ static unsigned long power9_idle_stop(unsigned long psscr)sprs.purr=mfspr(SPRN_PURR);sprs.spurr=mfspr(SPRN_SPURR);sprs.dscr=mfspr(SPRN_DSCR);-sprs.wort=mfspr(SPRN_WORT);sprs.ciabr=mfspr(SPRN_CIABR);sprs.mmcra=mfspr(SPRN_MMCRA);
@@ -785,7 +784,6 @@ static unsigned long power9_idle_stop(unsigned long psscr)mtspr(SPRN_PURR,sprs.purr);mtspr(SPRN_SPURR,sprs.spurr);mtspr(SPRN_DSCR,sprs.dscr);-mtspr(SPRN_WORT,sprs.wort);mtspr(SPRN_CIABR,sprs.ciabr);mtspr(SPRN_MMCRA,sprs.mmcra);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:54:15
The host Linux timer code arms the decrementer with the value
'decrementers_next_tb - current_tb' using set_dec(), which stores
val - 1 on Book3S-64, which is not quite the same as what KVM does
to re-arm the host decrementer when exiting the guest.
This shouldn't be a significant change, but it makes the logic match
and avoids this small extra change being brought into the next patch.
Suggested-by: Alexey Kardashevskiy <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
@@ -4019,7 +4019,7 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vc->entry_exit_map=0x101;vc->in_guest=0;-mtspr(SPRN_DEC,local_paca->kvm_hstate.dec_expires-mftb());+set_dec(local_paca->kvm_hstate.dec_expires-mftb());/* We may have raced with new irq work */if(test_irq_work_pending())set_dec(1);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:54:38
There is no need to save away the host DEC value, as it is derived
from the host timer subsystem which maintains the next timer time,
so it can be restored from there.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/time.h | 5 +++++
arch/powerpc/kernel/time.c | 1 +
arch/powerpc/kvm/book3s_hv.c | 14 +++++++-------
3 files changed, 13 insertions(+), 7 deletions(-)
@@ -4019,7 +4018,8 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vc->entry_exit_map=0x101;vc->in_guest=0;-set_dec(local_paca->kvm_hstate.dec_expires-mftb());+next_timer=timer_get_next_tb();+set_dec(next_timer-mftb());/* We may have raced with new irq work */if(test_irq_work_pending())set_dec(1);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:55:00
On processors that don't suppress the HDEC exceptions when LPCR[HDICE]=0,
this could help reduce needless guest exits due to leftover exceptions on
entering the guest.
Reviewed-by: Alexey Kardashevskiy <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/time.h | 2 ++
arch/powerpc/kernel/time.c | 1 +
arch/powerpc/kvm/book3s_hv_p9_entry.c | 3 ++-
3 files changed, 5 insertions(+), 1 deletion(-)
@@ -504,7 +504,8 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvc->tb_offset_applied=0;}-mtspr(SPRN_HDEC,0x7fffffff);+/* HDEC must be at least as large as DEC, so decrementer_max fits */+mtspr(SPRN_HDEC,decrementer_max);save_clear_guest_mmu(kvm,vcpu);switch_mmu_to_host(kvm,host_pidr);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:55:25
mftb is serialising (dispatch next-to-complete) so it is heavy weight
for a mfspr. Avoid reading it multiple times in the entry or exit paths.
A small number of cycles delay to timers is tolerable.
-118 cycles (9137) POWER9 virt-mode NULL hcall
Reviewed-by: Fabiano Rosas <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 4 ++--
arch/powerpc/kvm/book3s_hv_p9_entry.c | 5 +++--
2 files changed, 5 insertions(+), 4 deletions(-)
@@ -4019,7 +4019,7 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vc->in_guest=0;next_timer=timer_get_next_tb();-set_dec(next_timer-mftb());+set_dec(next_timer-tb);/* We may have raced with new irq work */if(test_irq_work_pending())set_dec(1);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:55:48
Rather than have KVM look up the host timer and fiddle with the
irq-work internal details, have the powerpc/time.c code provide a
function for KVM to re-arm the Linux timer code when exiting a
guest.
This is implementation has an improvement over existing code of
marking a decrementer interrupt as soft-pending if a timer has
expired, rather than setting DEC to a -ve value, which tended to
cause host timers to take two interrupts (first hdec to exit the
guest, then the immediate dec).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/time.h | 16 +++-------
arch/powerpc/kernel/time.c | 52 +++++++++++++++++++++++++++------
arch/powerpc/kvm/book3s_hv.c | 7 ++---
3 files changed, 49 insertions(+), 26 deletions(-)
@@ -542,13 +552,44 @@ void arch_irq_work_raise(void)preempt_enable();}+staticvoidset_dec_or_work(u64val)+{+set_dec(val);+/* We may have raced with new irq work */+if(unlikely(test_irq_work_pending()))+set_dec(1);+}+#else /* CONFIG_IRQ_WORK */#define test_irq_work_pending() 0#define clear_irq_work_pending()+staticvoidset_dec_or_work(u64val)+{+set_dec(val);+}#endif /* CONFIG_IRQ_WORK */+#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE+voidtimer_rearm_host_dec(u64now)+{+u64*next_tb=this_cpu_ptr(&decrementers_next_tb);++WARN_ON_ONCE(!arch_irqs_disabled());+WARN_ON_ONCE(mfmsr()&MSR_EE);++if(now>=*next_tb){+local_paca->irq_happened|=PACA_IRQ_DEC;+}else{+now=*next_tb-now;+if(now<=decrementer_max)+set_dec_or_work(now);+}+}+EXPORT_SYMBOL_GPL(timer_rearm_host_dec);+#endif+/**timer_interrupt-getscalledwhenthedecrementeroverflows,*withinterruptsdisabled.
@@ -609,10 +650,7 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(timer_interrupt)}else{now=*next_tb-now;if(now<=decrementer_max)-set_dec(now);-/* We may have raced with new irq work */-if(test_irq_work_pending())-set_dec(1);+set_dec_or_work(now);__this_cpu_inc(irq_stat.timer_irqs_others);}
@@ -854,11 +892,7 @@ static int decrementer_set_next_event(unsigned long evt,structclock_event_device*dev){__this_cpu_write(decrementers_next_tb,get_tb()+evt);-set_dec(evt);--/* We may have raced with new irq work */-if(test_irq_work_pending())-set_dec(1);+set_dec_or_work(evt);return0;}
@@ -4018,11 +4018,8 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vc->entry_exit_map=0x101;vc->in_guest=0;-next_timer=timer_get_next_tb();-set_dec(next_timer-tb);-/* We may have raced with new irq work */-if(test_irq_work_pending())-set_dec(1);+timer_rearm_host_dec(tb);+mtspr(SPRN_SPRG_VDSO_WRITE,local_paca->sprg_vdso);kvmhv_load_host_pmu();
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:56:26
HV interrupts may be taken with the MMU enabled when radix guests are
running. Enable LPCR[HAIL] on ISA v3.1 processors for radix guests.
Make this depend on the host LPCR[HAIL] being enabled. Currently that is
always enabled, but having this test means any issue that might require
LPCR[HAIL] to be disabled in the host will not have to be duplicated in
KVM.
-1380 cycles on P10 NULL hcall entry+exit
Reviewed-by: Fabiano Rosas <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 29 +++++++++++++++++++++++++----
1 file changed, 25 insertions(+), 4 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:56:50
This register controls supervisor SPR modifications, and as such is only
relevant for KVM. KVM always sets AMOR to ~0 on guest entry, and never
restores it coming back out to the host, so it can be kept constant and
avoid the mtSPR in KVM guest entry.
-21 cycles (9116) cycles POWER9 virt-mode NULL hcall
Reviewed-by: Fabiano Rosas <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/cpu_setup_power.c | 8 ++++++++
arch/powerpc/kernel/dt_cpu_ftrs.c | 2 ++
arch/powerpc/kvm/book3s_hv_p9_entry.c | 2 --
arch/powerpc/kvm/book3s_hv_rmhandlers.S | 2 --
arch/powerpc/mm/book3s64/radix_pgtable.c | 15 ---------------
arch/powerpc/platforms/powernv/idle.c | 8 +++-----
6 files changed, 13 insertions(+), 24 deletions(-)
@@ -306,8 +306,8 @@ struct p7_sprs {/* per thread SPRs that get lost in shallow states */u64amr;u64iamr;-u64amor;u64uamor;+/* amor is restored to constant ~0 */};staticunsignedlongpower7_idle_insn(unsignedlongtype)
@@ -378,7 +378,6 @@ static unsigned long power7_idle_insn(unsigned long type)if(cpu_has_feature(CPU_FTR_ARCH_207S)){sprs.amr=mfspr(SPRN_AMR);sprs.iamr=mfspr(SPRN_IAMR);-sprs.amor=mfspr(SPRN_AMOR);sprs.uamor=mfspr(SPRN_UAMOR);}
@@ -397,7 +396,7 @@ static unsigned long power7_idle_insn(unsigned long type)*/mtspr(SPRN_AMR,sprs.amr);mtspr(SPRN_IAMR,sprs.iamr);-mtspr(SPRN_AMOR,sprs.amor);+mtspr(SPRN_AMOR,~0);mtspr(SPRN_UAMOR,sprs.uamor);}}
@@ -687,7 +686,6 @@ static unsigned long power9_idle_stop(unsigned long psscr)sprs.amr=mfspr(SPRN_AMR);sprs.iamr=mfspr(SPRN_IAMR);-sprs.amor=mfspr(SPRN_AMOR);sprs.uamor=mfspr(SPRN_UAMOR);srr1=isa300_idle_stop_mayloss(psscr);/* go idle */
@@ -708,7 +706,7 @@ static unsigned long power9_idle_stop(unsigned long psscr)*/mtspr(SPRN_AMR,sprs.amr);mtspr(SPRN_IAMR,sprs.iamr);-mtspr(SPRN_AMOR,sprs.amor);+mtspr(SPRN_AMOR,~0);mtspr(SPRN_UAMOR,sprs.uamor);/*
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:57:14
Revert the workaround added by commit 63279eeb7f93a ("KVM: PPC: Book3S
HV: Always save guest pmu for guest capable of nesting").
Nested capable guests running with the earlier commit ("KVM: PPC: Book3S
HV Nested: Indicate guest PMU in-use in VPA") will now indicate the PMU
in-use status of their guests, which means the parent does not need to
unconditionally save the PMU for nested capable guests.
This will cause the PMU to break for nested guests when running older
nested hypervisor guests under a kernel with this change. It's unclear
there's an easy way to avoid that, so this could wait for a release or
so for the fix to filter into stable kernels.
-134 cycles (8982) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 2 --
1 file changed, 2 deletions(-)
@@ -4003,8 +4003,6 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu->arch.vpa.dirty=1;save_pmu=lp->pmcregs_in_use;}-/* Must save pmu if this guest is capable of running nested guests */-save_pmu|=nesting_enabled(vcpu->kvm);kvmhv_save_guest_pmu(vcpu,save_pmu);#ifdef CONFIG_PPC_PSERIES
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:57:37
KVM PMU management code looks for particular frozen/disabled bits in
the PMU registers so it knows whether it must clear them when coming
out of a guest or not. Setting this up helps KVM make these optimisations
without getting confused. Longer term the better approach might be to
move guest/host PMU switching to the perf subsystem.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/cpu_setup_power.c | 4 ++--
arch/powerpc/kernel/dt_cpu_ftrs.c | 6 +++---
arch/powerpc/kvm/book3s_hv.c | 5 +++++
3 files changed, 10 insertions(+), 5 deletions(-)
@@ -2691,6 +2691,11 @@ static int kvmppc_core_vcpu_create_hv(struct kvm_vcpu *vcpu)#endif#endifvcpu->arch.mmcr[0]=MMCR0_FC;+if(cpu_has_feature(CPU_FTR_ARCH_31)){+vcpu->arch.mmcr[0]|=MMCR0_PMCCEXT;+vcpu->arch.mmcra=MMCRA_BHRB_DISABLE;+}+vcpu->arch.ctrl=CTRL_RUNLATCH;/* default to host PVR, since we can't spoof it */kvmppc_set_pvr_hv(vcpu,mfspr(SPRN_PVR));
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:58:07
It can be useful in simulators (with very constrained environments)
to allow some PMCs to run from boot so they can be sampled directly
by a test harness, rather than having to run perf.
A previous change freezes counters at boot by default, so provide
a boot time option to un-freeze (plus a bit more flexibility).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
.../admin-guide/kernel-parameters.txt | 7 ++++
arch/powerpc/perf/core-book3s.c | 35 +++++++++++++++++++
2 files changed, 42 insertions(+)
@@ -4089,6 +4089,13 @@ Override pmtimer IOPort with a hex value. e.g. pmtmr=0x508+ pmu= [PPC] Manually enable the PMU.+ Enable the PMU by setting MMCR0 to 0 (clear FC bit).+ This option is implemented for Book3S processors.+ If a number is given, then MMCR1 is set to that number,+ otherwise (e.g., 'pmu=on'), it is left 0. The perf+ subsystem is disabled if this option is used.+ pm_debug_messages [SUSPEND,KNL] Enable suspend/resume debug messages during boot up.
@@ -2428,8 +2428,24 @@ int register_power_pmu(struct power_pmu *pmu)}#ifdef CONFIG_PPC64+staticboolpmu_override=false;+staticunsignedlongpmu_override_val;+staticvoiddo_pmu_override(void*data)+{+ppc_set_pmu_inuse(1);+if(pmu_override_val)+mtspr(SPRN_MMCR1,pmu_override_val);+mtspr(SPRN_MMCR0,mfspr(SPRN_MMCR0)&~MMCR0_FC);+}+staticint__initinit_ppc64_pmu(void){+if(cpu_has_feature(CPU_FTR_HVMODE)&&pmu_override){+printk(KERN_WARNING"perf: disabling perf due to pmu= command line option.\n");+on_each_cpu(do_pmu_override,NULL,1);+return0;+}+/* run through all the pmu drivers one at a time */if(!init_power5_pmu())return0;
@@ -2451,4 +2467,23 @@ static int __init init_ppc64_pmu(void)returninit_generic_compat_pmu();}early_initcall(init_ppc64_pmu);++staticint__initpmu_setup(char*str)+{+unsignedlongval;++if(!early_cpu_has_feature(CPU_FTR_HVMODE))+return0;++pmu_override=true;++if(kstrtoul(str,0,&val))+val=0;++pmu_override_val=val;++return1;+}+__setup("pmu=",pmu_setup);+#endif
@@ -3735,6 +3735,188 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc)trace_kvmppc_run_core(vc,1);}+/*+*Privileged(non-hypervisor)hostregisterstosave.+*/+structp9_host_os_sprs{+unsignedlongdscr;+unsignedlongtidr;+unsignedlongiamr;+unsignedlongamr;+unsignedlongfscr;++unsignedintpmc1;+unsignedintpmc2;+unsignedintpmc3;+unsignedintpmc4;+unsignedintpmc5;+unsignedintpmc6;+unsignedlongmmcr0;+unsignedlongmmcr1;+unsignedlongmmcr2;+unsignedlongmmcr3;+unsignedlongmmcra;+unsignedlongsiar;+unsignedlongsier1;+unsignedlongsier2;+unsignedlongsier3;+unsignedlongsdar;+};++staticvoidfreeze_pmu(unsignedlongmmcr0,unsignedlongmmcra)+{+if(!(mmcr0&MMCR0_FC))+gotodo_freeze;+if(mmcra&MMCRA_SAMPLE_ENABLE)+gotodo_freeze;+if(cpu_has_feature(CPU_FTR_ARCH_31)){+if(!(mmcr0&MMCR0_PMCCEXT))+gotodo_freeze;+if(!(mmcra&MMCRA_BHRB_DISABLE))+gotodo_freeze;+}+return;++do_freeze:+mmcr0=MMCR0_FC;+mmcra=0;+if(cpu_has_feature(CPU_FTR_ARCH_31)){+mmcr0|=MMCR0_PMCCEXT;+mmcra=MMCRA_BHRB_DISABLE;+}++mtspr(SPRN_MMCR0,mmcr0);+mtspr(SPRN_MMCRA,mmcra);+isync();+}++staticvoidsave_p9_host_pmu(structp9_host_os_sprs*host_os_sprs)+{+if(ppc_get_pmu_inuse()){+/*+*ItmightbebettertoputPMUhandling(atleastforthe+*host)intheperfsubsystembecauseitknowsmoreaboutwhat+*isbeingused.+*/++/* POWER9, POWER10 do not implement HPMC or SPMC */++host_os_sprs->mmcr0=mfspr(SPRN_MMCR0);+host_os_sprs->mmcra=mfspr(SPRN_MMCRA);++freeze_pmu(host_os_sprs->mmcr0,host_os_sprs->mmcra);++host_os_sprs->pmc1=mfspr(SPRN_PMC1);+host_os_sprs->pmc2=mfspr(SPRN_PMC2);+host_os_sprs->pmc3=mfspr(SPRN_PMC3);+host_os_sprs->pmc4=mfspr(SPRN_PMC4);+host_os_sprs->pmc5=mfspr(SPRN_PMC5);+host_os_sprs->pmc6=mfspr(SPRN_PMC6);+host_os_sprs->mmcr1=mfspr(SPRN_MMCR1);+host_os_sprs->mmcr2=mfspr(SPRN_MMCR2);+host_os_sprs->sdar=mfspr(SPRN_SDAR);+host_os_sprs->siar=mfspr(SPRN_SIAR);+host_os_sprs->sier1=mfspr(SPRN_SIER);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+host_os_sprs->mmcr3=mfspr(SPRN_MMCR3);+host_os_sprs->sier2=mfspr(SPRN_SIER2);+host_os_sprs->sier3=mfspr(SPRN_SIER3);+}+}+}++staticvoidload_p9_guest_pmu(structkvm_vcpu*vcpu)+{+mtspr(SPRN_PMC1,vcpu->arch.pmc[0]);+mtspr(SPRN_PMC2,vcpu->arch.pmc[1]);+mtspr(SPRN_PMC3,vcpu->arch.pmc[2]);+mtspr(SPRN_PMC4,vcpu->arch.pmc[3]);+mtspr(SPRN_PMC5,vcpu->arch.pmc[4]);+mtspr(SPRN_PMC6,vcpu->arch.pmc[5]);+mtspr(SPRN_MMCR1,vcpu->arch.mmcr[1]);+mtspr(SPRN_MMCR2,vcpu->arch.mmcr[2]);+mtspr(SPRN_SDAR,vcpu->arch.sdar);+mtspr(SPRN_SIAR,vcpu->arch.siar);+mtspr(SPRN_SIER,vcpu->arch.sier[0]);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+mtspr(SPRN_MMCR3,vcpu->arch.mmcr[3]);+mtspr(SPRN_SIER2,vcpu->arch.sier[1]);+mtspr(SPRN_SIER3,vcpu->arch.sier[2]);+}++/* Set MMCRA then MMCR0 last */+mtspr(SPRN_MMCRA,vcpu->arch.mmcra);+mtspr(SPRN_MMCR0,vcpu->arch.mmcr[0]);+/* No isync necessary because we're starting counters */+}++staticvoidsave_p9_guest_pmu(structkvm_vcpu*vcpu)+{+structlppaca*lp;+intsave_pmu=1;++lp=vcpu->arch.vpa.pinned_addr;+if(lp)+save_pmu=lp->pmcregs_in_use;++if(save_pmu){+vcpu->arch.mmcr[0]=mfspr(SPRN_MMCR0);+vcpu->arch.mmcra=mfspr(SPRN_MMCRA);++freeze_pmu(vcpu->arch.mmcr[0],vcpu->arch.mmcra);++vcpu->arch.pmc[0]=mfspr(SPRN_PMC1);+vcpu->arch.pmc[1]=mfspr(SPRN_PMC2);+vcpu->arch.pmc[2]=mfspr(SPRN_PMC3);+vcpu->arch.pmc[3]=mfspr(SPRN_PMC4);+vcpu->arch.pmc[4]=mfspr(SPRN_PMC5);+vcpu->arch.pmc[5]=mfspr(SPRN_PMC6);+vcpu->arch.mmcr[1]=mfspr(SPRN_MMCR1);+vcpu->arch.mmcr[2]=mfspr(SPRN_MMCR2);+vcpu->arch.sdar=mfspr(SPRN_SDAR);+vcpu->arch.siar=mfspr(SPRN_SIAR);+vcpu->arch.sier[0]=mfspr(SPRN_SIER);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+vcpu->arch.mmcr[3]=mfspr(SPRN_MMCR3);+vcpu->arch.sier[1]=mfspr(SPRN_SIER2);+vcpu->arch.sier[2]=mfspr(SPRN_SIER3);+}+}else{+freeze_pmu(mfspr(SPRN_MMCR0),mfspr(SPRN_MMCRA));+}+}++staticvoidload_p9_host_pmu(structp9_host_os_sprs*host_os_sprs)+{+if(ppc_get_pmu_inuse()){+mtspr(SPRN_PMC1,host_os_sprs->pmc1);+mtspr(SPRN_PMC2,host_os_sprs->pmc2);+mtspr(SPRN_PMC3,host_os_sprs->pmc3);+mtspr(SPRN_PMC4,host_os_sprs->pmc4);+mtspr(SPRN_PMC5,host_os_sprs->pmc5);+mtspr(SPRN_PMC6,host_os_sprs->pmc6);+mtspr(SPRN_MMCR1,host_os_sprs->mmcr1);+mtspr(SPRN_MMCR2,host_os_sprs->mmcr2);+mtspr(SPRN_SDAR,host_os_sprs->sdar);+mtspr(SPRN_SIAR,host_os_sprs->siar);+mtspr(SPRN_SIER,host_os_sprs->sier1);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+mtspr(SPRN_MMCR3,host_os_sprs->mmcr3);+mtspr(SPRN_SIER2,host_os_sprs->sier2);+mtspr(SPRN_SIER3,host_os_sprs->sier3);+}++/* Set MMCRA then MMCR0 last */+mtspr(SPRN_MMCRA,host_os_sprs->mmcra);+mtspr(SPRN_MMCR0,host_os_sprs->mmcr0);+isync();+}+}+staticvoidload_spr_state(structkvm_vcpu*vcpu){mtspr(SPRN_DSCR,vcpu->arch.dscr);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:58:54
Rather than guest/host save/retsore functions, implement context switch
functions that take care of details like the VPA update for nested.
The reason to split these kind of helpers into explicit save/load
functions is mainly to schedule SPR access nicely, but PMU is a special
case where the load requires mtSPR (to stop counters) and other
difficulties, so there's less possibility to schedule those nicely. The
SPR accesses also have side-effects if the PMU is running, and in later
changes we keep the host PMU running as long as possible so this code
can be better profiled, which also complicates scheduling.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 61 +++++++++++++++++-------------------
1 file changed, 28 insertions(+), 33 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 03:59:19
The pmcregs_in_use field in the guest VPA can not be trusted to reflect
what the guest is doing with PMU SPRs, so the PMU must always be managed
(stopped) when exiting the guest, and SPR values set when entering the
guest to ensure it can't cause a covert channel or otherwise cause other
guests or the host to misbehave.
So prevent guest access to the PMU with HFSCR[PM] if pmcregs_in_use is
clear, and avoid the PMU SPR access on every partition switch. Guests
that set pmcregs_in_use incorrectly or when first setting it and using
the PMU will take a hypervisor facility unavailable interrupt that will
bring in the PMU SPRs.
-774 cycles (7759) cycles POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 1 +
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_hv.c | 133 +++++++++++++++++------
arch/powerpc/kvm/book3s_hv_nested.c | 38 +++----
4 files changed, 119 insertions(+), 54 deletions(-)
@@ -39,6 +39,7 @@ struct kvm_nested_guest {pgd_t*shadow_pgtable;/* our page table for this guest */u64l1_gr_to_hr;/* L1's addr of part'n-scoped table */u64process_table;/* process table entry for this guest */+u64hfscr;/* L1's HFSCR */longrefcnt;/* number of pointers to this struct */structmutextlb_lock;/* serialize page faults and tlbies */structkvm_nested_guest*next;
@@ -2723,6 +2746,13 @@ static int kvmppc_core_vcpu_create_hv(struct kvm_vcpu *vcpu)if(cpu_has_feature(CPU_FTR_TM_COMP))vcpu->arch.hfscr|=HFSCR_TM;+vcpu->arch.hfscr_permitted=vcpu->arch.hfscr;++/*+*PMisdemand-faultedsostartwithitclear.+*/+vcpu->arch.hfscr&=~HFSCR_PM;+kvmppc_mmu_book3s_hv_init(vcpu);vcpu->arch.state=KVMPPC_VCPU_NOTREADY;
@@ -3793,6 +3823,14 @@ static void freeze_pmu(unsigned long mmcr0, unsigned long mmcra)staticvoidswitch_pmu_to_guest(structkvm_vcpu*vcpu,structp9_host_os_sprs*host_os_sprs){+structlppaca*lp;+intload_pmu=1;++lp=vcpu->arch.vpa.pinned_addr;+if(lp)+load_pmu=lp->pmcregs_in_use;++/* Save host */if(ppc_get_pmu_inuse()){/**ItmightbebettertoputPMUhandling(atleastforthe
@@ -3827,41 +3865,47 @@ static void switch_pmu_to_guest(struct kvm_vcpu *vcpu,}#ifdef CONFIG_PPC_PSERIES+/* After saving PMU, before loading guest PMU, flip pmcregs_in_use */if(kvmhv_on_pseries()){barrier();-if(vcpu->arch.vpa.pinned_addr){-structlppaca*lp=vcpu->arch.vpa.pinned_addr;-get_lppaca()->pmcregs_in_use=lp->pmcregs_in_use;-}else{-get_lppaca()->pmcregs_in_use=1;-}+get_lppaca()->pmcregs_in_use=load_pmu;barrier();}#endif-/* load guest */-mtspr(SPRN_PMC1,vcpu->arch.pmc[0]);-mtspr(SPRN_PMC2,vcpu->arch.pmc[1]);-mtspr(SPRN_PMC3,vcpu->arch.pmc[2]);-mtspr(SPRN_PMC4,vcpu->arch.pmc[3]);-mtspr(SPRN_PMC5,vcpu->arch.pmc[4]);-mtspr(SPRN_PMC6,vcpu->arch.pmc[5]);-mtspr(SPRN_MMCR1,vcpu->arch.mmcr[1]);-mtspr(SPRN_MMCR2,vcpu->arch.mmcr[2]);-mtspr(SPRN_SDAR,vcpu->arch.sdar);-mtspr(SPRN_SIAR,vcpu->arch.siar);-mtspr(SPRN_SIER,vcpu->arch.sier[0]);+/*+*Loadguest.IftheVPAsaidthePMCsarenotinusebuttheguest+*triedtoaccessthemanyway,HFSCR[PM]willbesetbytheHFAC+*faultsowecanmakeforwardprogress.+*/+if(load_pmu||(vcpu->arch.hfscr&HFSCR_PM)){+mtspr(SPRN_PMC1,vcpu->arch.pmc[0]);+mtspr(SPRN_PMC2,vcpu->arch.pmc[1]);+mtspr(SPRN_PMC3,vcpu->arch.pmc[2]);+mtspr(SPRN_PMC4,vcpu->arch.pmc[3]);+mtspr(SPRN_PMC5,vcpu->arch.pmc[4]);+mtspr(SPRN_PMC6,vcpu->arch.pmc[5]);+mtspr(SPRN_MMCR1,vcpu->arch.mmcr[1]);+mtspr(SPRN_MMCR2,vcpu->arch.mmcr[2]);+mtspr(SPRN_SDAR,vcpu->arch.sdar);+mtspr(SPRN_SIAR,vcpu->arch.siar);+mtspr(SPRN_SIER,vcpu->arch.sier[0]);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+mtspr(SPRN_MMCR3,vcpu->arch.mmcr[3]);+mtspr(SPRN_SIER2,vcpu->arch.sier[1]);+mtspr(SPRN_SIER3,vcpu->arch.sier[2]);+}-if(cpu_has_feature(CPU_FTR_ARCH_31)){-mtspr(SPRN_MMCR3,vcpu->arch.mmcr[3]);-mtspr(SPRN_SIER2,vcpu->arch.sier[1]);-mtspr(SPRN_SIER3,vcpu->arch.sier[2]);-}+/* Set MMCRA then MMCR0 last */+mtspr(SPRN_MMCRA,vcpu->arch.mmcra);+mtspr(SPRN_MMCR0,vcpu->arch.mmcr[0]);+/* No isync necessary because we're starting counters */-/* Set MMCRA then MMCR0 last */-mtspr(SPRN_MMCRA,vcpu->arch.mmcra);-mtspr(SPRN_MMCR0,vcpu->arch.mmcr[0]);-/* No isync necessary because we're starting counters */+if(!vcpu->arch.nested&&+(vcpu->arch.hfscr_permitted&HFSCR_PM))+vcpu->arch.hfscr|=HFSCR_PM;+}}staticvoidswitch_pmu_to_host(structkvm_vcpu*vcpu,
@@ -3897,9 +3941,32 @@ static void switch_pmu_to_host(struct kvm_vcpu *vcpu,vcpu->arch.sier[1]=mfspr(SPRN_SIER2);vcpu->arch.sier[2]=mfspr(SPRN_SIER3);}-}else{++}elseif(vcpu->arch.hfscr&HFSCR_PM){+/*+*TheguestaccessedPMCSPRswithoutspecifyingtheyshould+*bepreserved,oritclearedpmcregs_in_useafterthelast+*access.Justensuretheyarefrozen.+*/freeze_pmu(mfspr(SPRN_MMCR0),mfspr(SPRN_MMCRA));-}++/*+*Demand-faultPMUregisteraccessintheguest.+*+*Thisisusedtograbtheguest'sVPApmcregs_in_usevalue+*andreflectitintothehost'sVPAinthecaseofanested+*hypervisor.+*+*Italsoavoidshavingtozero-outSPRsaftereachguest+*exittoavoidside-channelswhen.+*+*Thisisclearedherewhenweexittheguest,solaterHFSCR+*interrupthandlingcanadditbacktoruntheguestwith+*PMenablednexttime.+*/+if(!vcpu->arch.nested)+vcpu->arch.hfscr&=~HFSCR_PM;+}/* otherwise the PMU should still be frozen */#ifdef CONFIG_PPC_PSERIESif(kvmhv_on_pseries()){
@@ -152,10 +151,6 @@ static void save_hv_return_state(struct kvm_vcpu *vcpu,*/vcpu->arch.trap=BOOK3S_INTERRUPT_H_EMUL_ASSIST;kvmppc_load_last_inst(vcpu,INST_GENERIC,&vcpu->arch.emul_inst);--/* Don't leak the cause field */-hr->hfscr&=~HFSCR_INTR_CAUSE;-fallthrough;}caseBOOK3S_INTERRUPT_H_EMUL_ASSIST:
@@ -299,10 +294,10 @@ static void load_l2_hv_regs(struct kvm_vcpu *vcpu,(vc->lpcr&~mask)|(*lpcr&mask));/*-*Don'tletL1enablefeaturesforL2whichwe'vedisabledforL1,-*butpreservetheinterruptcausefield.+*Don'tletL1enablefeaturesforL2whichwedisallowforL1.+*Preservetheinterruptcausefield.*/-vcpu->arch.hfscr=l2_hv->hfscr&(HFSCR_INTR_CAUSE|l1_hv->hfscr);+vcpu->arch.hfscr=l2_hv->hfscr&(HFSCR_INTR_CAUSE|vcpu->arch.hfscr_permitted);/* Don't let data address watchpoint match in hypervisor state */vcpu->arch.dawrx0=l2_hv->dawrx0&~DAWRX_HYP;
@@ -389,6 +384,7 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)/* set L1 state to L2 state */vcpu->arch.nested=l2;vcpu->arch.nested_vcpu_id=l2_hv.vcpu_token;+l2->hfscr=l2_hv.hfscr;vcpu->arch.regs=l2_regs;/* Guest must always run with ME enabled, HV disabled. */
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:00:13
Processors that support KVM HV do not require read-modify-write of
the CTRL SPR to set/clear their thread's runlatch. Just write 1 or 0
to it.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 2 +-
arch/powerpc/kvm/book3s_hv_rmhandlers.S | 15 ++++++---------
2 files changed, 7 insertions(+), 10 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:00:39
Move the SPR update into its relevant helper function. This will
help with SPR scheduling improvements in later changes.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:01:10
This reduces the number of mtmsrd required to enable facility bits when
saving/restoring registers, by having the KVM code set all bits up front
rather than using individual facility functions that set their particular
MSR bits.
-42 cycles (7803) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/process.c | 24 +++++++++++
arch/powerpc/kvm/book3s_hv.c | 61 ++++++++++++++++++---------
arch/powerpc/kvm/book3s_hv_p9_entry.c | 1 +
3 files changed, 67 insertions(+), 19 deletions(-)
@@ -4114,8 +4115,23 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,if(next_timer<time_limit)time_limit=next_timer;+vcpu->arch.ceded=0;+save_p9_host_os_sprs(&host_os_sprs);+/* MSR bits may have been cleared by context switch */+msr=0;+if(IS_ENABLED(CONFIG_PPC_FPU))+msr|=MSR_FP;+if(cpu_has_feature(CPU_FTR_ALTIVEC))+msr|=MSR_VEC;+if(cpu_has_feature(CPU_FTR_VSX))+msr|=MSR_VSX;+if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+msr|=MSR_TM;+msr=msr_check_and_set(msr);+kvmppc_subcore_enter_guest();vc->entry_exit_map=1;
@@ -4124,12 +4140,13 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu_vpa_increment_dispatch(vcpu);if(cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)){kvmppc_restore_tm_hv(vcpu,vcpu->arch.shregs.msr,true);+msr=mfmsr();/* TM restore can update msr */+}switch_pmu_to_guest(vcpu,&host_os_sprs);-msr_check_and_set(MSR_FP|MSR_VEC|MSR_VSX);load_fp_state(&vcpu->arch.fp);#ifdef CONFIG_ALTIVECload_vr_state(&vcpu->arch.vr);
@@ -4767,6 +4783,8 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,gotodone;}+voidsave_user_regs_kvm(void);+staticintkvmppc_vcpu_run_hv(structkvm_vcpu*vcpu){structkvm_run*run=vcpu->run;
@@ -4776,19 +4794,24 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)unsignedlonguser_tar=0;unsignedintuser_vrsave;structkvm*kvm;+unsignedlongmsr;if(!vcpu->arch.sane){run->exit_reason=KVM_EXIT_INTERNAL_ERROR;return-EINVAL;}+/* No need to go into the guest when all we'll do is come back out */+if(signal_pending(current)){+run->exit_reason=KVM_EXIT_INTR;+return-EINTR;+}++#ifdef CONFIG_PPC_TRANSACTIONAL_MEM/**Don'tallowentrywithasuspendedtransaction,because*theguestentry/exitcodewillloseit.-*IftheguesthasTMenabled,saveawaytheirTM-relatedSPRs-*(theywillgetrestoredbytheTMunavailableinterrupt).*/-#ifdef CONFIG_PPC_TRANSACTIONAL_MEMif(cpu_has_feature(CPU_FTR_TM)&¤t->thread.regs&&(current->thread.regs->msr&MSR_TM)){if(MSR_TM_ACTIVE(current->thread.regs->msr)){
@@ -4796,12 +4819,6 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)run->fail_entry.hardware_entry_failure_reason=0;return-EINVAL;}-/* Enable TM so we can read the TM SPRs */-mtmsr(mfmsr()|MSR_TM);-current->thread.tm_tfhar=mfspr(SPRN_TFHAR);-current->thread.tm_tfiar=mfspr(SPRN_TFIAR);-current->thread.tm_texasr=mfspr(SPRN_TEXASR);-current->thread.regs->msr&=~MSR_TM;}#endif
@@ -4816,18 +4833,24 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)kvmppc_core_prepare_to_enter(vcpu);-/* No need to go into the guest when all we'll do is come back out */-if(signal_pending(current)){-run->exit_reason=KVM_EXIT_INTR;-return-EINTR;-}-kvm=vcpu->kvm;atomic_inc(&kvm->arch.vcpus_running);/* Order vcpus_running vs. mmu_ready, see kvmppc_alloc_reset_hpt */smp_mb();-flush_all_to_thread(current);+msr=0;+if(IS_ENABLED(CONFIG_PPC_FPU))+msr|=MSR_FP;+if(cpu_has_feature(CPU_FTR_ALTIVEC))+msr|=MSR_VEC;+if(cpu_has_feature(CPU_FTR_VSX))+msr|=MSR_VSX;+if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+msr|=MSR_TM;+msr=msr_check_and_set(msr);++save_user_regs_kvm();/* Save userspace EBB and other register values */if(cpu_has_feature(CPU_FTR_ARCH_207S)){
@@ -224,6 +224,7 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvc->tb_offset_applied=vc->tb_offset;}+/* Could avoid mfmsr by passing around, but probably no big deal */msr=mfmsr();host_hfscr=mfspr(SPRN_HFSCR);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:01:35
Moving the mtmsrd after the host SPRs are saved and before the guest
SPRs start to be loaded can prevent an SPR scoreboard stall (because
the mtmsrd is L=1 type which does not cause context synchronisation.
This is also now more convenient to combined with the mtmsrd L=0
instruction to enable facilities just below, but that is not done yet.
-12 cycles (7791) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 23 ++++++++++++++++++-----
1 file changed, 18 insertions(+), 5 deletions(-)
@@ -4119,6 +4119,18 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,save_p9_host_os_sprs(&host_os_sprs);+/*+*ThiscouldbecombinedwithMSR[RI]clearing,butthatexpands+*theunrecoverablewindow.Itwouldbebettertocoverunrecoverable+*withKVMbadinterrupthandlingratherthanuseMSR[RI]atall.+*+*MuchmoredifficultandlessworthwhiletocombinewithIR/DR+*disable.+*/+hard_irq_disable();+if(lazy_irq_pending())+return0;+/* MSR bits may have been cleared by context switch */msr=0;if(IS_ENABLED(CONFIG_PPC_FPU))
@@ -4618,6 +4630,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,structkvmppc_vcore*vc;structkvm*kvm=vcpu->kvm;structkvm_nested_guest*nested=vcpu->arch.nested;+unsignedlongflags;trace_kvmppc_run_vcpu_enter(vcpu);
@@ -4661,11 +4674,11 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,if(kvm_is_radix(kvm))kvmppc_prepare_radix_vcpu(vcpu,pcpu);-local_irq_disable();-hard_irq_disable();+/* flags save not required, but irq_pmu has no disable/enable API */+powerpc_local_irq_pmu_save(flags);if(signal_pending(current))gotosigpend;-if(lazy_irq_pending()||need_resched()||!kvm->arch.mmu_ready)+if(need_resched()||!kvm->arch.mmu_ready)gotoout;if(!nested){
@@ -4720,7 +4733,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,guest_exit_irqoff();-local_irq_enable();+powerpc_local_irq_pmu_restore(flags);cpumask_clear_cpu(pcpu,&kvm->arch.cpu_in_guest);
@@ -4778,7 +4791,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,run->exit_reason=KVM_EXIT_INTR;vcpu->arch.ret=-EINTR;out:-local_irq_enable();+powerpc_local_irq_pmu_restore(flags);preempt_enable();gotodone;}
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:01:58
Small cleanup makes it a bit easier to match up entry and exit
operations.
Reviewed-by: Fabiano Rosas <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
@@ -3045,6 +3045,13 @@ static void kvmppc_start_thread(struct kvm_vcpu *vcpu, struct kvmppc_vcore *vc)kvmppc_ipi_thread(cpu);}+/* Old path does this in asm */+staticvoidkvmppc_stop_thread(structkvm_vcpu*vcpu)+{+vcpu->cpu=-1;+vcpu->arch.thread_cpu=-1;+}+staticvoidkvmppc_wait_for_nap(intn_threads){intcpu=smp_processor_id();
@@ -406,6 +406,12 @@ static inline ulong kvmppc_get_fault_dar(struct kvm_vcpu *vcpu)returnvcpu->arch.fault_dar;}+/* Expiry time of vcpu DEC relative to host TB */+staticinlineu64kvmppc_dec_expires_host_tb(structkvm_vcpu*vcpu)+{+returnvcpu->arch.dec_expires-vcpu->arch.vcore->tb_offset;+}+staticinlineboolis_kvmppc_resume_guest(intr){return(r==RESUME_GUEST||r==RESUME_GUEST_NV);
@@ -3355,7 +3353,7 @@ static void post_guest_process(struct kvmppc_vcore *vc, bool is_master)*/spin_unlock(&vc->lock);/* cancel pending dec exception if dec is positive */-if(now<vcpu->arch.dec_expires&&+if(now<kvmppc_dec_expires_host_tb(vcpu)&&kvmppc_core_pending_dec(vcpu))kvmppc_core_dequeue_dec(vcpu);
@@ -4224,6 +4225,12 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu->arch.psscr=mfspr(SPRN_PSSCR_PR);mtspr(SPRN_PSSCR_PR,host_psscr);+dec=mfspr(SPRN_DEC);+if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */+dec=(s32)dec;+tb=mftb();+vcpu->arch.dec_expires=dec+(tb+vc->tb_offset);+/* H_CEDE has to be handled now, not later */if(trap==BOOK3S_INTERRUPT_SYSCALL&&!vcpu->arch.nested&&kvmppc_get_gpr(vcpu,3)==H_CEDE){
@@ -4262,12 +4270,6 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu->arch.slb_max=0;}-dec=mfspr(SPRN_DEC);-if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */-dec=(s32)dec;-tb=mftb();-vcpu->arch.dec_expires=dec+tb;-store_spr_state(vcpu);restore_p9_host_os_sprs(vcpu,&host_os_sprs);
@@ -4752,7 +4754,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,*byL2andtheL1decrementerisprovidedinhdec_expires*/if(kvmppc_core_pending_dec(vcpu)&&-((get_tb()<vcpu->arch.dec_expires)||+((get_tb()<kvmppc_dec_expires_host_tb(vcpu))||(trap==BOOK3S_INTERRUPT_SYSCALL&&kvmppc_get_gpr(vcpu,3)==H_ENTER_NESTED)))kvmppc_core_dequeue_dec(vcpu);
@@ -380,6 +380,7 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)/* convert TB values/offsets to host (L0) values */hdec_exp=l2_hv.hdec_expiry-vc->tb_offset;vc->tb_offset+=l2_hv.tb_offset;+vcpu->arch.dec_expires+=l2_hv.tb_offset;/* set L1 state to L2 state */vcpu->arch.nested=l2;
@@ -421,6 +422,8 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)if(l2_regs.msr&MSR_TS_MASK)vcpu->arch.shregs.msr|=MSR_TS_S;vc->tb_offset=saved_l1_hv.tb_offset;+/* XXX: is this always the same delta as saved_l1_hv.tb_offset? */+vcpu->arch.dec_expires-=l2_hv.tb_offset;restore_hv_regs(vcpu,&saved_l1_hv);vcpu->arch.purr+=delta_purr;vcpu->arch.spurr+=delta_spurr;
@@ -188,7 +188,7 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcstructkvm*kvm=vcpu->kvm;structkvm_nested_guest*nested=vcpu->arch.nested;structkvmppc_vcore*vc=vcpu->arch.vcore;-s64hdec;+s64hdec,dec;u64tb,purr,spurr;u64*exsave;boolri_set;
@@ -317,6 +317,8 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpc*/mtspr(SPRN_HDEC,hdec);+mtspr(SPRN_DEC,vcpu->arch.dec_expires-tb);+#ifdef CONFIG_PPC_TRANSACTIONAL_MEMtm_return_to_guest:#endif
@@ -461,6 +463,12 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvcpu->arch.shregs.sprg2=mfspr(SPRN_SPRG2);vcpu->arch.shregs.sprg3=mfspr(SPRN_SPRG3);+dec=mfspr(SPRN_DEC);+if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */+dec=(s32)dec;+tb=mftb();+vcpu->arch.dec_expires=dec+tb;+/* Preserve PSSCR[FAKE_SUSPEND] until we've called kvmppc_save_tm_hv */mtspr(SPRN_PSSCR,host_psscr|(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:02:54
Move the TB updates between saving and loading guest and host SPRs,
to improve scheduling by keeping issue-NTC operations together as
much as possible.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv_p9_entry.c | 36 +++++++++++++--------------
1 file changed, 18 insertions(+), 18 deletions(-)
@@ -215,15 +215,6 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvcpu->arch.ceded=0;-if(vc->tb_offset){-u64new_tb=tb+vc->tb_offset;-mtspr(SPRN_TBU40,new_tb);-tb=mftb();-if((tb&0xffffff)<(new_tb&0xffffff))-mtspr(SPRN_TBU40,new_tb+0x1000000);-vc->tb_offset_applied=vc->tb_offset;-}-/* Could avoid mfmsr by passing around, but probably no big deal */msr=mfmsr();
@@ -238,6 +229,15 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpchost_dawrx1=mfspr(SPRN_DAWRX1);}+if(vc->tb_offset){+u64new_tb=tb+vc->tb_offset;+mtspr(SPRN_TBU40,new_tb);+tb=mftb();+if((tb&0xffffff)<(new_tb&0xffffff))+mtspr(SPRN_TBU40,new_tb+0x1000000);+vc->tb_offset_applied=vc->tb_offset;+}+if(vc->pcr)mtspr(SPRN_PCR,vc->pcr|PCR_MASK);mtspr(SPRN_DPDES,vc->dpdes);
@@ -469,6 +469,15 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpctb=mftb();vcpu->arch.dec_expires=dec+tb;+if(vc->tb_offset_applied){+u64new_tb=tb-vc->tb_offset_applied;+mtspr(SPRN_TBU40,new_tb);+tb=mftb();+if((tb&0xffffff)<(new_tb&0xffffff))+mtspr(SPRN_TBU40,new_tb+0x1000000);+vc->tb_offset_applied=0;+}+/* Preserve PSSCR[FAKE_SUSPEND] until we've called kvmppc_save_tm_hv */mtspr(SPRN_PSSCR,host_psscr|(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));
@@ -503,15 +512,6 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcif(vc->pcr)mtspr(SPRN_PCR,PCR_MASK);-if(vc->tb_offset_applied){-u64new_tb=mftb()-vc->tb_offset_applied;-mtspr(SPRN_TBU40,new_tb);-tb=mftb();-if((tb&0xffffff)<(new_tb&0xffffff))-mtspr(SPRN_TBU40,new_tb+0x1000000);-vc->tb_offset_applied=0;-}-/* HDEC must be at least as large as DEC, so decrementer_max fits */mtspr(SPRN_HDEC,decrementer_max);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:03:26
Reduce the number of mfTB executed by passing the current timebase
around entry and exit code rather than read it multiple times.
-213 cycles (7578) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 2 +-
arch/powerpc/kvm/book3s_hv.c | 88 +++++++++++++-----------
arch/powerpc/kvm/book3s_hv_p9_entry.c | 33 +++++----
3 files changed, 65 insertions(+), 58 deletions(-)
@@ -4228,8 +4227,8 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,dec=mfspr(SPRN_DEC);if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */dec=(s32)dec;-tb=mftb();-vcpu->arch.dec_expires=dec+(tb+vc->tb_offset);+*tb=mftb();+vcpu->arch.dec_expires=dec+(*tb+vc->tb_offset);/* H_CEDE has to be handled now, not later */if(trap==BOOK3S_INTERRUPT_SYSCALL&&!vcpu->arch.nested&&
@@ -4534,7 +4533,7 @@ static int kvmppc_run_vcpu(struct kvm_vcpu *vcpu)if((vc->vcore_state==VCORE_PIGGYBACK||vc->vcore_state==VCORE_RUNNING)&&!VCORE_IS_EXITING(vc)){-kvmppc_create_dtl_entry(vcpu,vc);+kvmppc_create_dtl_entry(vcpu,vc,mftb());kvmppc_start_thread(vcpu,vc);trace_kvm_guest_enter(vcpu);}elseif(vc->vcore_state==VCORE_SLEEPING){
@@ -4569,7 +4568,7 @@ static int kvmppc_run_vcpu(struct kvm_vcpu *vcpu)for_each_runnable_thread(i,v,vc){kvmppc_core_prepare_to_enter(v);if(signal_pending(v->arch.run_task)){-kvmppc_remove_runnable(vc,v);+kvmppc_remove_runnable(vc,v,mftb());v->stat.signal_exits++;v->run->exit_reason=KVM_EXIT_INTR;v->arch.ret=-EINTR;
@@ -4610,7 +4609,7 @@ static int kvmppc_run_vcpu(struct kvm_vcpu *vcpu)kvmppc_vcore_end_preempt(vc);if(vcpu->arch.state==KVMPPC_VCPU_RUNNABLE){-kvmppc_remove_runnable(vc,vcpu);+kvmppc_remove_runnable(vc,vcpu,mftb());vcpu->stat.signal_exits++;run->exit_reason=KVM_EXIT_INTR;vcpu->arch.ret=-EINTR;
@@ -4638,6 +4637,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,structkvm*kvm=vcpu->kvm;structkvm_nested_guest*nested=vcpu->arch.nested;unsignedlongflags;+u64tb;trace_kvmppc_run_vcpu_enter(vcpu);
@@ -4648,7 +4648,6 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,vc=vcpu->arch.vcore;vcpu->arch.ceded=0;vcpu->arch.run_task=current;-vcpu->arch.stolen_logged=vcore_stolen_time(vc,mftb());vcpu->arch.state=KVMPPC_VCPU_RUNNABLE;vcpu->arch.busy_preempt=TB_NIL;vcpu->arch.last_inst=KVM_INST_FETCH_FAILED;
@@ -4673,7 +4672,6 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,kvmppc_update_vpas(vcpu);init_vcore_to_run(vc);-vc->preempt_tb=TB_NIL;preempt_disable();pcpu=smp_processor_id();
@@ -4683,6 +4681,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,/* flags save not required, but irq_pmu has no disable/enable API */powerpc_local_irq_pmu_save(flags);+if(signal_pending(current))gotosigpend;if(need_resched()||!kvm->arch.mmu_ready)
@@ -4705,12 +4704,17 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,gotoout;}+tb=mftb();++vcpu->arch.stolen_logged=vcore_stolen_time(vc,tb);+vc->preempt_tb=TB_NIL;+kvmppc_clear_host_core(pcpu);local_paca->kvm_hstate.napping=0;local_paca->kvm_hstate.kvm_split_mode=NULL;kvmppc_start_thread(vcpu,vc);-kvmppc_create_dtl_entry(vcpu,vc);+kvmppc_create_dtl_entry(vcpu,vc,tb);trace_kvm_guest_enter(vcpu);vc->vcore_state=VCORE_RUNNING;
@@ -4725,7 +4729,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,/* Tell lockdep that we're about to enable interrupts */trace_hardirqs_on();-trap=kvmhv_p9_guest_entry(vcpu,time_limit,lpcr);+trap=kvmhv_p9_guest_entry(vcpu,time_limit,lpcr,&tb);vcpu->arch.trap=trap;trace_hardirqs_off();
@@ -4754,7 +4758,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,*byL2andtheL1decrementerisprovidedinhdec_expires*/if(kvmppc_core_pending_dec(vcpu)&&-((get_tb()<kvmppc_dec_expires_host_tb(vcpu))||+((tb<kvmppc_dec_expires_host_tb(vcpu))||(trap==BOOK3S_INTERRUPT_SYSCALL&&kvmppc_get_gpr(vcpu,3)==H_ENTER_NESTED)))kvmppc_core_dequeue_dec(vcpu);
@@ -4790,7 +4794,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,trace_kvmppc_run_core(vc,1);done:-kvmppc_remove_runnable(vc,vcpu);+kvmppc_remove_runnable(vc,vcpu,tb);trace_kvmppc_run_vcpu_exit(vcpu);returnvcpu->arch.ret;
@@ -203,8 +203,7 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcunsignedlonghost_dawr1;unsignedlonghost_dawrx1;-tb=mftb();-hdec=time_limit-tb;+hdec=time_limit-*tb;if(hdec<0)returnBOOK3S_INTERRUPT_HV_DECREMENTER;
@@ -230,11 +229,13 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpc}if(vc->tb_offset){-u64new_tb=tb+vc->tb_offset;+u64new_tb=*tb+vc->tb_offset;mtspr(SPRN_TBU40,new_tb);-tb=mftb();-if((tb&0xffffff)<(new_tb&0xffffff))-mtspr(SPRN_TBU40,new_tb+0x1000000);+if((mftb()&0xffffff)<(new_tb&0xffffff)){+new_tb+=0x1000000;+mtspr(SPRN_TBU40,new_tb);+}+*tb=new_tb;vc->tb_offset_applied=vc->tb_offset;}
@@ -317,7 +318,7 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpc*/mtspr(SPRN_HDEC,hdec);-mtspr(SPRN_DEC,vcpu->arch.dec_expires-tb);+mtspr(SPRN_DEC,vcpu->arch.dec_expires-*tb);#ifdef CONFIG_PPC_TRANSACTIONAL_MEMtm_return_to_guest:
@@ -466,15 +467,17 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcdec=mfspr(SPRN_DEC);if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */dec=(s32)dec;-tb=mftb();-vcpu->arch.dec_expires=dec+tb;+*tb=mftb();+vcpu->arch.dec_expires=dec+*tb;if(vc->tb_offset_applied){-u64new_tb=tb-vc->tb_offset_applied;+u64new_tb=*tb-vc->tb_offset_applied;mtspr(SPRN_TBU40,new_tb);-tb=mftb();-if((tb&0xffffff)<(new_tb&0xffffff))-mtspr(SPRN_TBU40,new_tb+0x1000000);+if((mftb()&0xffffff)<(new_tb&0xffffff)){+new_tb+=0x1000000;+mtspr(SPRN_TBU40,new_tb);+}+*tb=new_tb;vc->tb_offset_applied=0;}
@@ -228,6 +228,9 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpchost_dawrx1=mfspr(SPRN_DAWRX1);}+local_paca->kvm_hstate.host_purr=mfspr(SPRN_PURR);+local_paca->kvm_hstate.host_spurr=mfspr(SPRN_SPURR);+if(vc->tb_offset){u64new_tb=*tb+vc->tb_offset;mtspr(SPRN_TBU40,new_tb);
@@ -244,8 +247,6 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcmtspr(SPRN_DPDES,vc->dpdes);mtspr(SPRN_VTB,vc->vtb);-local_paca->kvm_hstate.host_purr=mfspr(SPRN_PURR);-local_paca->kvm_hstate.host_spurr=mfspr(SPRN_SPURR);mtspr(SPRN_PURR,vcpu->arch.purr);mtspr(SPRN_SPURR,vcpu->arch.spurr);
@@ -448,10 +449,8 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpc/* Advance host PURR/SPURR by the amount used by guest */purr=mfspr(SPRN_PURR);spurr=mfspr(SPRN_SPURR);-mtspr(SPRN_PURR,local_paca->kvm_hstate.host_purr+-purr-vcpu->arch.purr);-mtspr(SPRN_SPURR,local_paca->kvm_hstate.host_spurr+-spurr-vcpu->arch.spurr);+local_paca->kvm_hstate.host_purr+=purr-vcpu->arch.purr;+local_paca->kvm_hstate.host_spurr+=spurr-vcpu->arch.spurr;vcpu->arch.purr=purr;vcpu->arch.spurr=spurr;
@@ -464,6 +463,9 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvcpu->arch.shregs.sprg2=mfspr(SPRN_SPRG2);vcpu->arch.shregs.sprg3=mfspr(SPRN_SPRG3);+vc->dpdes=mfspr(SPRN_DPDES);+vc->vtb=mfspr(SPRN_VTB);+dec=mfspr(SPRN_DEC);if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */dec=(s32)dec;
@@ -481,6 +483,9 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvc->tb_offset_applied=0;}+mtspr(SPRN_PURR,local_paca->kvm_hstate.host_purr);+mtspr(SPRN_SPURR,local_paca->kvm_hstate.host_spurr);+/* Preserve PSSCR[FAKE_SUSPEND] until we've called kvmppc_save_tm_hv */mtspr(SPRN_PSSCR,host_psscr|(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));
@@ -509,8 +514,6 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcif(cpu_has_feature(CPU_FTR_ARCH_31))asmvolatile(PPC_CP_ABORT);-vc->dpdes=mfspr(SPRN_DPDES);-vc->vtb=mfspr(SPRN_VTB);mtspr(SPRN_DPDES,0);if(vc->pcr)mtspr(SPRN_PCR,PCR_MASK);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:04:21
Keep better track of the current SPR value in places where
they are to be loaded with a new context, to reduce expensive
mtSPR operations.
-73 cycles (7354) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 64 ++++++++++++++++++++++--------------
1 file changed, 39 insertions(+), 25 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:04:44
This juggles SPR switching on the entry and exit sides to be more
symmetric, which makes the next refactoring patch possible with no
functional change.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:05:12
This should be no functional difference but makes the caller easier
to read.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 65 +++++++++++++++++++++++-------------
1 file changed, 41 insertions(+), 24 deletions(-)
@@ -4062,6 +4062,44 @@ static void store_spr_state(struct kvm_vcpu *vcpu)vcpu->arch.ctrl=mfspr(SPRN_CTRLF);}+/* Returns true if current MSR and/or guest MSR may have changed */+staticboolload_vcpu_state(structkvm_vcpu*vcpu,+structp9_host_os_sprs*host_os_sprs)+{+boolret=false;++if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)){+kvmppc_restore_tm_hv(vcpu,vcpu->arch.shregs.msr,true);+ret=true;+}++load_spr_state(vcpu,host_os_sprs);++load_fp_state(&vcpu->arch.fp);+#ifdef CONFIG_ALTIVEC+load_vr_state(&vcpu->arch.vr);+#endif+mtspr(SPRN_VRSAVE,vcpu->arch.vrsave);++returnret;+}++staticvoidstore_vcpu_state(structkvm_vcpu*vcpu)+{+store_spr_state(vcpu);++store_fp_state(&vcpu->arch.fp);+#ifdef CONFIG_ALTIVEC+store_vr_state(&vcpu->arch.vr);+#endif+vcpu->arch.vrsave=mfspr(SPRN_VRSAVE);++if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+kvmppc_save_tm_hv(vcpu,vcpu->arch.shregs.msr,true);+}+staticvoidsave_p9_host_os_sprs(structp9_host_os_sprs*host_os_sprs){if(!cpu_has_feature(CPU_FTR_ARCH_31))
@@ -4169,19 +4207,8 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu_vpa_increment_dispatch(vcpu);-if(cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)){-kvmppc_restore_tm_hv(vcpu,vcpu->arch.shregs.msr,true);-msr=mfmsr();/* TM restore can update msr */-}--load_spr_state(vcpu,&host_os_sprs);--load_fp_state(&vcpu->arch.fp);-#ifdef CONFIG_ALTIVEC-load_vr_state(&vcpu->arch.vr);-#endif-mtspr(SPRN_VRSAVE,vcpu->arch.vrsave);+if(unlikely(load_vcpu_state(vcpu,&host_os_sprs)))+msr=mfmsr();/* MSR may have been updated */switch_pmu_to_guest(vcpu,&host_os_sprs);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:05:38
Move the P9 guest/host register switching functions to the built-in
P9 entry code, and export it for nested to use as well.
This allows more flexibility in scheduling these supervisor privileged
SPR accesses with the HV privileged and PR SPR accesses in the low level
entry code.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 365 +-------------------------
arch/powerpc/kvm/book3s_hv.h | 39 +++
arch/powerpc/kvm/book3s_hv_p9_entry.c | 345 ++++++++++++++++++++++++
3 files changed, 385 insertions(+), 364 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_hv.h
@@ -3772,370 +3773,6 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc)trace_kvmppc_run_core(vc,1);}-/*-*Privileged(non-hypervisor)hostregisterstosave.-*/-structp9_host_os_sprs{-unsignedlongdscr;-unsignedlongtidr;-unsignedlongiamr;-unsignedlongamr;-unsignedlongfscr;--unsignedintpmc1;-unsignedintpmc2;-unsignedintpmc3;-unsignedintpmc4;-unsignedintpmc5;-unsignedintpmc6;-unsignedlongmmcr0;-unsignedlongmmcr1;-unsignedlongmmcr2;-unsignedlongmmcr3;-unsignedlongmmcra;-unsignedlongsiar;-unsignedlongsier1;-unsignedlongsier2;-unsignedlongsier3;-unsignedlongsdar;-};--staticvoidfreeze_pmu(unsignedlongmmcr0,unsignedlongmmcra)-{-if(!(mmcr0&MMCR0_FC))-gotodo_freeze;-if(mmcra&MMCRA_SAMPLE_ENABLE)-gotodo_freeze;-if(cpu_has_feature(CPU_FTR_ARCH_31)){-if(!(mmcr0&MMCR0_PMCCEXT))-gotodo_freeze;-if(!(mmcra&MMCRA_BHRB_DISABLE))-gotodo_freeze;-}-return;--do_freeze:-mmcr0=MMCR0_FC;-mmcra=0;-if(cpu_has_feature(CPU_FTR_ARCH_31)){-mmcr0|=MMCR0_PMCCEXT;-mmcra=MMCRA_BHRB_DISABLE;-}--mtspr(SPRN_MMCR0,mmcr0);-mtspr(SPRN_MMCRA,mmcra);-isync();-}--staticvoidswitch_pmu_to_guest(structkvm_vcpu*vcpu,-structp9_host_os_sprs*host_os_sprs)-{-structlppaca*lp;-intload_pmu=1;--lp=vcpu->arch.vpa.pinned_addr;-if(lp)-load_pmu=lp->pmcregs_in_use;--/* Save host */-if(ppc_get_pmu_inuse()){-/*-*ItmightbebettertoputPMUhandling(atleastforthe-*host)intheperfsubsystembecauseitknowsmoreaboutwhat-*isbeingused.-*/--/* POWER9, POWER10 do not implement HPMC or SPMC */--host_os_sprs->mmcr0=mfspr(SPRN_MMCR0);-host_os_sprs->mmcra=mfspr(SPRN_MMCRA);--freeze_pmu(host_os_sprs->mmcr0,host_os_sprs->mmcra);--host_os_sprs->pmc1=mfspr(SPRN_PMC1);-host_os_sprs->pmc2=mfspr(SPRN_PMC2);-host_os_sprs->pmc3=mfspr(SPRN_PMC3);-host_os_sprs->pmc4=mfspr(SPRN_PMC4);-host_os_sprs->pmc5=mfspr(SPRN_PMC5);-host_os_sprs->pmc6=mfspr(SPRN_PMC6);-host_os_sprs->mmcr1=mfspr(SPRN_MMCR1);-host_os_sprs->mmcr2=mfspr(SPRN_MMCR2);-host_os_sprs->sdar=mfspr(SPRN_SDAR);-host_os_sprs->siar=mfspr(SPRN_SIAR);-host_os_sprs->sier1=mfspr(SPRN_SIER);--if(cpu_has_feature(CPU_FTR_ARCH_31)){-host_os_sprs->mmcr3=mfspr(SPRN_MMCR3);-host_os_sprs->sier2=mfspr(SPRN_SIER2);-host_os_sprs->sier3=mfspr(SPRN_SIER3);-}-}--#ifdef CONFIG_PPC_PSERIES-/* After saving PMU, before loading guest PMU, flip pmcregs_in_use */-if(kvmhv_on_pseries()){-barrier();-get_lppaca()->pmcregs_in_use=load_pmu;-barrier();-}-#endif--/*-*Loadguest.IftheVPAsaidthePMCsarenotinusebuttheguest-*triedtoaccessthemanyway,HFSCR[PM]willbesetbytheHFAC-*faultsowecanmakeforwardprogress.-*/-if(load_pmu||(vcpu->arch.hfscr&HFSCR_PM)){-mtspr(SPRN_PMC1,vcpu->arch.pmc[0]);-mtspr(SPRN_PMC2,vcpu->arch.pmc[1]);-mtspr(SPRN_PMC3,vcpu->arch.pmc[2]);-mtspr(SPRN_PMC4,vcpu->arch.pmc[3]);-mtspr(SPRN_PMC5,vcpu->arch.pmc[4]);-mtspr(SPRN_PMC6,vcpu->arch.pmc[5]);-mtspr(SPRN_MMCR1,vcpu->arch.mmcr[1]);-mtspr(SPRN_MMCR2,vcpu->arch.mmcr[2]);-mtspr(SPRN_SDAR,vcpu->arch.sdar);-mtspr(SPRN_SIAR,vcpu->arch.siar);-mtspr(SPRN_SIER,vcpu->arch.sier[0]);--if(cpu_has_feature(CPU_FTR_ARCH_31)){-mtspr(SPRN_MMCR3,vcpu->arch.mmcr[3]);-mtspr(SPRN_SIER2,vcpu->arch.sier[1]);-mtspr(SPRN_SIER3,vcpu->arch.sier[2]);-}--/* Set MMCRA then MMCR0 last */-mtspr(SPRN_MMCRA,vcpu->arch.mmcra);-mtspr(SPRN_MMCR0,vcpu->arch.mmcr[0]);-/* No isync necessary because we're starting counters */--if(!vcpu->arch.nested&&-(vcpu->arch.hfscr_permitted&HFSCR_PM))-vcpu->arch.hfscr|=HFSCR_PM;-}-}--staticvoidswitch_pmu_to_host(structkvm_vcpu*vcpu,-structp9_host_os_sprs*host_os_sprs)-{-structlppaca*lp;-intsave_pmu=1;--lp=vcpu->arch.vpa.pinned_addr;-if(lp)-save_pmu=lp->pmcregs_in_use;--if(save_pmu){-vcpu->arch.mmcr[0]=mfspr(SPRN_MMCR0);-vcpu->arch.mmcra=mfspr(SPRN_MMCRA);--freeze_pmu(vcpu->arch.mmcr[0],vcpu->arch.mmcra);--vcpu->arch.pmc[0]=mfspr(SPRN_PMC1);-vcpu->arch.pmc[1]=mfspr(SPRN_PMC2);-vcpu->arch.pmc[2]=mfspr(SPRN_PMC3);-vcpu->arch.pmc[3]=mfspr(SPRN_PMC4);-vcpu->arch.pmc[4]=mfspr(SPRN_PMC5);-vcpu->arch.pmc[5]=mfspr(SPRN_PMC6);-vcpu->arch.mmcr[1]=mfspr(SPRN_MMCR1);-vcpu->arch.mmcr[2]=mfspr(SPRN_MMCR2);-vcpu->arch.sdar=mfspr(SPRN_SDAR);-vcpu->arch.siar=mfspr(SPRN_SIAR);-vcpu->arch.sier[0]=mfspr(SPRN_SIER);--if(cpu_has_feature(CPU_FTR_ARCH_31)){-vcpu->arch.mmcr[3]=mfspr(SPRN_MMCR3);-vcpu->arch.sier[1]=mfspr(SPRN_SIER2);-vcpu->arch.sier[2]=mfspr(SPRN_SIER3);-}--}elseif(vcpu->arch.hfscr&HFSCR_PM){-/*-*TheguestaccessedPMCSPRswithoutspecifyingtheyshould-*bepreserved,oritclearedpmcregs_in_useafterthelast-*access.Justensuretheyarefrozen.-*/-freeze_pmu(mfspr(SPRN_MMCR0),mfspr(SPRN_MMCRA));--/*-*Demand-faultPMUregisteraccessintheguest.-*-*Thisisusedtograbtheguest'sVPApmcregs_in_usevalue-*andreflectitintothehost'sVPAinthecaseofanested-*hypervisor.-*-*Italsoavoidshavingtozero-outSPRsaftereachguest-*exittoavoidside-channelswhen.-*-*Thisisclearedherewhenweexittheguest,solaterHFSCR-*interrupthandlingcanadditbacktoruntheguestwith-*PMenablednexttime.-*/-if(!vcpu->arch.nested)-vcpu->arch.hfscr&=~HFSCR_PM;-}/* otherwise the PMU should still be frozen */--#ifdef CONFIG_PPC_PSERIES-if(kvmhv_on_pseries()){-barrier();-get_lppaca()->pmcregs_in_use=ppc_get_pmu_inuse();-barrier();-}-#endif--if(ppc_get_pmu_inuse()){-mtspr(SPRN_PMC1,host_os_sprs->pmc1);-mtspr(SPRN_PMC2,host_os_sprs->pmc2);-mtspr(SPRN_PMC3,host_os_sprs->pmc3);-mtspr(SPRN_PMC4,host_os_sprs->pmc4);-mtspr(SPRN_PMC5,host_os_sprs->pmc5);-mtspr(SPRN_PMC6,host_os_sprs->pmc6);-mtspr(SPRN_MMCR1,host_os_sprs->mmcr1);-mtspr(SPRN_MMCR2,host_os_sprs->mmcr2);-mtspr(SPRN_SDAR,host_os_sprs->sdar);-mtspr(SPRN_SIAR,host_os_sprs->siar);-mtspr(SPRN_SIER,host_os_sprs->sier1);--if(cpu_has_feature(CPU_FTR_ARCH_31)){-mtspr(SPRN_MMCR3,host_os_sprs->mmcr3);-mtspr(SPRN_SIER2,host_os_sprs->sier2);-mtspr(SPRN_SIER3,host_os_sprs->sier3);-}--/* Set MMCRA then MMCR0 last */-mtspr(SPRN_MMCRA,host_os_sprs->mmcra);-mtspr(SPRN_MMCR0,host_os_sprs->mmcr0);-isync();-}-}--staticvoidload_spr_state(structkvm_vcpu*vcpu,-structp9_host_os_sprs*host_os_sprs)-{-mtspr(SPRN_TAR,vcpu->arch.tar);-mtspr(SPRN_EBBHR,vcpu->arch.ebbhr);-mtspr(SPRN_EBBRR,vcpu->arch.ebbrr);-mtspr(SPRN_BESCR,vcpu->arch.bescr);--if(!cpu_has_feature(CPU_FTR_ARCH_31))-mtspr(SPRN_TIDR,vcpu->arch.tid);-if(host_os_sprs->iamr!=vcpu->arch.iamr)-mtspr(SPRN_IAMR,vcpu->arch.iamr);-if(host_os_sprs->amr!=vcpu->arch.amr)-mtspr(SPRN_AMR,vcpu->arch.amr);-if(vcpu->arch.uamor!=0)-mtspr(SPRN_UAMOR,vcpu->arch.uamor);-if(host_os_sprs->fscr!=vcpu->arch.fscr)-mtspr(SPRN_FSCR,vcpu->arch.fscr);-if(host_os_sprs->dscr!=vcpu->arch.dscr)-mtspr(SPRN_DSCR,vcpu->arch.dscr);-if(vcpu->arch.pspb!=0)-mtspr(SPRN_PSPB,vcpu->arch.pspb);--/*-*DAR,DSISR,andfornestedHV,SPRGsmustbesetwithMSR[RI]-*clear(orhstatesetappropriatelytocatchthoseregisters-*beingclobberedifwetakeaMCEorSRESET),sothosearedone-*later.-*/--if(!(vcpu->arch.ctrl&1))-mtspr(SPRN_CTRLT,0);-}--staticvoidstore_spr_state(structkvm_vcpu*vcpu)-{-vcpu->arch.tar=mfspr(SPRN_TAR);-vcpu->arch.ebbhr=mfspr(SPRN_EBBHR);-vcpu->arch.ebbrr=mfspr(SPRN_EBBRR);-vcpu->arch.bescr=mfspr(SPRN_BESCR);--if(!cpu_has_feature(CPU_FTR_ARCH_31))-vcpu->arch.tid=mfspr(SPRN_TIDR);-vcpu->arch.iamr=mfspr(SPRN_IAMR);-vcpu->arch.amr=mfspr(SPRN_AMR);-vcpu->arch.uamor=mfspr(SPRN_UAMOR);-vcpu->arch.fscr=mfspr(SPRN_FSCR);-vcpu->arch.dscr=mfspr(SPRN_DSCR);-vcpu->arch.pspb=mfspr(SPRN_PSPB);--vcpu->arch.ctrl=mfspr(SPRN_CTRLF);-}--/* Returns true if current MSR and/or guest MSR may have changed */-staticboolload_vcpu_state(structkvm_vcpu*vcpu,-structp9_host_os_sprs*host_os_sprs)-{-boolret=false;--if(cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)){-kvmppc_restore_tm_hv(vcpu,vcpu->arch.shregs.msr,true);-ret=true;-}--load_spr_state(vcpu,host_os_sprs);--load_fp_state(&vcpu->arch.fp);-#ifdef CONFIG_ALTIVEC-load_vr_state(&vcpu->arch.vr);-#endif-mtspr(SPRN_VRSAVE,vcpu->arch.vrsave);--returnret;-}--staticvoidstore_vcpu_state(structkvm_vcpu*vcpu)-{-store_spr_state(vcpu);--store_fp_state(&vcpu->arch.fp);-#ifdef CONFIG_ALTIVEC-store_vr_state(&vcpu->arch.vr);-#endif-vcpu->arch.vrsave=mfspr(SPRN_VRSAVE);--if(cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))-kvmppc_save_tm_hv(vcpu,vcpu->arch.shregs.msr,true);-}--staticvoidsave_p9_host_os_sprs(structp9_host_os_sprs*host_os_sprs)-{-if(!cpu_has_feature(CPU_FTR_ARCH_31))-host_os_sprs->tidr=mfspr(SPRN_TIDR);-host_os_sprs->iamr=mfspr(SPRN_IAMR);-host_os_sprs->amr=mfspr(SPRN_AMR);-host_os_sprs->fscr=mfspr(SPRN_FSCR);-host_os_sprs->dscr=mfspr(SPRN_DSCR);-}--/* vcpu guest regs must already be saved */-staticvoidrestore_p9_host_os_sprs(structkvm_vcpu*vcpu,-structp9_host_os_sprs*host_os_sprs)-{-mtspr(SPRN_SPRG_VDSO_WRITE,local_paca->sprg_vdso);--if(!cpu_has_feature(CPU_FTR_ARCH_31))-mtspr(SPRN_TIDR,host_os_sprs->tidr);-if(host_os_sprs->iamr!=vcpu->arch.iamr)-mtspr(SPRN_IAMR,host_os_sprs->iamr);-if(vcpu->arch.uamor!=0)-mtspr(SPRN_UAMOR,0);-if(host_os_sprs->amr!=vcpu->arch.amr)-mtspr(SPRN_AMR,host_os_sprs->amr);-if(host_os_sprs->fscr!=vcpu->arch.fscr)-mtspr(SPRN_FSCR,host_os_sprs->fscr);-if(host_os_sprs->dscr!=vcpu->arch.dscr)-mtspr(SPRN_DSCR,host_os_sprs->dscr);-if(vcpu->arch.pspb!=0)-mtspr(SPRN_PSPB,0);--/* Save guest CTRL register, set runlatch to 1 */-if(!(vcpu->arch.ctrl&1))-mtspr(SPRN_CTRLT,1);-}-staticinlineboolhcall_is_xics(unsignedlongreq){returnreq==H_EOI||req==H_CPPR||req==H_IPI||
@@ -4,8 +4,353 @@#include<asm/asm-prototypes.h>#include<asm/dbell.h>#include<asm/kvm_ppc.h>+#include<asm/pmc.h>#include<asm/ppc-opcode.h>+#include"book3s_hv.h"++staticvoidfreeze_pmu(unsignedlongmmcr0,unsignedlongmmcra)+{+if(!(mmcr0&MMCR0_FC))+gotodo_freeze;+if(mmcra&MMCRA_SAMPLE_ENABLE)+gotodo_freeze;+if(cpu_has_feature(CPU_FTR_ARCH_31)){+if(!(mmcr0&MMCR0_PMCCEXT))+gotodo_freeze;+if(!(mmcra&MMCRA_BHRB_DISABLE))+gotodo_freeze;+}+return;++do_freeze:+mmcr0=MMCR0_FC;+mmcra=0;+if(cpu_has_feature(CPU_FTR_ARCH_31)){+mmcr0|=MMCR0_PMCCEXT;+mmcra=MMCRA_BHRB_DISABLE;+}++mtspr(SPRN_MMCR0,mmcr0);+mtspr(SPRN_MMCRA,mmcra);+isync();+}++voidswitch_pmu_to_guest(structkvm_vcpu*vcpu,+structp9_host_os_sprs*host_os_sprs)+{+structlppaca*lp;+intload_pmu=1;++lp=vcpu->arch.vpa.pinned_addr;+if(lp)+load_pmu=lp->pmcregs_in_use;++/* Save host */+if(ppc_get_pmu_inuse()){+/*+*ItmightbebettertoputPMUhandling(atleastforthe+*host)intheperfsubsystembecauseitknowsmoreaboutwhat+*isbeingused.+*/++/* POWER9, POWER10 do not implement HPMC or SPMC */++host_os_sprs->mmcr0=mfspr(SPRN_MMCR0);+host_os_sprs->mmcra=mfspr(SPRN_MMCRA);++freeze_pmu(host_os_sprs->mmcr0,host_os_sprs->mmcra);++host_os_sprs->pmc1=mfspr(SPRN_PMC1);+host_os_sprs->pmc2=mfspr(SPRN_PMC2);+host_os_sprs->pmc3=mfspr(SPRN_PMC3);+host_os_sprs->pmc4=mfspr(SPRN_PMC4);+host_os_sprs->pmc5=mfspr(SPRN_PMC5);+host_os_sprs->pmc6=mfspr(SPRN_PMC6);+host_os_sprs->mmcr1=mfspr(SPRN_MMCR1);+host_os_sprs->mmcr2=mfspr(SPRN_MMCR2);+host_os_sprs->sdar=mfspr(SPRN_SDAR);+host_os_sprs->siar=mfspr(SPRN_SIAR);+host_os_sprs->sier1=mfspr(SPRN_SIER);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+host_os_sprs->mmcr3=mfspr(SPRN_MMCR3);+host_os_sprs->sier2=mfspr(SPRN_SIER2);+host_os_sprs->sier3=mfspr(SPRN_SIER3);+}+}++#ifdef CONFIG_PPC_PSERIES+/* After saving PMU, before loading guest PMU, flip pmcregs_in_use */+if(kvmhv_on_pseries()){+barrier();+get_lppaca()->pmcregs_in_use=load_pmu;+barrier();+}+#endif++/*+*Loadguest.IftheVPAsaidthePMCsarenotinusebuttheguest+*triedtoaccessthemanyway,HFSCR[PM]willbesetbytheHFAC+*faultsowecanmakeforwardprogress.+*/+if(load_pmu||(vcpu->arch.hfscr&HFSCR_PM)){+mtspr(SPRN_PMC1,vcpu->arch.pmc[0]);+mtspr(SPRN_PMC2,vcpu->arch.pmc[1]);+mtspr(SPRN_PMC3,vcpu->arch.pmc[2]);+mtspr(SPRN_PMC4,vcpu->arch.pmc[3]);+mtspr(SPRN_PMC5,vcpu->arch.pmc[4]);+mtspr(SPRN_PMC6,vcpu->arch.pmc[5]);+mtspr(SPRN_MMCR1,vcpu->arch.mmcr[1]);+mtspr(SPRN_MMCR2,vcpu->arch.mmcr[2]);+mtspr(SPRN_SDAR,vcpu->arch.sdar);+mtspr(SPRN_SIAR,vcpu->arch.siar);+mtspr(SPRN_SIER,vcpu->arch.sier[0]);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+mtspr(SPRN_MMCR3,vcpu->arch.mmcr[3]);+mtspr(SPRN_SIER2,vcpu->arch.sier[1]);+mtspr(SPRN_SIER3,vcpu->arch.sier[2]);+}++/* Set MMCRA then MMCR0 last */+mtspr(SPRN_MMCRA,vcpu->arch.mmcra);+mtspr(SPRN_MMCR0,vcpu->arch.mmcr[0]);+/* No isync necessary because we're starting counters */++if(!vcpu->arch.nested&&+(vcpu->arch.hfscr_permitted&HFSCR_PM))+vcpu->arch.hfscr|=HFSCR_PM;+}+}+EXPORT_SYMBOL_GPL(switch_pmu_to_guest);++voidswitch_pmu_to_host(structkvm_vcpu*vcpu,+structp9_host_os_sprs*host_os_sprs)+{+structlppaca*lp;+intsave_pmu=1;++lp=vcpu->arch.vpa.pinned_addr;+if(lp)+save_pmu=lp->pmcregs_in_use;++if(save_pmu){+vcpu->arch.mmcr[0]=mfspr(SPRN_MMCR0);+vcpu->arch.mmcra=mfspr(SPRN_MMCRA);++freeze_pmu(vcpu->arch.mmcr[0],vcpu->arch.mmcra);++vcpu->arch.pmc[0]=mfspr(SPRN_PMC1);+vcpu->arch.pmc[1]=mfspr(SPRN_PMC2);+vcpu->arch.pmc[2]=mfspr(SPRN_PMC3);+vcpu->arch.pmc[3]=mfspr(SPRN_PMC4);+vcpu->arch.pmc[4]=mfspr(SPRN_PMC5);+vcpu->arch.pmc[5]=mfspr(SPRN_PMC6);+vcpu->arch.mmcr[1]=mfspr(SPRN_MMCR1);+vcpu->arch.mmcr[2]=mfspr(SPRN_MMCR2);+vcpu->arch.sdar=mfspr(SPRN_SDAR);+vcpu->arch.siar=mfspr(SPRN_SIAR);+vcpu->arch.sier[0]=mfspr(SPRN_SIER);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+vcpu->arch.mmcr[3]=mfspr(SPRN_MMCR3);+vcpu->arch.sier[1]=mfspr(SPRN_SIER2);+vcpu->arch.sier[2]=mfspr(SPRN_SIER3);+}++}elseif(vcpu->arch.hfscr&HFSCR_PM){+/*+*TheguestaccessedPMCSPRswithoutspecifyingtheyshould+*bepreserved,oritclearedpmcregs_in_useafterthelast+*access.Justensuretheyarefrozen.+*/+freeze_pmu(mfspr(SPRN_MMCR0),mfspr(SPRN_MMCRA));++/*+*Demand-faultPMUregisteraccessintheguest.+*+*Thisisusedtograbtheguest'sVPApmcregs_in_usevalue+*andreflectitintothehost'sVPAinthecaseofanested+*hypervisor.+*+*Italsoavoidshavingtozero-outSPRsaftereachguest+*exittoavoidside-channelswhen.+*+*Thisisclearedherewhenweexittheguest,solaterHFSCR+*interrupthandlingcanadditbacktoruntheguestwith+*PMenablednexttime.+*/+if(!vcpu->arch.nested)+vcpu->arch.hfscr&=~HFSCR_PM;+}/* otherwise the PMU should still be frozen */++#ifdef CONFIG_PPC_PSERIES+if(kvmhv_on_pseries()){+barrier();+get_lppaca()->pmcregs_in_use=ppc_get_pmu_inuse();+barrier();+}+#endif++if(ppc_get_pmu_inuse()){+mtspr(SPRN_PMC1,host_os_sprs->pmc1);+mtspr(SPRN_PMC2,host_os_sprs->pmc2);+mtspr(SPRN_PMC3,host_os_sprs->pmc3);+mtspr(SPRN_PMC4,host_os_sprs->pmc4);+mtspr(SPRN_PMC5,host_os_sprs->pmc5);+mtspr(SPRN_PMC6,host_os_sprs->pmc6);+mtspr(SPRN_MMCR1,host_os_sprs->mmcr1);+mtspr(SPRN_MMCR2,host_os_sprs->mmcr2);+mtspr(SPRN_SDAR,host_os_sprs->sdar);+mtspr(SPRN_SIAR,host_os_sprs->siar);+mtspr(SPRN_SIER,host_os_sprs->sier1);++if(cpu_has_feature(CPU_FTR_ARCH_31)){+mtspr(SPRN_MMCR3,host_os_sprs->mmcr3);+mtspr(SPRN_SIER2,host_os_sprs->sier2);+mtspr(SPRN_SIER3,host_os_sprs->sier3);+}++/* Set MMCRA then MMCR0 last */+mtspr(SPRN_MMCRA,host_os_sprs->mmcra);+mtspr(SPRN_MMCR0,host_os_sprs->mmcr0);+isync();+}+}+EXPORT_SYMBOL_GPL(switch_pmu_to_host);++staticvoidload_spr_state(structkvm_vcpu*vcpu,+structp9_host_os_sprs*host_os_sprs)+{+mtspr(SPRN_TAR,vcpu->arch.tar);+mtspr(SPRN_EBBHR,vcpu->arch.ebbhr);+mtspr(SPRN_EBBRR,vcpu->arch.ebbrr);+mtspr(SPRN_BESCR,vcpu->arch.bescr);++if(!cpu_has_feature(CPU_FTR_ARCH_31))+mtspr(SPRN_TIDR,vcpu->arch.tid);+if(host_os_sprs->iamr!=vcpu->arch.iamr)+mtspr(SPRN_IAMR,vcpu->arch.iamr);+if(host_os_sprs->amr!=vcpu->arch.amr)+mtspr(SPRN_AMR,vcpu->arch.amr);+if(vcpu->arch.uamor!=0)+mtspr(SPRN_UAMOR,vcpu->arch.uamor);+if(host_os_sprs->fscr!=vcpu->arch.fscr)+mtspr(SPRN_FSCR,vcpu->arch.fscr);+if(host_os_sprs->dscr!=vcpu->arch.dscr)+mtspr(SPRN_DSCR,vcpu->arch.dscr);+if(vcpu->arch.pspb!=0)+mtspr(SPRN_PSPB,vcpu->arch.pspb);++/*+*DAR,DSISR,andfornestedHV,SPRGsmustbesetwithMSR[RI]+*clear(orhstatesetappropriatelytocatchthoseregisters+*beingclobberedifwetakeaMCEorSRESET),sothosearedone+*later.+*/++if(!(vcpu->arch.ctrl&1))+mtspr(SPRN_CTRLT,0);+}++staticvoidstore_spr_state(structkvm_vcpu*vcpu)+{+vcpu->arch.tar=mfspr(SPRN_TAR);+vcpu->arch.ebbhr=mfspr(SPRN_EBBHR);+vcpu->arch.ebbrr=mfspr(SPRN_EBBRR);+vcpu->arch.bescr=mfspr(SPRN_BESCR);++if(!cpu_has_feature(CPU_FTR_ARCH_31))+vcpu->arch.tid=mfspr(SPRN_TIDR);+vcpu->arch.iamr=mfspr(SPRN_IAMR);+vcpu->arch.amr=mfspr(SPRN_AMR);+vcpu->arch.uamor=mfspr(SPRN_UAMOR);+vcpu->arch.fscr=mfspr(SPRN_FSCR);+vcpu->arch.dscr=mfspr(SPRN_DSCR);+vcpu->arch.pspb=mfspr(SPRN_PSPB);++vcpu->arch.ctrl=mfspr(SPRN_CTRLF);+}++/* Returns true if current MSR and/or guest MSR may have changed */+boolload_vcpu_state(structkvm_vcpu*vcpu,+structp9_host_os_sprs*host_os_sprs)+{+boolret=false;++if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)){+kvmppc_restore_tm_hv(vcpu,vcpu->arch.shregs.msr,true);+ret=true;+}++load_spr_state(vcpu,host_os_sprs);++load_fp_state(&vcpu->arch.fp);+#ifdef CONFIG_ALTIVEC+load_vr_state(&vcpu->arch.vr);+#endif+mtspr(SPRN_VRSAVE,vcpu->arch.vrsave);++returnret;+}+EXPORT_SYMBOL_GPL(load_vcpu_state);++voidstore_vcpu_state(structkvm_vcpu*vcpu)+{+store_spr_state(vcpu);++store_fp_state(&vcpu->arch.fp);+#ifdef CONFIG_ALTIVEC+store_vr_state(&vcpu->arch.vr);+#endif+vcpu->arch.vrsave=mfspr(SPRN_VRSAVE);++if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+kvmppc_save_tm_hv(vcpu,vcpu->arch.shregs.msr,true);+}+EXPORT_SYMBOL_GPL(store_vcpu_state);++voidsave_p9_host_os_sprs(structp9_host_os_sprs*host_os_sprs)+{+if(!cpu_has_feature(CPU_FTR_ARCH_31))+host_os_sprs->tidr=mfspr(SPRN_TIDR);+host_os_sprs->iamr=mfspr(SPRN_IAMR);+host_os_sprs->amr=mfspr(SPRN_AMR);+host_os_sprs->fscr=mfspr(SPRN_FSCR);+host_os_sprs->dscr=mfspr(SPRN_DSCR);+}+EXPORT_SYMBOL_GPL(save_p9_host_os_sprs);++/* vcpu guest regs must already be saved */+voidrestore_p9_host_os_sprs(structkvm_vcpu*vcpu,+structp9_host_os_sprs*host_os_sprs)+{+mtspr(SPRN_SPRG_VDSO_WRITE,local_paca->sprg_vdso);++if(!cpu_has_feature(CPU_FTR_ARCH_31))+mtspr(SPRN_TIDR,host_os_sprs->tidr);+if(host_os_sprs->iamr!=vcpu->arch.iamr)+mtspr(SPRN_IAMR,host_os_sprs->iamr);+if(vcpu->arch.uamor!=0)+mtspr(SPRN_UAMOR,0);+if(host_os_sprs->amr!=vcpu->arch.amr)+mtspr(SPRN_AMR,host_os_sprs->amr);+if(host_os_sprs->fscr!=vcpu->arch.fscr)+mtspr(SPRN_FSCR,host_os_sprs->fscr);+if(host_os_sprs->dscr!=vcpu->arch.dscr)+mtspr(SPRN_DSCR,host_os_sprs->dscr);+if(vcpu->arch.pspb!=0)+mtspr(SPRN_PSPB,0);++/* Save guest CTRL register, set runlatch to 1 */+if(!(vcpu->arch.ctrl&1))+mtspr(SPRN_CTRLT,1);+}+EXPORT_SYMBOL_GPL(restore_p9_host_os_sprs);+#ifdef CONFIG_KVM_BOOK3S_HV_EXIT_TIMINGstaticvoid__start_timing(structkvm_vcpu*vcpu,structkvmhv_tb_accumulator*next){
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:06:02
This is just refactoring.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 125 +++++++++++++++++++----------------
1 file changed, 67 insertions(+), 58 deletions(-)
@@ -3789,6 +3789,72 @@ static void vcpu_vpa_increment_dispatch(struct kvm_vcpu *vcpu)}}+/* call our hypervisor to load up HV regs and go */+staticintkvmhv_vcpu_entry_p9_nested(structkvm_vcpu*vcpu,u64time_limit,unsignedlonglpcr,u64*tb)+{+structkvmppc_vcore*vc=vcpu->arch.vcore;+unsignedlonghost_psscr;+structhv_guest_statehvregs;+inttrap;+s64dec;++/*+*Weneedtosaveandrestoretheguestvisiblepartofthe+*psscr(i.e.usingSPRN_PSSCR_PR)sincethehypervisor+*doesn'tdothisforus.Noteonlyrequiredifpseriessince+*thisisdoneinkvmhv_vcpu_entry_p9()belowotherwise.+*/+host_psscr=mfspr(SPRN_PSSCR_PR);+mtspr(SPRN_PSSCR_PR,vcpu->arch.psscr);+kvmhv_save_hv_regs(vcpu,&hvregs);+hvregs.lpcr=lpcr;+vcpu->arch.regs.msr=vcpu->arch.shregs.msr;+hvregs.version=HV_GUEST_STATE_VERSION;+if(vcpu->arch.nested){+hvregs.lpid=vcpu->arch.nested->shadow_lpid;+hvregs.vcpu_token=vcpu->arch.nested_vcpu_id;+}else{+hvregs.lpid=vcpu->kvm->arch.lpid;+hvregs.vcpu_token=vcpu->vcpu_id;+}+hvregs.hdec_expiry=time_limit;++/*+*WhensettingDEC,wemustalwaysdealwithirq_work_raise+*viaNMIvssettingDEC.Theproblemoccursrightaswe+*switchintoguestmodeifaNMIhitsandsetspendingwork+*andsetsDEC,thenthatwillapplytotheguestandnot+*bringusbacktothehost.+*+*irq_work_raisecouldcheckaflag(orpossiblyLPCR[HDICE]+*forexample)andsetHDECto1?Thatwouldn'tsolvethe+*nestedhvcasewhichneedstoabortthehcallorzerothe+*timelimit.+*+*XXX:Anotherday'sproblem.+*/+mtspr(SPRN_DEC,kvmppc_dec_expires_host_tb(vcpu)-*tb);++mtspr(SPRN_DAR,vcpu->arch.shregs.dar);+mtspr(SPRN_DSISR,vcpu->arch.shregs.dsisr);+trap=plpar_hcall_norets(H_ENTER_NESTED,__pa(&hvregs),+__pa(&vcpu->arch.regs));+kvmhv_restore_hv_return_state(vcpu,&hvregs);+vcpu->arch.shregs.msr=vcpu->arch.regs.msr;+vcpu->arch.shregs.dar=mfspr(SPRN_DAR);+vcpu->arch.shregs.dsisr=mfspr(SPRN_DSISR);+vcpu->arch.psscr=mfspr(SPRN_PSSCR_PR);+mtspr(SPRN_PSSCR_PR,host_psscr);++dec=mfspr(SPRN_DEC);+if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */+dec=(s32)dec;+*tb=mftb();+vcpu->arch.dec_expires=dec+(*tb+vc->tb_offset);++returntrap;+}+/**GuestentryforPOWER9andlaterCPUs.*/
@@ -3850,63 +3915,7 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,switch_pmu_to_guest(vcpu,&host_os_sprs);if(kvmhv_on_pseries()){-/*-*Weneedtosaveandrestoretheguestvisiblepartofthe-*psscr(i.e.usingSPRN_PSSCR_PR)sincethehypervisor-*doesn'tdothisforus.Noteonlyrequiredifpseriessince-*thisisdoneinkvmhv_vcpu_entry_p9()belowotherwise.-*/-unsignedlonghost_psscr;-/* call our hypervisor to load up HV regs and go */-structhv_guest_statehvregs;--host_psscr=mfspr(SPRN_PSSCR_PR);-mtspr(SPRN_PSSCR_PR,vcpu->arch.psscr);-kvmhv_save_hv_regs(vcpu,&hvregs);-hvregs.lpcr=lpcr;-vcpu->arch.regs.msr=vcpu->arch.shregs.msr;-hvregs.version=HV_GUEST_STATE_VERSION;-if(vcpu->arch.nested){-hvregs.lpid=vcpu->arch.nested->shadow_lpid;-hvregs.vcpu_token=vcpu->arch.nested_vcpu_id;-}else{-hvregs.lpid=vcpu->kvm->arch.lpid;-hvregs.vcpu_token=vcpu->vcpu_id;-}-hvregs.hdec_expiry=time_limit;--/*-*WhensettingDEC,wemustalwaysdealwithirq_work_raise-*viaNMIvssettingDEC.Theproblemoccursrightaswe-*switchintoguestmodeifaNMIhitsandsetspendingwork-*andsetsDEC,thenthatwillapplytotheguestandnot-*bringusbacktothehost.-*-*irq_work_raisecouldcheckaflag(orpossiblyLPCR[HDICE]-*forexample)andsetHDECto1?Thatwouldn'tsolvethe-*nestedhvcasewhichneedstoabortthehcallorzerothe-*timelimit.-*-*XXX:Anotherday'sproblem.-*/-mtspr(SPRN_DEC,kvmppc_dec_expires_host_tb(vcpu)-*tb);--mtspr(SPRN_DAR,vcpu->arch.shregs.dar);-mtspr(SPRN_DSISR,vcpu->arch.shregs.dsisr);-trap=plpar_hcall_norets(H_ENTER_NESTED,__pa(&hvregs),-__pa(&vcpu->arch.regs));-kvmhv_restore_hv_return_state(vcpu,&hvregs);-vcpu->arch.shregs.msr=vcpu->arch.regs.msr;-vcpu->arch.shregs.dar=mfspr(SPRN_DAR);-vcpu->arch.shregs.dsisr=mfspr(SPRN_DSISR);-vcpu->arch.psscr=mfspr(SPRN_PSSCR_PR);-mtspr(SPRN_PSSCR_PR,host_psscr);--dec=mfspr(SPRN_DEC);-if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */-dec=(s32)dec;-*tb=mftb();-vcpu->arch.dec_expires=dec+(*tb+vc->tb_offset);+trap=kvmhv_vcpu_entry_p9_nested(vcpu,time_limit,lpcr,tb);/* H_CEDE has to be handled now, not later */if(trap==BOOK3S_INTERRUPT_SYSCALL&&!vcpu->arch.nested&&
@@ -3805,6 +3811,27 @@ static int kvmhv_vcpu_entry_p9_nested(struct kvm_vcpu *vcpu, u64 time_limit, uns*thisisdoneinkvmhv_vcpu_entry_p9()belowotherwise.*/host_psscr=mfspr(SPRN_PSSCR_PR);++hard_irq_disable();+if(lazy_irq_pending())+return0;++/* MSR bits may have been cleared by context switch */+msr=0;+if(IS_ENABLED(CONFIG_PPC_FPU))+msr|=MSR_FP;+if(cpu_has_feature(CPU_FTR_ALTIVEC))+msr|=MSR_VEC;+if(cpu_has_feature(CPU_FTR_VSX))+msr|=MSR_VSX;+if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+msr|=MSR_TM;+msr=msr_check_and_set(msr);++if(unlikely(load_vcpu_state(vcpu,&host_os_sprs)))+msr=mfmsr();/* TM restore can update msr */+mtspr(SPRN_PSSCR_PR,vcpu->arch.psscr);kvmhv_save_hv_regs(vcpu,&hvregs);hvregs.lpcr=lpcr;
@@ -3846,12 +3873,20 @@ static int kvmhv_vcpu_entry_p9_nested(struct kvm_vcpu *vcpu, u64 time_limit, unsvcpu->arch.psscr=mfspr(SPRN_PSSCR_PR);mtspr(SPRN_PSSCR_PR,host_psscr);+store_vcpu_state(vcpu);+dec=mfspr(SPRN_DEC);if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */dec=(s32)dec;*tb=mftb();vcpu->arch.dec_expires=dec+(*tb+vc->tb_offset);+timer_rearm_host_dec(*tb);++restore_p9_host_os_sprs(vcpu,&host_os_sprs);++switch_pmu_to_host(vcpu,&host_os_sprs);+returntrap;}
@@ -3875,33 +3908,6 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu->arch.ceded=0;-save_p9_host_os_sprs(&host_os_sprs);--/*-*ThiscouldbecombinedwithMSR[RI]clearing,butthatexpands-*theunrecoverablewindow.Itwouldbebettertocoverunrecoverable-*withKVMbadinterrupthandlingratherthanuseMSR[RI]atall.-*-*MuchmoredifficultandlessworthwhiletocombinewithIR/DR-*disable.-*/-hard_irq_disable();-if(lazy_irq_pending())-return0;--/* MSR bits may have been cleared by context switch */-msr=0;-if(IS_ENABLED(CONFIG_PPC_FPU))-msr|=MSR_FP;-if(cpu_has_feature(CPU_FTR_ALTIVEC))-msr|=MSR_VEC;-if(cpu_has_feature(CPU_FTR_VSX))-msr|=MSR_VSX;-if(cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))-msr|=MSR_TM;-msr=msr_check_and_set(msr);-kvmppc_subcore_enter_guest();vc->entry_exit_map=1;
@@ -3909,11 +3915,6 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu_vpa_increment_dispatch(vcpu);-if(unlikely(load_vcpu_state(vcpu,&host_os_sprs)))-msr=mfmsr();/* MSR may have been updated */--switch_pmu_to_guest(vcpu,&host_os_sprs);-if(kvmhv_on_pseries()){trap=kvmhv_vcpu_entry_p9_nested(vcpu,time_limit,lpcr,tb);
@@ -559,9 +560,6 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvcpu->arch.ceded=0;-/* Could avoid mfmsr by passing around, but probably no big deal */-msr=mfmsr();-host_hfscr=mfspr(SPRN_HFSCR);host_ciabr=mfspr(SPRN_CIABR);host_dawr0=mfspr(SPRN_DAWR0);
@@ -576,6 +574,41 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpclocal_paca->kvm_hstate.host_purr=mfspr(SPRN_PURR);local_paca->kvm_hstate.host_spurr=mfspr(SPRN_SPURR);+switch_pmu_to_guest(vcpu,&host_os_sprs);++save_p9_host_os_sprs(&host_os_sprs);++/*+*ThiscouldbecombinedwithMSR[RI]clearing,butthatexpands+*theunrecoverablewindow.Itwouldbebettertocoverunrecoverable+*withKVMbadinterrupthandlingratherthanuseMSR[RI]atall.+*+*MuchmoredifficultandlessworthwhiletocombinewithIR/DR+*disable.+*/+hard_irq_disable();+if(lazy_irq_pending()){+trap=0;+gotoout;+}++/* MSR bits may have been cleared by context switch */+msr=0;+if(IS_ENABLED(CONFIG_PPC_FPU))+msr|=MSR_FP;+if(cpu_has_feature(CPU_FTR_ALTIVEC))+msr|=MSR_VEC;+if(cpu_has_feature(CPU_FTR_VSX))+msr|=MSR_VSX;+if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+msr|=MSR_TM;+msr=msr_check_and_set(msr);+/* Save MSR for restore. This is after hard disable, so EE is clear. */++if(unlikely(load_vcpu_state(vcpu,&host_os_sprs)))+msr=mfmsr();/* MSR may have been updated */+if(vc->tb_offset){u64new_tb=*tb+vc->tb_offset;mtspr(SPRN_TBU40,new_tb);
@@ -634,6 +667,14 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcmtspr(SPRN_SPRG2,vcpu->arch.shregs.sprg2);mtspr(SPRN_SPRG3,vcpu->arch.shregs.sprg3);+/*+*Itmightbepreferabletoload_vcpu_statehere,inordertogetthe+*GPR/FPregisterloadsexecutinginparallelwiththepreviousmtSPR+*instructions,butfornowthatcan'tbedonebecausetheTMhandling+*inload_vcpu_statecanchangesomeSPRsandvcpustate(nip,msr).+*ButTMcouldbesplitoutifthiswouldbeasignificantbenefit.+*/+local_paca->kvm_hstate.in_guest=KVM_GUEST_MODE_HV_P9;/*
@@ -811,6 +852,20 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvc->dpdes=mfspr(SPRN_DPDES);vc->vtb=mfspr(SPRN_VTB);+save_clear_guest_mmu(kvm,vcpu);+switch_mmu_to_host(kvm,host_pidr);++/*+*Ifweareinrealmode,onlyswitchMMUonaftertheMMUis+*switchedtohost,toavoidtheP9_RADIX_PREFETCH_BUG.+*/+if(IS_ENABLED(CONFIG_PPC_TRANSACTIONAL_MEM)&&+vcpu->arch.shregs.msr&MSR_TS_MASK)+msr|=MSR_TS_S;+__mtmsrd(msr,0);++store_vcpu_state(vcpu);+dec=mfspr(SPRN_DEC);if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */dec=(s32)dec;
@@ -843,6 +898,19 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcmtspr(SPRN_DAWRX1,host_dawrx1);}+mtspr(SPRN_DPDES,0);+if(vc->pcr)+mtspr(SPRN_PCR,PCR_MASK);++/* HDEC must be at least as large as DEC, so decrementer_max fits */+mtspr(SPRN_HDEC,decrementer_max);++timer_rearm_host_dec(*tb);++restore_p9_host_os_sprs(vcpu,&host_os_sprs);++local_paca->kvm_hstate.in_guest=KVM_GUEST_MODE_NONE;+if(kvm_is_radix(kvm)){/**Sincethisisradix,doaeieio;tlbsync;ptesyncsequence
@@ -859,26 +927,8 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcif(cpu_has_feature(CPU_FTR_ARCH_31))asmvolatile(PPC_CP_ABORT);-mtspr(SPRN_DPDES,0);-if(vc->pcr)-mtspr(SPRN_PCR,PCR_MASK);--/* HDEC must be at least as large as DEC, so decrementer_max fits */-mtspr(SPRN_HDEC,decrementer_max);--save_clear_guest_mmu(kvm,vcpu);-switch_mmu_to_host(kvm,host_pidr);-local_paca->kvm_hstate.in_guest=KVM_GUEST_MODE_NONE;--/*-*Ifweareinrealmode,onlyswitchMMUonaftertheMMUis-*switchedtohost,toavoidtheP9_RADIX_PREFETCH_BUG.-*/-if(IS_ENABLED(CONFIG_PPC_TRANSACTIONAL_MEM)&&-vcpu->arch.shregs.msr&MSR_TS_MASK)-msr|=MSR_TS_S;--__mtmsrd(msr,0);+out:+switch_pmu_to_host(vcpu,&host_os_sprs);end_timing(vcpu);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:06:58
If TM is not active, only TM register state needs to be saved.
-348 cycles (7218) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv_p9_entry.c | 23 +++++++++++++++++++----
1 file changed, 19 insertions(+), 4 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:07:22
This moves PMU switch to guest as late as possible in entry, and switch
back to host as early as possible at exit. This helps the host get the
most perf coverage of KVM entry/exit code as possible.
This is slightly suboptimal for SPR scheduling point of view when the
PMU is enabled, but when perf is disabled there is no real difference.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 6 ++----
arch/powerpc/kvm/book3s_hv_p9_entry.c | 6 ++----
2 files changed, 4 insertions(+), 8 deletions(-)
@@ -1598,7 +1598,8 @@ XXX benchmark guest exitsunsignedlongvsid;longerr;-if(vcpu->arch.fault_dsisr==HDSISR_CANARY){+if(cpu_has_feature(CPU_FTR_P9_RADIX_PREFETCH_BUG)&&+unlikely(vcpu->arch.fault_dsisr==HDSISR_CANARY)){r=RESUME_GUEST;/* Just retry if it's the canary */break;}
@@ -633,24 +633,29 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvc->tb_offset_applied=vc->tb_offset;}-if(vc->pcr)-mtspr(SPRN_PCR,vc->pcr|PCR_MASK);-mtspr(SPRN_DPDES,vc->dpdes);mtspr(SPRN_VTB,vc->vtb);-mtspr(SPRN_PURR,vcpu->arch.purr);mtspr(SPRN_SPURR,vcpu->arch.spurr);+if(vc->pcr)+mtspr(SPRN_PCR,vc->pcr|PCR_MASK);+if(vc->dpdes)+mtspr(SPRN_DPDES,vc->dpdes);+if(dawr_enabled()){-mtspr(SPRN_DAWR0,vcpu->arch.dawr0);-mtspr(SPRN_DAWRX0,vcpu->arch.dawrx0);+if(vcpu->arch.dawr0!=host_dawr0)+mtspr(SPRN_DAWR0,vcpu->arch.dawr0);+if(vcpu->arch.dawrx0!=host_dawrx0)+mtspr(SPRN_DAWRX0,vcpu->arch.dawrx0);if(cpu_has_feature(CPU_FTR_DAWR1)){-mtspr(SPRN_DAWR1,vcpu->arch.dawr1);-mtspr(SPRN_DAWRX1,vcpu->arch.dawrx1);+if(vcpu->arch.dawr1!=host_dawr1)+mtspr(SPRN_DAWR1,vcpu->arch.dawr1);+if(vcpu->arch.dawrx1!=host_dawrx1)+mtspr(SPRN_DAWRX1,vcpu->arch.dawrx1);}}-mtspr(SPRN_CIABR,vcpu->arch.ciabr);-mtspr(SPRN_IC,vcpu->arch.ic);+if(vcpu->arch.ciabr!=host_ciabr)+mtspr(SPRN_CIABR,vcpu->arch.ciabr);mtspr(SPRN_PSSCR,vcpu->arch.psscr|PSSCR_EC|(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));
@@ -869,20 +874,6 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvc->dpdes=mfspr(SPRN_DPDES);vc->vtb=mfspr(SPRN_VTB);-save_clear_guest_mmu(kvm,vcpu);-switch_mmu_to_host(kvm,host_pidr);--/*-*Ifweareinrealmode,onlyswitchMMUonaftertheMMUis-*switchedtohost,toavoidtheP9_RADIX_PREFETCH_BUG.-*/-if(IS_ENABLED(CONFIG_PPC_TRANSACTIONAL_MEM)&&-vcpu->arch.shregs.msr&MSR_TS_MASK)-msr|=MSR_TS_S;-__mtmsrd(msr,0);--store_vcpu_state(vcpu);-dec=mfspr(SPRN_DEC);if(!(lpcr&LPCR_LD))/* Sign extend if not using large decrementer */dec=(s32)dec;
@@ -900,6 +891,22 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvc->tb_offset_applied=0;}+save_clear_guest_mmu(kvm,vcpu);+switch_mmu_to_host(kvm,host_pidr);++/*+*EnableMSRhereinordertohavefacilitiesenabledtosave+*guestregisters.ThisenablesMMU(ifwewereinrealmode),so+*onlyswitchMMUonaftertheMMUisswitchedtohost,toavoid+*theP9_RADIX_PREFETCH_BUGorhashguestcontext.+*/+if(IS_ENABLED(CONFIG_PPC_TRANSACTIONAL_MEM)&&+vcpu->arch.shregs.msr&MSR_TS_MASK)+msr|=MSR_TS_S;+__mtmsrd(msr,0);++store_vcpu_state(vcpu);+mtspr(SPRN_PURR,local_paca->kvm_hstate.host_purr);mtspr(SPRN_SPURR,local_paca->kvm_hstate.host_spurr);
@@ -907,15 +914,21 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcmtspr(SPRN_PSSCR,host_psscr|(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));mtspr(SPRN_HFSCR,host_hfscr);-mtspr(SPRN_CIABR,host_ciabr);-mtspr(SPRN_DAWR0,host_dawr0);-mtspr(SPRN_DAWRX0,host_dawrx0);+if(vcpu->arch.ciabr!=host_ciabr)+mtspr(SPRN_CIABR,host_ciabr);+if(vcpu->arch.dawr0!=host_dawr0)+mtspr(SPRN_DAWR0,host_dawr0);+if(vcpu->arch.dawrx0!=host_dawrx0)+mtspr(SPRN_DAWRX0,host_dawrx0);if(cpu_has_feature(CPU_FTR_DAWR1)){-mtspr(SPRN_DAWR1,host_dawr1);-mtspr(SPRN_DAWRX1,host_dawrx1);+if(vcpu->arch.dawr1!=host_dawr1)+mtspr(SPRN_DAWR1,host_dawr1);+if(vcpu->arch.dawrx1!=host_dawrx1)+mtspr(SPRN_DAWRX1,host_dawrx1);}-mtspr(SPRN_DPDES,0);+if(vc->dpdes)+mtspr(SPRN_DPDES,0);if(vc->pcr)mtspr(SPRN_PCR,PCR_MASK);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:08:38
Use HFSCR facility disabling to implement demand faulting for EBB, with
a hysteresis counter similar to the load_fp etc counters in context
switching that implement the equivalent demand faulting for userspace
facilities.
This speeds up guest entry/exit by avoiding the register save/restore
when a guest is not frequently using them. When a guest does use them
often, there will be some additional demand fault overhead, but these
are not commonly used facilities.
Reviewed-by: Fabiano Rosas <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_hv.c | 16 +++++++++++++--
arch/powerpc/kvm/book3s_hv_p9_entry.c | 28 +++++++++++++++++++++------
3 files changed, 37 insertions(+), 8 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:09:02
Use HFSCR facility disabling to implement demand faulting for TM, with
a hysteresis counter similar to the load_fp etc counters in context
switching that implement the equivalent demand faulting for userspace
facilities.
This speeds up guest entry/exit by avoiding the register save/restore
when a guest is not frequently using them. When a guest does use them
often, there will be some additional demand fault overhead, but these
are not commonly used facilities.
-304 cycles (6681) POWER9 virt-mode NULL hcall with the previous patch
Reviewed-by: Fabiano Rosas <redacted>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_hv.c | 26 ++++++++++++++++++++------
arch/powerpc/kvm/book3s_hv_p9_entry.c | 25 +++++++++++++++++--------
3 files changed, 38 insertions(+), 14 deletions(-)
@@ -629,8 +637,9 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcmsr|=MSR_VEC;if(cpu_has_feature(CPU_FTR_VSX))msr|=MSR_VSX;-if(cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+if((cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))&&+(vcpu->arch.hfscr&HFSCR_TM))msr|=MSR_TM;msr=msr_check_and_set(msr);/* Save MSR for restore. This is after hard disable, so EE is clear. */
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:09:34
Linux implements SPR save/restore including storage space for registers
in the task struct for process context switching. Make use of this
similarly to the way we make use of the context switching fp/vec save
restore.
This improves code reuse, allows some stack space to be saved, and helps
with avoiding VRSAVE updates if they are not required.
-61 cycles (6620) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/switch_to.h | 2 +
arch/powerpc/kernel/process.c | 6 ++
arch/powerpc/kvm/book3s_hv.c | 21 +-----
arch/powerpc/kvm/book3s_hv.h | 3 -
arch/powerpc/kvm/book3s_hv_p9_entry.c | 93 +++++++++++++++++++--------
5 files changed, 74 insertions(+), 51 deletions(-)
@@ -4510,9 +4510,6 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)structkvm_run*run=vcpu->run;intr;intsrcu_idx;-unsignedlongebb_regs[3]={};/* shut up GCC */-unsignedlonguser_tar=0;-unsignedintuser_vrsave;structkvm*kvm;unsignedlongmsr;
@@ -4573,14 +4570,7 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)save_user_regs_kvm();-/* Save userspace EBB and other register values */-if(cpu_has_feature(CPU_FTR_ARCH_207S)){-ebb_regs[0]=mfspr(SPRN_EBBHR);-ebb_regs[1]=mfspr(SPRN_EBBRR);-ebb_regs[2]=mfspr(SPRN_BESCR);-user_tar=mfspr(SPRN_TAR);-}-user_vrsave=mfspr(SPRN_VRSAVE);+kvmppc_save_current_sprs();vcpu->arch.waitp=&vcpu->arch.vcore->wait;vcpu->arch.pgdir=kvm->mm->pgd;
@@ -4621,15 +4611,6 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)}}while(is_kvmppc_resume_guest(r));-/* Restore userspace EBB and other register values */-if(cpu_has_feature(CPU_FTR_ARCH_207S)){-mtspr(SPRN_EBBHR,ebb_regs[0]);-mtspr(SPRN_EBBRR,ebb_regs[1]);-mtspr(SPRN_BESCR,ebb_regs[2]);-mtspr(SPRN_TAR,user_tar);-}-mtspr(SPRN_VRSAVE,user_vrsave);-vcpu->arch.state=KVMPPC_VCPU_NOTREADY;atomic_dec(&kvm->arch.vcpus_running);
@@ -223,15 +223,26 @@ EXPORT_SYMBOL_GPL(switch_pmu_to_host);staticvoidload_spr_state(structkvm_vcpu*vcpu,structp9_host_os_sprs*host_os_sprs){+/* TAR is very fast */mtspr(SPRN_TAR,vcpu->arch.tar);+#ifdef CONFIG_ALTIVEC+if(cpu_has_feature(CPU_FTR_ALTIVEC)&&+current->thread.vrsave!=vcpu->arch.vrsave)+mtspr(SPRN_VRSAVE,vcpu->arch.vrsave);+#endif+if(vcpu->arch.hfscr&HFSCR_EBB){-mtspr(SPRN_EBBHR,vcpu->arch.ebbhr);-mtspr(SPRN_EBBRR,vcpu->arch.ebbrr);-mtspr(SPRN_BESCR,vcpu->arch.bescr);+if(current->thread.ebbhr!=vcpu->arch.ebbhr)+mtspr(SPRN_EBBHR,vcpu->arch.ebbhr);+if(current->thread.ebbrr!=vcpu->arch.ebbrr)+mtspr(SPRN_EBBRR,vcpu->arch.ebbrr);+if(current->thread.bescr!=vcpu->arch.bescr)+mtspr(SPRN_BESCR,vcpu->arch.bescr);}-if(!cpu_has_feature(CPU_FTR_ARCH_31))+if(!cpu_has_feature(CPU_FTR_ARCH_31)&&+current->thread.tidr!=vcpu->arch.tid)mtspr(SPRN_TIDR,vcpu->arch.tid);if(host_os_sprs->iamr!=vcpu->arch.iamr)mtspr(SPRN_IAMR,vcpu->arch.iamr);
@@ -367,26 +367,63 @@ EXPORT_SYMBOL_GPL(save_p9_host_os_sprs);voidrestore_p9_host_os_sprs(structkvm_vcpu*vcpu,structp9_host_os_sprs*host_os_sprs){+/*+*current->thread.xxxregistersmustallberestoredtohost+*valuesbeforeapotentialcontextswitch,othrewisethecontext+*switchitselfwilloverwritecurrent->thread.xxxwiththevalues+*fromtheguestSPRs.+*/+mtspr(SPRN_SPRG_VDSO_WRITE,local_paca->sprg_vdso);-if(!cpu_has_feature(CPU_FTR_ARCH_31))-mtspr(SPRN_TIDR,host_os_sprs->tidr);+if(!cpu_has_feature(CPU_FTR_ARCH_31)&&+current->thread.tidr!=vcpu->arch.tid)+mtspr(SPRN_TIDR,current->thread.tidr);if(host_os_sprs->iamr!=vcpu->arch.iamr)mtspr(SPRN_IAMR,host_os_sprs->iamr);if(vcpu->arch.uamor!=0)mtspr(SPRN_UAMOR,0);if(host_os_sprs->amr!=vcpu->arch.amr)mtspr(SPRN_AMR,host_os_sprs->amr);-if(host_os_sprs->fscr!=vcpu->arch.fscr)-mtspr(SPRN_FSCR,host_os_sprs->fscr);-if(host_os_sprs->dscr!=vcpu->arch.dscr)-mtspr(SPRN_DSCR,host_os_sprs->dscr);+if(current->thread.fscr!=vcpu->arch.fscr)+mtspr(SPRN_FSCR,current->thread.fscr);+if(current->thread.dscr!=vcpu->arch.dscr)+mtspr(SPRN_DSCR,current->thread.dscr);if(vcpu->arch.pspb!=0)mtspr(SPRN_PSPB,0);/* Save guest CTRL register, set runlatch to 1 */if(!(vcpu->arch.ctrl&1))mtspr(SPRN_CTRLT,1);++#ifdef CONFIG_ALTIVEC+if(cpu_has_feature(CPU_FTR_ALTIVEC)&&+vcpu->arch.vrsave!=current->thread.vrsave)+mtspr(SPRN_VRSAVE,current->thread.vrsave);+#endif+if(vcpu->arch.hfscr&HFSCR_EBB){+if(vcpu->arch.bescr!=current->thread.bescr)+mtspr(SPRN_BESCR,current->thread.bescr);+if(vcpu->arch.ebbhr!=current->thread.ebbhr)+mtspr(SPRN_EBBHR,current->thread.ebbhr);+if(vcpu->arch.ebbrr!=current->thread.ebbrr)+mtspr(SPRN_EBBRR,current->thread.ebbrr);++if(!vcpu->arch.nested){+/*+*Thisislikeload_fpincontextswitching,turnoff+*thefacilityafteritwrapstheu8totryavoiding+*savingandrestoringtheregisterseachpartition+*switch.+*/+vcpu->arch.load_ebb++;+if(!vcpu->arch.load_ebb)+vcpu->arch.hfscr&=~HFSCR_EBB;+}+}++if(vcpu->arch.tar!=current->thread.tar)+mtspr(SPRN_TAR,current->thread.tar);}EXPORT_SYMBOL_GPL(restore_p9_host_os_sprs);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:09:58
Tighten up partition switching code synchronisation and comments.
In particular, hwsync ; isync is required after the last access that is
performed in the context of a partition, before the partition is
switched away from.
-301 cycles (6319) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_64_mmu_radix.c | 4 +++
arch/powerpc/kvm/book3s_hv_p9_entry.c | 40 +++++++++++++++++++-------
2 files changed, 33 insertions(+), 11 deletions(-)
@@ -54,6 +54,8 @@ unsigned long __kvmhv_copy_tofrom_guest_radix(int lpid, int pid,preempt_disable();+asmvolatile("hwsync":::"memory");+isync();/* switch the lpid first to avoid running host with unallocated pid */old_lpid=mfspr(SPRN_LPID);if(old_lpid!=lpid)
@@ -70,6 +72,8 @@ unsigned long __kvmhv_copy_tofrom_guest_radix(int lpid, int pid,elseret=copy_to_user_nofault((void__user*)to,from,n);+asmvolatile("hwsync":::"memory");+isync();/* switch the pid first to avoid running host with unallocated pid */if(quadrant==1&&pid!=old_pid)mtspr(SPRN_PID,old_pid);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:10:27
Some of the DAWR SPR access is already predicated on dawr_enabled(),
apply this to the remainder of the accesses.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv_p9_entry.c | 34 ++++++++++++++++-----------
1 file changed, 20 insertions(+), 14 deletions(-)
@@ -639,6 +639,7 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcunsignedlonghost_dawr0;unsignedlonghost_dawrx0;unsignedlonghost_psscr;+unsignedlonghost_hpsscr;unsignedlonghost_pidr;unsignedlonghost_dawr1;unsignedlonghost_dawrx1;
@@ -656,7 +657,9 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpchost_hfscr=mfspr(SPRN_HFSCR);host_ciabr=mfspr(SPRN_CIABR);-host_psscr=mfspr(SPRN_PSSCR);+host_psscr=mfspr(SPRN_PSSCR_PR);+if(cpu_has_feature(CPU_FTRS_POWER9_DD2_2))+host_hpsscr=mfspr(SPRN_PSSCR);host_pidr=mfspr(SPRN_PID);if(dawr_enabled()){
@@ -740,8 +743,14 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcif(vcpu->arch.ciabr!=host_ciabr)mtspr(SPRN_CIABR,vcpu->arch.ciabr);-mtspr(SPRN_PSSCR,vcpu->arch.psscr|PSSCR_EC|-(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));++if(cpu_has_feature(CPU_FTRS_POWER9_DD2_2)){+mtspr(SPRN_PSSCR,vcpu->arch.psscr|PSSCR_EC|+(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));+}else{+if(vcpu->arch.psscr!=host_psscr)+mtspr(SPRN_PSSCR_PR,vcpu->arch.psscr);+}mtspr(SPRN_HFSCR,vcpu->arch.hfscr);
@@ -947,7 +956,7 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvcpu->arch.ic=mfspr(SPRN_IC);vcpu->arch.pid=mfspr(SPRN_PID);-vcpu->arch.psscr=mfspr(SPRN_PSSCR)&PSSCR_GUEST_VIS;+vcpu->arch.psscr=mfspr(SPRN_PSSCR_PR);vcpu->arch.shregs.sprg0=mfspr(SPRN_SPRG0);vcpu->arch.shregs.sprg1=mfspr(SPRN_SPRG1);
@@ -993,9 +1002,12 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcmtspr(SPRN_PURR,local_paca->kvm_hstate.host_purr);mtspr(SPRN_SPURR,local_paca->kvm_hstate.host_spurr);-/* Preserve PSSCR[FAKE_SUSPEND] until we've called kvmppc_save_tm_hv */-mtspr(SPRN_PSSCR,host_psscr|-(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));+if(cpu_has_feature(CPU_FTRS_POWER9_DD2_2)){+/* Preserve PSSCR[FAKE_SUSPEND] until we've called kvmppc_save_tm_hv */+mtspr(SPRN_PSSCR,host_hpsscr|+(local_paca->kvm_hstate.fake_suspend<<PSSCR_FAKE_SUSPEND_LG));+}+mtspr(SPRN_HFSCR,host_hfscr);if(vcpu->arch.ciabr!=host_ciabr)mtspr(SPRN_CIABR,host_ciabr);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:11:24
Use the existing TLB flushing logic to IPI the previous CPU and run the
necessary barriers before running a guest vCPU on a new physical CPU,
to do the necessary radix GTSE barriers for handling the case of an
interrupted guest tlbie sequence.
This results in more IPIs than the TLB flush logic requires, but it's
a significant win for common case scheduling when the vCPU remains on
the same physical CPU.
-522 cycles (5754) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 31 +++++++++++++++++++++++----
arch/powerpc/kvm/book3s_hv_p9_entry.c | 9 --------
2 files changed, 27 insertions(+), 13 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:11:49
mftb() is expensive and one can be avoided on nested guest dispatch.
If the time checking code distinguishes between the L0 timer and the
nested HV timer, then both can be tested in the same place with the
same mftb() value.
This also nicely illustrates the relationship between the L0 and nested
HV timers.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/kvm_asm.h | 1 +
arch/powerpc/kvm/book3s_hv.c | 12 ++++++++++++
arch/powerpc/kvm/book3s_hv_nested.c | 5 -----
3 files changed, 13 insertions(+), 5 deletions(-)
@@ -1491,6 +1491,10 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,run->ready_for_interrupt_injection=1;switch(vcpu->arch.trap){/* We're good on these - the host merely wanted to get our attention */+caseBOOK3S_INTERRUPT_NESTED_HV_DECREMENTER:+WARN_ON_ONCE(1);/* Should never happen */+vcpu->arch.trap=BOOK3S_INTERRUPT_HV_DECREMENTER;+fallthrough;caseBOOK3S_INTERRUPT_HV_DECREMENTER:vcpu->stat.dec_exits++;r=RESUME_GUEST;
@@ -1821,6 +1825,12 @@ static int kvmppc_handle_nested_exit(struct kvm_vcpu *vcpu)vcpu->stat.ext_intr_exits++;r=RESUME_GUEST;break;+/* These need to go to the nested HV */+caseBOOK3S_INTERRUPT_NESTED_HV_DECREMENTER:+vcpu->arch.trap=BOOK3S_INTERRUPT_HV_DECREMENTER;+vcpu->stat.dec_exits++;+r=RESUME_HOST;+break;/* SR/HMI/PMI are HV interrupts that host has handled. Resume guest.*/caseBOOK3S_INTERRUPT_HMI:caseBOOK3S_INTERRUPT_PERFMON:
@@ -3955,6 +3965,8 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,returnBOOK3S_INTERRUPT_HV_DECREMENTER;if(next_timer<time_limit)time_limit=next_timer;+elseif(*tb>=time_limit)/* nested time limit */+returnBOOK3S_INTERRUPT_NESTED_HV_DECREMENTER;vcpu->arch.ceded=0;
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:12:15
Rearrange the MSR saving on entry so it does not follow the mtmsrd to
disable interrupts, avoiding a possible RAW scoreboard stall.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 2 +
arch/powerpc/kvm/book3s_hv.c | 18 ++-----
arch/powerpc/kvm/book3s_hv_p9_entry.c | 66 +++++++++++++++---------
3 files changed, 47 insertions(+), 39 deletions(-)
@@ -3868,24 +3870,10 @@ static int kvmhv_vcpu_entry_p9_nested(struct kvm_vcpu *vcpu, u64 time_limit, uns*/host_psscr=mfspr(SPRN_PSSCR_PR);-hard_irq_disable();+kvmppc_msr_hard_disable_set_facilities(vcpu,msr);if(lazy_irq_pending())return0;-/* MSR bits may have been cleared by context switch */-msr=0;-if(IS_ENABLED(CONFIG_PPC_FPU))-msr|=MSR_FP;-if(cpu_has_feature(CPU_FTR_ALTIVEC))-msr|=MSR_VEC;-if(cpu_has_feature(CPU_FTR_VSX))-msr|=MSR_VSX;-if((cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))&&-(vcpu->arch.hfscr&HFSCR_TM))-msr|=MSR_TM;-msr=msr_check_and_set(msr);-if(unlikely(load_vcpu_state(vcpu,&host_os_sprs)))msr=mfmsr();/* TM restore can update msr */
@@ -622,6 +622,44 @@ static void save_clear_guest_mmu(struct kvm *kvm, struct kvm_vcpu *vcpu)}}+unsignedlongkvmppc_msr_hard_disable_set_facilities(structkvm_vcpu*vcpu,unsignedlongmsr)+{+unsignedlongmsr_needed=0;++msr&=~MSR_EE;++/* MSR bits may have been cleared by context switch so must recheck */+if(IS_ENABLED(CONFIG_PPC_FPU))+msr_needed|=MSR_FP;+if(cpu_has_feature(CPU_FTR_ALTIVEC))+msr_needed|=MSR_VEC;+if(cpu_has_feature(CPU_FTR_VSX))+msr_needed|=MSR_VSX;+if((cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))&&+(vcpu->arch.hfscr&HFSCR_TM))+msr_needed|=MSR_TM;++/*+*ThiscouldbecombinedwithMSR[RI]clearing,butthatexpands+*theunrecoverablewindow.Itwouldbebettertocoverunrecoverable+*withKVMbadinterrupthandlingratherthanuseMSR[RI]atall.+*+*MuchmoredifficultandlessworthwhiletocombinewithIR/DR+*disable.+*/+if((msr&msr_needed)!=msr_needed){+msr|=msr_needed;+__mtmsrd(msr,0);+}else{+__hard_irq_disable();+}+local_paca->irq_happened|=PACA_IRQ_HARD_DIS;++returnmsr;+}+EXPORT_SYMBOL_GPL(kvmppc_msr_hard_disable_set_facilities);+intkvmhv_vcpu_entry_p9(structkvm_vcpu*vcpu,u64time_limit,unsignedlonglpcr,u64*tb){structp9_host_os_sprshost_os_sprs;
@@ -655,6 +693,9 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcvcpu->arch.ceded=0;+/* Save MSR for restore, with EE clear. */+msr=mfmsr()&~MSR_EE;+host_hfscr=mfspr(SPRN_HFSCR);host_ciabr=mfspr(SPRN_CIABR);host_psscr=mfspr(SPRN_PSSCR_PR);
@@ -676,35 +717,12 @@ int kvmhv_vcpu_entry_p9(struct kvm_vcpu *vcpu, u64 time_limit, unsigned long lpcsave_p9_host_os_sprs(&host_os_sprs);-/*-*ThiscouldbecombinedwithMSR[RI]clearing,butthatexpands-*theunrecoverablewindow.Itwouldbebettertocoverunrecoverable-*withKVMbadinterrupthandlingratherthanuseMSR[RI]atall.-*-*MuchmoredifficultandlessworthwhiletocombinewithIR/DR-*disable.-*/-hard_irq_disable();+msr=kvmppc_msr_hard_disable_set_facilities(vcpu,msr);if(lazy_irq_pending()){trap=0;gotoout;}-/* MSR bits may have been cleared by context switch */-msr=0;-if(IS_ENABLED(CONFIG_PPC_FPU))-msr|=MSR_FP;-if(cpu_has_feature(CPU_FTR_ALTIVEC))-msr|=MSR_VEC;-if(cpu_has_feature(CPU_FTR_VSX))-msr|=MSR_VSX;-if((cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))&&-(vcpu->arch.hfscr&HFSCR_TM))-msr|=MSR_TM;-msr=msr_check_and_set(msr);-/* Save MSR for restore. This is after hard disable, so EE is clear. */-if(unlikely(load_vcpu_state(vcpu,&host_os_sprs)))msr=mfmsr();/* MSR may have been updated */
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:12:37
slbmfee/slbmfev instructions are very expensive, moreso than a regular
mfspr instruction, so minimising them significantly improves hash guest
exit performance. The slbmfev is only required if slbmfee found a valid
SLB entry.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv_p9_entry.c | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:13:00
The mmu will almost always be ready.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
@@ -4376,7 +4376,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,vc->runner=vcpu;/* See if the MMU is ready to go */-if(!kvm->arch.mmu_ready){+if(unlikely(!kvm->arch.mmu_ready)){r=kvmhv_setup_mmu(vcpu);if(r){run->exit_reason=KVM_EXIT_FAIL_ENTRY;
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:13:23
cpu_in_guest is set to determine if a CPU needs to be IPI'ed to exit
the guest and notice the need_tlb_flush bit.
This can be implemented as a global per-CPU pointer to the currently
running guest instead of per-guest cpumasks, saving 2 atomics per
entry/exit. P7/8 doesn't require cpu_in_guest, nor does a nested HV
(only the L0 does), so move it to the P9 HV path.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/kvm_book3s_64.h | 1 -
arch/powerpc/include/asm/kvm_host.h | 1 -
arch/powerpc/kvm/book3s_hv.c | 38 +++++++++++++-----------
3 files changed, 21 insertions(+), 19 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:13:53
The P9 path always uses one vcpu per vcore, so none of the the vcore,
locks, stolen time, blocking logic, shared waitq, etc., is required.
Remove most of it.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 147 ++++++++++++++++++++---------------
1 file changed, 85 insertions(+), 62 deletions(-)
@@ -903,13 +925,14 @@ static int kvm_arch_vcpu_yield_to(struct kvm_vcpu *target)*modehandlerisnotcalledbutnootherthreadsareinthe*sourcevcore.*/--spin_lock(&vcore->lock);-if(target->arch.state==KVMPPC_VCPU_RUNNABLE&&-vcore->vcore_state!=VCORE_INACTIVE&&-vcore->runner)-target=vcore->runner;-spin_unlock(&vcore->lock);+if(!cpu_has_feature(CPU_FTR_ARCH_300)){+spin_lock(&vcore->lock);+if(target->arch.state==KVMPPC_VCPU_RUNNABLE&&+vcore->vcore_state!=VCORE_INACTIVE&&+vcore->runner)+target=vcore->runner;+spin_unlock(&vcore->lock);+}returnkvm_vcpu_yield_to(target);}
@@ -3105,13 +3128,6 @@ static void kvmppc_start_thread(struct kvm_vcpu *vcpu, struct kvmppc_vcore *vc)kvmppc_ipi_thread(cpu);}-/* Old path does this in asm */-staticvoidkvmppc_stop_thread(structkvm_vcpu*vcpu)-{-vcpu->cpu=-1;-vcpu->arch.thread_cpu=-1;-}-staticvoidkvmppc_wait_for_nap(intn_threads){intcpu=smp_processor_id();
@@ -4093,7 +4113,7 @@ static int kvmppc_vcore_check_block(struct kvmppc_vcore *vc)inti;for_each_runnable_thread(i,vcpu,vc){-if(!vcpu->arch.ceded||kvmppc_vcpu_woken(vcpu))+if(kvmppc_vcpu_check_block(vcpu))return1;}
@@ -4110,6 +4130,8 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)intdo_sleep=1;u64block_ns;+WARN_ON_ONCE(cpu_has_feature(CPU_FTR_ARCH_300));+/* Poll for pending exceptions and ceded state */cur=start_poll=ktime_get();if(vc->halt_poll_ns){
@@ -4375,11 +4397,7 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,vcpu->arch.ceded=0;vcpu->arch.run_task=current;vcpu->arch.state=KVMPPC_VCPU_RUNNABLE;-vcpu->arch.busy_preempt=TB_NIL;vcpu->arch.last_inst=KVM_INST_FETCH_FAILED;-vc->runnable_threads[0]=vcpu;-vc->n_runnable=1;-vc->runner=vcpu;/* See if the MMU is ready to go */if(unlikely(!kvm->arch.mmu_ready)){
@@ -4397,11 +4415,8 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,kvmppc_update_vpas(vcpu);-init_vcore_to_run(vc);-preempt_disable();pcpu=smp_processor_id();-vc->pcpu=pcpu;if(kvm_is_radix(kvm))kvmppc_prepare_radix_vcpu(vcpu,pcpu);
@@ -4430,21 +4445,23 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,gotoout;}-tb=mftb();+if(vcpu->arch.timer_running){+hrtimer_try_to_cancel(&vcpu->arch.dec_timer);+vcpu->arch.timer_running=0;+}-vcpu->arch.stolen_logged=vcore_stolen_time(vc,tb);-vc->preempt_tb=TB_NIL;+tb=mftb();-kvmppc_clear_host_core(pcpu);+vcpu->cpu=pcpu;+vcpu->arch.thread_cpu=pcpu;+local_paca->kvm_hstate.kvm_vcpu=vcpu;+local_paca->kvm_hstate.ptid=0;+local_paca->kvm_hstate.fake_suspend=0;-local_paca->kvm_hstate.napping=0;-local_paca->kvm_hstate.kvm_split_mode=NULL;-kvmppc_start_thread(vcpu,vc);+vc->pcpu=pcpu;// for kvmppc_create_dtl_entrykvmppc_create_dtl_entry(vcpu,vc,tb);-trace_kvm_guest_enter(vcpu);-vc->vcore_state=VCORE_RUNNING;-trace_kvmppc_run_core(vc,0);+trace_kvm_guest_enter(vcpu);guest_enter_irqoff();
@@ -4466,11 +4483,10 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,set_irq_happened(trap);-kvmppc_set_host_core(pcpu);-guest_exit_irqoff();-kvmppc_stop_thread(vcpu);+vcpu->cpu=-1;+vcpu->arch.thread_cpu=-1;powerpc_local_irq_pmu_restore(flags);
@@ -4497,28 +4513,31 @@ int kvmhv_run_single_vcpu(struct kvm_vcpu *vcpu, u64 time_limit,}vcpu->arch.ret=r;-if(is_kvmppc_resume_guest(r)&&vcpu->arch.ceded&&-!kvmppc_vcpu_woken(vcpu)){+if(is_kvmppc_resume_guest(r)&&!kvmppc_vcpu_check_block(vcpu)){kvmppc_set_timer(vcpu);-while(vcpu->arch.ceded&&!kvmppc_vcpu_woken(vcpu)){++prepare_to_rcuwait(&vcpu->wait);+for(;;){+set_current_state(TASK_INTERRUPTIBLE);if(signal_pending(current)){vcpu->stat.signal_exits++;run->exit_reason=KVM_EXIT_INTR;vcpu->arch.ret=-EINTR;break;}-spin_lock(&vc->lock);-kvmppc_vcore_blocked(vc);-spin_unlock(&vc->lock);++if(kvmppc_vcpu_check_block(vcpu))+break;++trace_kvmppc_vcore_blocked(vc,0);+schedule();+trace_kvmppc_vcore_blocked(vc,1);}+finish_rcuwait(&vcpu->wait);}vcpu->arch.ceded=0;-vc->vcore_state=VCORE_INACTIVE;-trace_kvmppc_run_core(vc,1);-done:-kvmppc_remove_runnable(vc,vcpu,tb);trace_kvmppc_run_vcpu_exit(vcpu);returnvcpu->arch.ret;
@@ -4602,7 +4621,8 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)kvmppc_save_current_sprs();-vcpu->arch.waitp=&vcpu->arch.vcore->wait;+if(!cpu_has_feature(CPU_FTR_ARCH_300))+vcpu->arch.waitp=&vcpu->arch.vcore->wait;vcpu->arch.pgdir=kvm->mm->pgd;vcpu->arch.state=KVMPPC_VCPU_BUSY_IN_HOST;
@@ -5064,6 +5084,9 @@ void kvmppc_alloc_host_rm_ops(void)intcpu,core;intsize;+if(cpu_has_feature(CPU_FTR_ARCH_300))+return;+/* Not the first time here ? */if(kvmppc_host_rm_ops_hv!=NULL)return;
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:14:18
This goes further to removing vcores from the P9 path. Also avoid the
memset in favour of explicitly initialising all fields.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 61 +++++++++++++++++++++---------------
1 file changed, 35 insertions(+), 26 deletions(-)
@@ -748,6 +737,27 @@ static void kvmppc_create_dtl_entry(struct kvm_vcpu *vcpu,vcpu->arch.dtl.dirty=true;}+staticvoidkvmppc_create_dtl_entry(structkvm_vcpu*vcpu,+structkvmppc_vcore*vc)+{+unsignedlongstolen;+unsignedlongcore_stolen;+u64now;+unsignedlongflags;++now=mftb();++core_stolen=vcore_stolen_time(vc,now);+stolen=core_stolen-vcpu->arch.stolen_logged;+vcpu->arch.stolen_logged=core_stolen;+spin_lock_irqsave(&vcpu->arch.tbacct_lock,flags);+stolen+=vcpu->arch.busy_stolen;+vcpu->arch.busy_stolen=0;+spin_unlock_irqrestore(&vcpu->arch.tbacct_lock,flags);++__kvmppc_create_dtl_entry(vcpu,vc->pcpu,now+vc->tb_offset,stolen);+}+/* See if there is a doorbell interrupt pending for a vcpu */staticboolkvmppc_doorbell_pending(structkvm_vcpu*vcpu){
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:14:47
The P9 path uses vc->dpdes only for msgsndp / SMT emulation. This adds
an ordering requirement between vcpu->doorbell_request and vc->dpdes for
no real benefit. Use vcpu->doorbell_request directly.
XXX: verify msgsndp / DPDES emulation works properly.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 18 ++++++++++--------
arch/powerpc/kvm/book3s_hv_builtin.c | 2 ++
arch/powerpc/kvm/book3s_hv_p9_entry.c | 14 ++++++++++----
3 files changed, 22 insertions(+), 12 deletions(-)
@@ -660,6 +660,8 @@ void kvmppc_guest_entry_inject_int(struct kvm_vcpu *vcpu)intext;unsignedlonglpcr;+WARN_ON_ONCE(cpu_has_feature(CPU_FTR_ARCH_300));+/* Insert EXTERNAL bit into LPCR at the MER bit position */ext=(vcpu->arch.pending_exceptions>>BOOK3S_IRQPRIO_EXTERNAL)&1;lpcr=mfspr(SPRN_LPCR);
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-07-26 04:15:15
On POWER9 and newer, rather than the complex HMI synchronisation and
subcore state, have each thread un-apply the guest TB offset before
calling into the early HMI handler.
This allows the subcore state to be avoided, including subcore enter
/ exit guest, which includes an expensive divide that shows up
slightly in profiles.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kvm/book3s_hv.c | 12 +++++-----
arch/powerpc/kvm/book3s_hv_hmi.c | 7 +++++-
arch/powerpc/kvm/book3s_hv_p9_entry.c | 32 ++++++++++++++++++++++++++-
arch/powerpc/kvm/book3s_hv_ras.c | 4 ++++
4 files changed, 46 insertions(+), 9 deletions(-)
@@ -136,6 +136,10 @@ void kvmppc_realmode_machine_check(struct kvm_vcpu *vcpu)vcpu->arch.mce_evt=mce_evt;}+/*+*ThissubcoreHMIhandlingisallonlyforpre-POWER9CPUs.+*/+/* Check if dynamic split is in force and return subcore size accordingly. */staticinlineintkvmppc_cur_subcore_size(void){
From: kernel test robot <hidden> Date: 2021-07-26 06:59:49
Hi Nicholas,
I love your patch! Perhaps something to improve:
[auto build test WARNING on linus/master]
[also build test WARNING on v5.14-rc3 next-20210723]
[cannot apply to powerpc/next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Nicholas-Piggin/KVM-PPC-Book3S-HV-P9-entry-exit-optimisations/20210726-115329
base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ff1176468d368232b684f75e82563369208bc371
config: powerpc-randconfig-r022-20210726 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project c63dbd850182797bc4b76124d08e1c320ab2365d)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install powerpc cross compiling tool for clang build
# apt-get install binutils-powerpc-linux-gnu
# https://github.com/0day-ci/linux/commit/d173e4690cf13578686dbbce48e1f81e925b96af
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Nicholas-Piggin/KVM-PPC-Book3S-HV-P9-entry-exit-optimisations/20210726-115329
git checkout d173e4690cf13578686dbbce48e1f81e925b96af
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=powerpc
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <redacted>
All warnings (new ones prefixed by >>):
arch/powerpc/kernel/process.c:612:33: error: no member named 'tm_tfhar' in 'struct thread_struct'
current->thread.tm_tfhar = mfspr(SPRN_TFHAR);
~~~~~~~~~~~~~~~ ^
arch/powerpc/kernel/process.c:613:33: error: no member named 'tm_tfiar' in 'struct thread_struct'
current->thread.tm_tfiar = mfspr(SPRN_TFIAR);
~~~~~~~~~~~~~~~ ^
arch/powerpc/kernel/process.c:614:33: error: no member named 'tm_texasr' in 'struct thread_struct'
current->thread.tm_texasr = mfspr(SPRN_TEXASR);
~~~~~~~~~~~~~~~ ^
quoted
arch/powerpc/kernel/process.c:596:6: warning: no previous prototype for function 'save_user_regs_kvm' [-Wmissing-prototypes]
void save_user_regs_kvm(void)
^
arch/powerpc/kernel/process.c:596:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
void save_user_regs_kvm(void)
^
static
quoted
arch/powerpc/kernel/process.c:611:16: warning: shift count >= width of type [-Wshift-count-overflow]
if (usermsr & MSR_TM) {
^~~~~~
arch/powerpc/include/asm/reg.h:115:17: note: expanded from macro 'MSR_TM'
#define MSR_TM __MASK(MSR_TM_LG) /* Transactional Mem Available */
^~~~~~~~~~~~~~~~~
arch/powerpc/include/asm/reg.h:66:23: note: expanded from macro '__MASK'
#define __MASK(X) (1UL<<(X))
^ ~~~
arch/powerpc/kernel/process.c:615:47: warning: shift count >= width of type [-Wshift-count-overflow]
current->thread.regs->msr &= ~MSR_TM;
^~~~~~
arch/powerpc/include/asm/reg.h:115:17: note: expanded from macro 'MSR_TM'
#define MSR_TM __MASK(MSR_TM_LG) /* Transactional Mem Available */
^~~~~~~~~~~~~~~~~
arch/powerpc/include/asm/reg.h:66:23: note: expanded from macro '__MASK'
#define __MASK(X) (1UL<<(X))
^ ~~~
3 warnings and 3 errors generated.
vim +/save_user_regs_kvm +596 arch/powerpc/kernel/process.c
595
> 596 void save_user_regs_kvm(void)
597 {
598 unsigned long usermsr;
599
600 if (!current->thread.regs)
601 return;
602
603 usermsr = current->thread.regs->msr;
604
605 if (usermsr & MSR_FP)
606 save_fpu(current);
607
608 if (usermsr & MSR_VEC)
609 save_altivec(current);
610
> 611 if (usermsr & MSR_TM) {
612 current->thread.tm_tfhar = mfspr(SPRN_TFHAR);
613 current->thread.tm_tfiar = mfspr(SPRN_TFIAR);
614 current->thread.tm_texasr = mfspr(SPRN_TEXASR);
615 current->thread.regs->msr &= ~MSR_TM;
616 }
617 }
618 EXPORT_SYMBOL_GPL(save_user_regs_kvm);
619
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
From: kernel test robot <hidden> Date: 2021-07-26 07:03:33
Hi Nicholas,
I love your patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v5.14-rc3 next-20210723]
[cannot apply to powerpc/next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Nicholas-Piggin/KVM-PPC-Book3S-HV-P9-entry-exit-optimisations/20210726-115329
base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ff1176468d368232b684f75e82563369208bc371
config: powerpc64-randconfig-r024-20210726 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project c63dbd850182797bc4b76124d08e1c320ab2365d)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install powerpc64 cross compiling tool for clang build
# apt-get install binutils-powerpc64-linux-gnu
# https://github.com/0day-ci/linux/commit/d173e4690cf13578686dbbce48e1f81e925b96af
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Nicholas-Piggin/KVM-PPC-Book3S-HV-P9-entry-exit-optimisations/20210726-115329
git checkout d173e4690cf13578686dbbce48e1f81e925b96af
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross O=build_dir ARCH=powerpc SHELL=/bin/bash arch/powerpc/kernel/
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <redacted>
All errors (new ones prefixed by >>):
arch/powerpc/include/asm/io-defs.h:45:1: error: performing pointer arithmetic on a null pointer has undefined behavior [-Werror,-Wnull-pointer-arithmetic]
DEF_PCI_AC_NORET(insw, (unsigned long p, void *b, unsigned long c),
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/powerpc/include/asm/io.h:616:3: note: expanded from macro 'DEF_PCI_AC_NORET'
__do_##name al; \
^~~~~~~~~~~~~~
<scratch space>:121:1: note: expanded from here
__do_insw
^
arch/powerpc/include/asm/io.h:557:56: note: expanded from macro '__do_insw'
#define __do_insw(p, b, n) readsw((PCI_IO_ADDR)_IO_BASE+(p), (b), (n))
~~~~~~~~~~~~~~~~~~~~~^
In file included from arch/powerpc/kernel/process.c:28:
In file included from include/linux/init_task.h:9:
In file included from include/linux/ftrace.h:10:
In file included from include/linux/trace_recursion.h:5:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/powerpc/include/asm/hardirq.h:6:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/powerpc/include/asm/io.h:619:
arch/powerpc/include/asm/io-defs.h:47:1: error: performing pointer arithmetic on a null pointer has undefined behavior [-Werror,-Wnull-pointer-arithmetic]
DEF_PCI_AC_NORET(insl, (unsigned long p, void *b, unsigned long c),
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/powerpc/include/asm/io.h:616:3: note: expanded from macro 'DEF_PCI_AC_NORET'
__do_##name al; \
^~~~~~~~~~~~~~
<scratch space>:123:1: note: expanded from here
__do_insl
^
arch/powerpc/include/asm/io.h:558:56: note: expanded from macro '__do_insl'
#define __do_insl(p, b, n) readsl((PCI_IO_ADDR)_IO_BASE+(p), (b), (n))
~~~~~~~~~~~~~~~~~~~~~^
In file included from arch/powerpc/kernel/process.c:28:
In file included from include/linux/init_task.h:9:
In file included from include/linux/ftrace.h:10:
In file included from include/linux/trace_recursion.h:5:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/powerpc/include/asm/hardirq.h:6:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/powerpc/include/asm/io.h:619:
arch/powerpc/include/asm/io-defs.h:49:1: error: performing pointer arithmetic on a null pointer has undefined behavior [-Werror,-Wnull-pointer-arithmetic]
DEF_PCI_AC_NORET(outsb, (unsigned long p, const void *b, unsigned long c),
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/powerpc/include/asm/io.h:616:3: note: expanded from macro 'DEF_PCI_AC_NORET'
__do_##name al; \
^~~~~~~~~~~~~~
<scratch space>:125:1: note: expanded from here
__do_outsb
^
arch/powerpc/include/asm/io.h:559:58: note: expanded from macro '__do_outsb'
#define __do_outsb(p, b, n) writesb((PCI_IO_ADDR)_IO_BASE+(p),(b),(n))
~~~~~~~~~~~~~~~~~~~~~^
In file included from arch/powerpc/kernel/process.c:28:
In file included from include/linux/init_task.h:9:
In file included from include/linux/ftrace.h:10:
In file included from include/linux/trace_recursion.h:5:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/powerpc/include/asm/hardirq.h:6:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/powerpc/include/asm/io.h:619:
arch/powerpc/include/asm/io-defs.h:51:1: error: performing pointer arithmetic on a null pointer has undefined behavior [-Werror,-Wnull-pointer-arithmetic]
DEF_PCI_AC_NORET(outsw, (unsigned long p, const void *b, unsigned long c),
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/powerpc/include/asm/io.h:616:3: note: expanded from macro 'DEF_PCI_AC_NORET'
__do_##name al; \
^~~~~~~~~~~~~~
<scratch space>:127:1: note: expanded from here
__do_outsw
^
arch/powerpc/include/asm/io.h:560:58: note: expanded from macro '__do_outsw'
#define __do_outsw(p, b, n) writesw((PCI_IO_ADDR)_IO_BASE+(p),(b),(n))
~~~~~~~~~~~~~~~~~~~~~^
In file included from arch/powerpc/kernel/process.c:28:
In file included from include/linux/init_task.h:9:
In file included from include/linux/ftrace.h:10:
In file included from include/linux/trace_recursion.h:5:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/powerpc/include/asm/hardirq.h:6:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/powerpc/include/asm/io.h:619:
arch/powerpc/include/asm/io-defs.h:53:1: error: performing pointer arithmetic on a null pointer has undefined behavior [-Werror,-Wnull-pointer-arithmetic]
DEF_PCI_AC_NORET(outsl, (unsigned long p, const void *b, unsigned long c),
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/powerpc/include/asm/io.h:616:3: note: expanded from macro 'DEF_PCI_AC_NORET'
__do_##name al; \
^~~~~~~~~~~~~~
<scratch space>:129:1: note: expanded from here
__do_outsl
^
arch/powerpc/include/asm/io.h:561:58: note: expanded from macro '__do_outsl'
#define __do_outsl(p, b, n) writesl((PCI_IO_ADDR)_IO_BASE+(p),(b),(n))
~~~~~~~~~~~~~~~~~~~~~^
quoted
arch/powerpc/kernel/process.c:612:33: error: no member named 'tm_tfhar' in 'struct thread_struct'
Rather than have KVM look up the host timer and fiddle with the
irq-work internal details, have the powerpc/time.c code provide a
function for KVM to re-arm the Linux timer code when exiting a
guest.
This is implementation has an improvement over existing code of
marking a decrementer interrupt as soft-pending if a timer has
expired, rather than setting DEC to a -ve value, which tended to
cause host timers to take two interrupts (first hdec to exit the
guest, then the immediate dec).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/time.h | 16 +++-------
arch/powerpc/kernel/time.c | 52 +++++++++++++++++++++++++++------
arch/powerpc/kvm/book3s_hv.c | 7 ++---
3 files changed, 49 insertions(+), 26 deletions(-)
@@ -542,13 +552,44 @@ void arch_irq_work_raise(void) preempt_enable(); }+static void set_dec_or_work(u64 val)+{+ set_dec(val);+ /* We may have raced with new irq work */+ if (unlikely(test_irq_work_pending()))+ set_dec(1);+}+ #else /* CONFIG_IRQ_WORK */ #define test_irq_work_pending() 0 #define clear_irq_work_pending()+static void set_dec_or_work(u64 val)+{+ set_dec(val);+} #endif /* CONFIG_IRQ_WORK */+#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE+void timer_rearm_host_dec(u64 now)+{+ u64 *next_tb = this_cpu_ptr(&decrementers_next_tb);++ WARN_ON_ONCE(!arch_irqs_disabled());+ WARN_ON_ONCE(mfmsr() & MSR_EE);++ if (now >= *next_tb) {+ local_paca->irq_happened |= PACA_IRQ_DEC;+ } else {+ now = *next_tb - now;+ if (now <= decrementer_max)+ set_dec_or_work(now);+ }+}+EXPORT_SYMBOL_GPL(timer_rearm_host_dec);+#endif+ /* * timer_interrupt - gets called when the decrementer overflows, * with interrupts disabled.
@@ -609,10 +650,7 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(timer_interrupt) } else { now = *next_tb - now; if (now <= decrementer_max)- set_dec(now);- /* We may have raced with new irq work */- if (test_irq_work_pending())- set_dec(1);+ set_dec_or_work(now); __this_cpu_inc(irq_stat.timer_irqs_others); }
@@ -854,11 +892,7 @@ static int decrementer_set_next_event(unsigned long evt, struct clock_event_device *dev) { __this_cpu_write(decrementers_next_tb, get_tb() + evt);- set_dec(evt);-- /* We may have raced with new irq work */- if (test_irq_work_pending())- set_dec(1);+ set_dec_or_work(evt); return 0; }
@@ -4018,11 +4018,8 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vc->entry_exit_map=0x101;vc->in_guest=0;-next_timer=timer_get_next_tb();-set_dec(next_timer-tb);-/* We may have raced with new irq work */-if(test_irq_work_pending())-set_dec(1);+timer_rearm_host_dec(tb);+mtspr(SPRN_SPRG_VDSO_WRITE,local_paca->sprg_vdso);kvmhv_load_host_pmu();
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2021-08-06 01:17:13
Nicholas Piggin [off-list ref] writes:
The softpatch interrupt sets HSRR0 to the faulting instruction +4, so
it should subtract 4 for the faulting instruction address. Also have it
emulate and deliver HFAC interrupts correctly, which is important for
nested HV and facility demand-faulting in future.
The nip being off by 4 sounds bad. But I guess it's not that big a deal
because it's only used for reporting the instruction address?
Would also be good to have some more explanation of why it's OK to
change from illegal to HFAC, which is a guest visible change.
It can be useful in simulators (with very constrained environments)
to allow some PMCs to run from boot so they can be sampled directly
by a test harness, rather than having to run perf.
A previous change freezes counters at boot by default, so provide
a boot time option to un-freeze (plus a bit more flexibility).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
.../admin-guide/kernel-parameters.txt | 7 ++++
arch/powerpc/perf/core-book3s.c | 35 +++++++++++++++++++
2 files changed, 42 insertions(+)
@@ -4089,6 +4089,13 @@ Override pmtimer IOPort with a hex value. e.g. pmtmr=0x508+ pmu= [PPC] Manually enable the PMU.
This is bit confusing, IIUC, we are manually disabling the perf
registration
with this option and not pmu. If this option is used, we will unfreeze the
MMCR0_FC (only in the HV_mode) and not register perf subsystem.
Since this option is valid only for HV_mode, canwe call it
kvm_disable_perf or kvm_dis_perf.
quoted hunk
+ Enable the PMU by setting MMCR0 to 0 (clear FC bit).
+ This option is implemented for Book3S processors.
+ If a number is given, then MMCR1 is set to that number,
+ otherwise (e.g., 'pmu=on'), it is left 0. The perf
+ subsystem is disabled if this option is used.
+
pm_debug_messages [SUSPEND,KNL]
Enable suspend/resume debug messages during boot up.
@@ -2428,8 +2428,24 @@ int register_power_pmu(struct power_pmu *pmu)}#ifdef CONFIG_PPC64+staticboolpmu_override=false;+staticunsignedlongpmu_override_val;+staticvoiddo_pmu_override(void*data)+{+ppc_set_pmu_inuse(1);+if(pmu_override_val)+mtspr(SPRN_MMCR1,pmu_override_val);+mtspr(SPRN_MMCR0,mfspr(SPRN_MMCR0)&~MMCR0_FC);+}+staticint__initinit_ppc64_pmu(void){+if(cpu_has_feature(CPU_FTR_HVMODE)&&pmu_override){+printk(KERN_WARNING"perf: disabling perf due to pmu= command line option.\n");+on_each_cpu(do_pmu_override,NULL,1);+return0;+}+/* run through all the pmu drivers one at a time */if(!init_power5_pmu())return0;
@@ -2451,4 +2467,23 @@ static int __init init_ppc64_pmu(void)returninit_generic_compat_pmu();}early_initcall(init_ppc64_pmu);++staticint__initpmu_setup(char*str)+{+unsignedlongval;++if(!early_cpu_has_feature(CPU_FTR_HVMODE))+return0;++pmu_override=true;++if(kstrtoul(str,0,&val))+val=0;++pmu_override_val=val;++return1;+}+__setup("pmu=",pmu_setup);+#endif
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2021-08-06 07:34:30
Nicholas Piggin [off-list ref] writes:
Revert the workaround added by commit 63279eeb7f93a ("KVM: PPC: Book3S
HV: Always save guest pmu for guest capable of nesting").
Nested capable guests running with the earlier commit ("KVM: PPC: Book3S
HV Nested: Indicate guest PMU in-use in VPA") will now indicate the PMU
in-use status of their guests, which means the parent does not need to
unconditionally save the PMU for nested capable guests.
This will cause the PMU to break for nested guests when running older
nested hypervisor guests under a kernel with this change. It's unclear
there's an easy way to avoid that, so this could wait for a release or
so for the fix to filter into stable kernels.
I'm not sure PMU inside nested guests is getting much use, but I don't
think we can break this quite so casually :)
Especially as the failure mode will be PMU counts that don't match
reality, which is hard to diagnose. It took nearly a year for us to find
the original bug.
I think we need to hold this back for a while.
We could put it under a CONFIG option, and then flip that option to off
at some point in the future.
cheers
@@ -4003,8 +4003,6 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu->arch.vpa.dirty=1;save_pmu=lp->pmcregs_in_use;}-/* Must save pmu if this guest is capable of running nested guests */-save_pmu|=nesting_enabled(vcpu->kvm);kvmhv_save_guest_pmu(vcpu,save_pmu);#ifdef CONFIG_PPC_PSERIES
On 26-Jul-2021, at 9:19 AM, Nicholas Piggin [off-list ref] wrote:
It can be useful in simulators (with very constrained environments)
to allow some PMCs to run from boot so they can be sampled directly
by a test harness, rather than having to run perf.
A previous change freezes counters at boot by default, so provide
a boot time option to un-freeze (plus a bit more flexibility).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
.../admin-guide/kernel-parameters.txt | 7 ++++
arch/powerpc/perf/core-book3s.c | 35 +++++++++++++++++++
2 files changed, 42 insertions(+)
Override pmtimer IOPort with a hex value.
e.g. pmtmr=0x508
+ pmu= [PPC] Manually enable the PMU.
+ Enable the PMU by setting MMCR0 to 0 (clear FC bit).
+ This option is implemented for Book3S processors.
+ If a number is given, then MMCR1 is set to that number,
+ otherwise (e.g., 'pmu=on'), it is left 0. The perf
+ subsystem is disabled if this option is used.
+
pm_debug_messages [SUSPEND,KNL]
Enable suspend/resume debug messages during boot up.
Hi Nick
Here, we are not doing any validity check for the value used to set MMCR1.
For advanced users, the option to pass value for MMCR1 is fine. But other cases, it could result in
invalid event getting used. Do we need to restrict this boot time option for only PMC5/6 ?
Thanks
Athira
quoted hunk
+}
+
static int __init init_ppc64_pmu(void)
{
+ if (cpu_has_feature(CPU_FTR_HVMODE) && pmu_override) {
+ printk(KERN_WARNING "perf: disabling perf due to pmu= command line option.\n");
+ on_each_cpu(do_pmu_override, NULL, 1);
+ return 0;
+ }
+
/* run through all the pmu drivers one at a time */
if (!init_power5_pmu())
return 0;
@@ -2451,4 +2467,23 @@ static int __init init_ppc64_pmu(void)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-08-06 10:26:24
Excerpts from Michael Ellerman's message of August 6, 2021 11:16 am:
Nicholas Piggin [off-list ref] writes:
quoted
The softpatch interrupt sets HSRR0 to the faulting instruction +4, so
it should subtract 4 for the faulting instruction address. Also have it
emulate and deliver HFAC interrupts correctly, which is important for
nested HV and facility demand-faulting in future.
The nip being off by 4 sounds bad. But I guess it's not that big a deal
because it's only used for reporting the instruction address?
Yeah currently I think so. It's not that bad of a bug.
Would also be good to have some more explanation of why it's OK to
change from illegal to HFAC, which is a guest visible change.
Good point. Again for now it doesn't really matter because the HFAC
handler turns everything (except msgsndp) into a sigill anyway, so
becomes important when we start using HFACs. Put that way I'll probably
split it out.
@@ -74,19 +74,23 @@ int kvmhv_p9_tm_emulation(struct kvm_vcpu *vcpu)casePPC_INST_RFEBB:if((msr&MSR_PR)&&(vcpu->arch.vcore->pcr&PCR_ARCH_206)){/* generate an illegal instruction interrupt */+vcpu->arch.regs.nip-=4;kvmppc_core_queue_program(vcpu,SRR1_PROGILL);returnRESUME_GUEST;}/* check EBB facility is available */if(!(vcpu->arch.hfscr&HFSCR_EBB)){-/* generate an illegal instruction interrupt */-kvmppc_core_queue_program(vcpu,SRR1_PROGILL);-returnRESUME_GUEST;+vcpu->arch.regs.nip-=4;+vcpu->arch.hfscr&=~HFSCR_INTR_CAUSE;+vcpu->arch.hfscr|=(u64)FSCR_EBB_LG<<56;+vcpu->arch.trap=BOOK3S_INTERRUPT_H_FAC_UNAVAIL;+return-1;/* rerun host interrupt handler */
This is EBB not TM. Probably OK to leave it in this patch as long as
it's mentioned in the change log?
It is, but you can get a softpatch interrupt on rfebb changing TM state.
Although I haven't actually tested to see if you get a softpatch when
HFSCR disables EBB or the hardware just gives you the HFAC. For that
matter, same for all the other facility tests.
Thanks,
Nick
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-08-06 10:31:02
Excerpts from Christophe Leroy's message of August 5, 2021 5:22 pm:
Le 26/07/2021 à 05:49, Nicholas Piggin a écrit :
quoted
Rather than have KVM look up the host timer and fiddle with the
irq-work internal details, have the powerpc/time.c code provide a
function for KVM to re-arm the Linux timer code when exiting a
guest.
This is implementation has an improvement over existing code of
marking a decrementer interrupt as soft-pending if a timer has
expired, rather than setting DEC to a -ve value, which tended to
cause host timers to take two interrupts (first hdec to exit the
guest, then the immediate dec).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/time.h | 16 +++-------
arch/powerpc/kernel/time.c | 52 +++++++++++++++++++++++++++------
arch/powerpc/kvm/book3s_hv.c | 7 ++---
3 files changed, 49 insertions(+), 26 deletions(-)
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-08-06 10:33:17
Excerpts from Michael Ellerman's message of August 6, 2021 5:34 pm:
Nicholas Piggin [off-list ref] writes:
quoted
Revert the workaround added by commit 63279eeb7f93a ("KVM: PPC: Book3S
HV: Always save guest pmu for guest capable of nesting").
Nested capable guests running with the earlier commit ("KVM: PPC: Book3S
HV Nested: Indicate guest PMU in-use in VPA") will now indicate the PMU
in-use status of their guests, which means the parent does not need to
unconditionally save the PMU for nested capable guests.
This will cause the PMU to break for nested guests when running older
nested hypervisor guests under a kernel with this change. It's unclear
there's an easy way to avoid that, so this could wait for a release or
so for the fix to filter into stable kernels.
I'm not sure PMU inside nested guests is getting much use, but I don't
think we can break this quite so casually :)
Especially as the failure mode will be PMU counts that don't match
reality, which is hard to diagnose. It took nearly a year for us to find
the original bug.
I think we need to hold this back for a while.
We could put it under a CONFIG option, and then flip that option to off
at some point in the future.
Okay that might be a good compromise, I'll do that.
Thanks,
Nick
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-08-06 10:39:31
Excerpts from Madhavan Srinivasan's message of August 6, 2021 5:33 pm:
On 7/26/21 9:19 AM, Nicholas Piggin wrote:
quoted
It can be useful in simulators (with very constrained environments)
to allow some PMCs to run from boot so they can be sampled directly
by a test harness, rather than having to run perf.
A previous change freezes counters at boot by default, so provide
a boot time option to un-freeze (plus a bit more flexibility).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
.../admin-guide/kernel-parameters.txt | 7 ++++
arch/powerpc/perf/core-book3s.c | 35 +++++++++++++++++++
2 files changed, 42 insertions(+)
@@ -4089,6 +4089,13 @@ Override pmtimer IOPort with a hex value. e.g. pmtmr=0x508+ pmu= [PPC] Manually enable the PMU.
This is bit confusing, IIUC, we are manually disabling the perf
registration
with this option and not pmu.
If this option is used, we will unfreeze the
MMCR0_FC (only in the HV_mode) and not register perf subsystem.
With the previous patch, this option un-freezes the PMU
(and disables perf).
Since this option is valid only for HV_mode, canwe call it
kvm_disable_perf or kvm_dis_perf.
It's only disabled for guests because it would require a bit
of logic to set pmcregs_in_use when we register our lppaca. We could
add that if needed, but the intention is for use on BML, not exactly
KVM specific.
I can add HV restriction to the help text. And we could rename the
option. free_run_pmu= or something?
Thanks,
Nick
quoted
+ Enable the PMU by setting MMCR0 to 0 (clear FC bit).
+ This option is implemented for Book3S processors.
+ If a number is given, then MMCR1 is set to that number,
+ otherwise (e.g., 'pmu=on'), it is left 0. The perf
+ subsystem is disabled if this option is used.
+
pm_debug_messages [SUSPEND,KNL]
Enable suspend/resume debug messages during boot up.
@@ -2428,8 +2428,24 @@ int register_power_pmu(struct power_pmu *pmu)}#ifdef CONFIG_PPC64+staticboolpmu_override=false;+staticunsignedlongpmu_override_val;+staticvoiddo_pmu_override(void*data)+{+ppc_set_pmu_inuse(1);+if(pmu_override_val)+mtspr(SPRN_MMCR1,pmu_override_val);+mtspr(SPRN_MMCR0,mfspr(SPRN_MMCR0)&~MMCR0_FC);+}+staticint__initinit_ppc64_pmu(void){+if(cpu_has_feature(CPU_FTR_HVMODE)&&pmu_override){+printk(KERN_WARNING"perf: disabling perf due to pmu= command line option.\n");+on_each_cpu(do_pmu_override,NULL,1);+return0;+}+/* run through all the pmu drivers one at a time */if(!init_power5_pmu())return0;
@@ -2451,4 +2467,23 @@ static int __init init_ppc64_pmu(void)returninit_generic_compat_pmu();}early_initcall(init_ppc64_pmu);++staticint__initpmu_setup(char*str)+{+unsignedlongval;++if(!early_cpu_has_feature(CPU_FTR_HVMODE))+return0;++pmu_override=true;++if(kstrtoul(str,0,&val))+val=0;++pmu_override_val=val;++return1;+}+__setup("pmu=",pmu_setup);+#endif
From: Nicholas Piggin <npiggin@gmail.com> Date: 2021-08-06 10:42:43
Excerpts from Athira Rajeev's message of August 6, 2021 7:28 pm:
quoted
On 26-Jul-2021, at 9:19 AM, Nicholas Piggin [off-list ref] wrote:
It can be useful in simulators (with very constrained environments)
to allow some PMCs to run from boot so they can be sampled directly
by a test harness, rather than having to run perf.
A previous change freezes counters at boot by default, so provide
a boot time option to un-freeze (plus a bit more flexibility).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
.../admin-guide/kernel-parameters.txt | 7 ++++
arch/powerpc/perf/core-book3s.c | 35 +++++++++++++++++++
2 files changed, 42 insertions(+)
Override pmtimer IOPort with a hex value.
e.g. pmtmr=0x508
+ pmu= [PPC] Manually enable the PMU.
+ Enable the PMU by setting MMCR0 to 0 (clear FC bit).
+ This option is implemented for Book3S processors.
+ If a number is given, then MMCR1 is set to that number,
+ otherwise (e.g., 'pmu=on'), it is left 0. The perf
+ subsystem is disabled if this option is used.
+
pm_debug_messages [SUSPEND,KNL]
Enable suspend/resume debug messages during boot up.
Hi Nick
Here, we are not doing any validity check for the value used to set MMCR1.
For advanced users, the option to pass value for MMCR1 is fine. But other cases, it could result in
invalid event getting used. Do we need to restrict this boot time option for only PMC5/6 ?
Depends what would be useful. We don't have to prevent the admin shooting
themselves in the foot with options like this, but if we can make it
safer without making it less useful then that's always a good option.
Thanks,
Nick
Keep better track of the current SPR value in places where
they are to be loaded with a new context, to reduce expensive
mtSPR operations.
-73 cycles (7354) POWER9 virt-mode NULL hcall
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
This juggles SPR switching on the entry and exit sides to be more
symmetric, which makes the next refactoring patch possible with no
functional change.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
@@ -4062,6 +4062,44 @@ static void store_spr_state(struct kvm_vcpu *vcpu)vcpu->arch.ctrl=mfspr(SPRN_CTRLF);}+/* Returns true if current MSR and/or guest MSR may have changed */+staticboolload_vcpu_state(structkvm_vcpu*vcpu,+structp9_host_os_sprs*host_os_sprs)+{+boolret=false;++if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)){+kvmppc_restore_tm_hv(vcpu,vcpu->arch.shregs.msr,true);+ret=true;+}++load_spr_state(vcpu,host_os_sprs);++load_fp_state(&vcpu->arch.fp);+#ifdef CONFIG_ALTIVEC+load_vr_state(&vcpu->arch.vr);+#endif+mtspr(SPRN_VRSAVE,vcpu->arch.vrsave);++returnret;+}++staticvoidstore_vcpu_state(structkvm_vcpu*vcpu)+{+store_spr_state(vcpu);++store_fp_state(&vcpu->arch.fp);+#ifdef CONFIG_ALTIVEC+store_vr_state(&vcpu->arch.vr);+#endif+vcpu->arch.vrsave=mfspr(SPRN_VRSAVE);++if(cpu_has_feature(CPU_FTR_TM)||+cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST))+kvmppc_save_tm_hv(vcpu,vcpu->arch.shregs.msr,true);+}+staticvoidsave_p9_host_os_sprs(structp9_host_os_sprs*host_os_sprs){if(!cpu_has_feature(CPU_FTR_ARCH_31))
@@ -4169,19 +4207,8 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,vcpu_vpa_increment_dispatch(vcpu);-if(cpu_has_feature(CPU_FTR_TM)||-cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)){-kvmppc_restore_tm_hv(vcpu,vcpu->arch.shregs.msr,true);-msr=mfmsr();/* TM restore can update msr */-}--load_spr_state(vcpu,&host_os_sprs);--load_fp_state(&vcpu->arch.fp);-#ifdef CONFIG_ALTIVEC-load_vr_state(&vcpu->arch.vr);-#endif-mtspr(SPRN_VRSAVE,vcpu->arch.vrsave);+if(unlikely(load_vcpu_state(vcpu,&host_os_sprs)))+msr=mfmsr();/* MSR may have been updated */switch_pmu_to_guest(vcpu,&host_os_sprs);
trace_kvmppc_run_core(vc, 1);
}
+/*
+ * Privileged (non-hypervisor) host registers to save.
+ */
+struct p9_host_os_sprs {
+ unsigned long dscr;
+ unsigned long tidr;
+ unsigned long iamr;
+ unsigned long amr;
+ unsigned long fscr;
+
+ unsigned int pmc1;
+ unsigned int pmc2;
+ unsigned int pmc3;
+ unsigned int pmc4;
+ unsigned int pmc5;
+ unsigned int pmc6;
+ unsigned long mmcr0;
+ unsigned long mmcr1;
+ unsigned long mmcr2;
+ unsigned long mmcr3;
+ unsigned long mmcra;
+ unsigned long siar;
+ unsigned long sier1;
+ unsigned long sier2;
+ unsigned long sier3;
+ unsigned long sdar;
+};
+
+static void freeze_pmu(unsigned long mmcr0, unsigned long mmcra)
+{
+ if (!(mmcr0 & MMCR0_FC))
+ goto do_freeze;
+ if (mmcra & MMCRA_SAMPLE_ENABLE)
+ goto do_freeze;
+ if (cpu_has_feature(CPU_FTR_ARCH_31)) {
+ if (!(mmcr0 & MMCR0_PMCCEXT))
+ goto do_freeze;
+ if (!(mmcra & MMCRA_BHRB_DISABLE))
+ goto do_freeze;
+ }
+ return;
+
+do_freeze:
+ mmcr0 = MMCR0_FC;
+ mmcra = 0;
+ if (cpu_has_feature(CPU_FTR_ARCH_31)) {
+ mmcr0 |= MMCR0_PMCCEXT;
+ mmcra = MMCRA_BHRB_DISABLE;
+ }
+
+ mtspr(SPRN_MMCR0, mmcr0);
+ mtspr(SPRN_MMCRA, mmcra);
+ isync();
+}
+
Hi Nick,
After feezing pmu, do we need to clear “pmcregs_in_use” as well?
Also can’t we unconditionally do the MMCR0/MMCRA/ freeze settings in here ? do we need the if conditions for FC/PMCCEXT/BHRB ?
Thanks
Athira
blr
/*
+ * void kvmhv_load_host_pmu(void)
+ *
* Reload host PMU state saved in the PACA by kvmhv_save_host_pmu.
*/
-_GLOBAL(kvmhv_load_host_pmu)
-EXPORT_SYMBOL_GPL(kvmhv_load_host_pmu)
+kvmhv_load_host_pmu:
mflr r0
lbz r4, PACA_PMCINUSE(r13) /* is the host using the PMU? */
cmpwi r4, 0
@@ -2844,25 +2836,18 @@ BEGIN_FTR_SECTION
mtspr SPRN_MMCR2, r8
mtspr SPRN_SIER, r9
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
-BEGIN_FTR_SECTION
- ld r5, HSTATE_MMCR3(r13)
- ld r6, HSTATE_SIER2(r13)
- ld r7, HSTATE_SIER3(r13)
- mtspr SPRN_MMCR3, r5
- mtspr SPRN_SIER2, r6
- mtspr SPRN_SIER3, r7
-END_FTR_SECTION_IFSET(CPU_FTR_ARCH_31)
mtspr SPRN_MMCR0, r3
isync
mtlr r0
23: blr
/*
+ * void kvmhv_save_guest_pmu(struct kvm_vcpu *vcpu, bool pmu_in_use)
+ *
* Save guest PMU state into the vcpu struct.
* r3 = vcpu, r4 = full save flag (PMU in use flag set in VPA)
*/
-_GLOBAL(kvmhv_save_guest_pmu)
-EXPORT_SYMBOL_GPL(kvmhv_save_guest_pmu)
+kvmhv_save_guest_pmu:
mr r9, r3
mr r8, r4
BEGIN_FTR_SECTION
On 06-Aug-2021, at 4:12 PM, Nicholas Piggin [off-list ref] wrote:
Excerpts from Athira Rajeev's message of August 6, 2021 7:28 pm:
quoted
quoted
On 26-Jul-2021, at 9:19 AM, Nicholas Piggin [off-list ref] wrote:
It can be useful in simulators (with very constrained environments)
to allow some PMCs to run from boot so they can be sampled directly
by a test harness, rather than having to run perf.
A previous change freezes counters at boot by default, so provide
a boot time option to un-freeze (plus a bit more flexibility).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
.../admin-guide/kernel-parameters.txt | 7 ++++
arch/powerpc/perf/core-book3s.c | 35 +++++++++++++++++++
2 files changed, 42 insertions(+)
Override pmtimer IOPort with a hex value.
e.g. pmtmr=0x508
+ pmu= [PPC] Manually enable the PMU.
+ Enable the PMU by setting MMCR0 to 0 (clear FC bit).
+ This option is implemented for Book3S processors.
+ If a number is given, then MMCR1 is set to that number,
+ otherwise (e.g., 'pmu=on'), it is left 0. The perf
+ subsystem is disabled if this option is used.
+
pm_debug_messages [SUSPEND,KNL]
Enable suspend/resume debug messages during boot up.
Hi Nick
Here, we are not doing any validity check for the value used to set MMCR1.
For advanced users, the option to pass value for MMCR1 is fine. But other cases, it could result in
invalid event getting used. Do we need to restrict this boot time option for only PMC5/6 ?
Depends what would be useful. We don't have to prevent the admin shooting
themselves in the foot with options like this, but if we can make it
safer without making it less useful then that's always a good option.
Hi Nick
I checked back on my comment and it will be difficult to add/maintain validity check for MMCR1 considering different platforms that we have.
We can go ahead with present approach you have in this patch. Changes looks good to me.
Reviewed-by: Athira Rajeev <redacted>
Excerpts from Madhavan Srinivasan's message of August 6, 2021 5:33 pm:
quoted
On 7/26/21 9:19 AM, Nicholas Piggin wrote:
quoted
It can be useful in simulators (with very constrained environments)
to allow some PMCs to run from boot so they can be sampled directly
by a test harness, rather than having to run perf.
A previous change freezes counters at boot by default, so provide
a boot time option to un-freeze (plus a bit more flexibility).
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
.../admin-guide/kernel-parameters.txt | 7 ++++
arch/powerpc/perf/core-book3s.c | 35 +++++++++++++++++++
2 files changed, 42 insertions(+)
@@ -4089,6 +4089,13 @@ Override pmtimer IOPort with a hex value. e.g. pmtmr=0x508+ pmu= [PPC] Manually enable the PMU.
This is bit confusing, IIUC, we are manually disabling the perf
registration
with this option and not pmu.
If this option is used, we will unfreeze the
MMCR0_FC (only in the HV_mode) and not register perf subsystem.
With the previous patch, this option un-freezes the PMU
(and disables perf).
quoted
Since this option is valid only for HV_mode, canwe call it
kvm_disable_perf or kvm_dis_perf.
It's only disabled for guests because it would require a bit
of logic to set pmcregs_in_use when we register our lppaca. We could
add that if needed, but the intention is for use on BML, not exactly
KVM specific.
I can add HV restriction to the help text. And we could rename the
option. free_run_pmu= or something?
yeah having it a different name will be better. I am not sure
whether we should say "[PPC] Manually enable the PMU",
because IIUC, if we dont provide this option PMU and perf is
anyway enabled, but rest looks good to me.
Maddy
Thanks,
Nick
quoted
quoted
+ Enable the PMU by setting MMCR0 to 0 (clear FC bit).
+ This option is implemented for Book3S processors.
+ If a number is given, then MMCR1 is set to that number,
+ otherwise (e.g., 'pmu=on'), it is left 0. The perf
+ subsystem is disabled if this option is used.
+
pm_debug_messages [SUSPEND,KNL]
Enable suspend/resume debug messages during boot up.
@@ -2428,8 +2428,24 @@ int register_power_pmu(struct power_pmu *pmu)}#ifdef CONFIG_PPC64+staticboolpmu_override=false;+staticunsignedlongpmu_override_val;+staticvoiddo_pmu_override(void*data)+{+ppc_set_pmu_inuse(1);+if(pmu_override_val)+mtspr(SPRN_MMCR1,pmu_override_val);+mtspr(SPRN_MMCR0,mfspr(SPRN_MMCR0)&~MMCR0_FC);+}+staticint__initinit_ppc64_pmu(void){+if(cpu_has_feature(CPU_FTR_HVMODE)&&pmu_override){+printk(KERN_WARNING"perf: disabling perf due to pmu= command line option.\n");+on_each_cpu(do_pmu_override,NULL,1);+return0;+}+/* run through all the pmu drivers one at a time */if(!init_power5_pmu())return0;
@@ -2451,4 +2467,23 @@ static int __init init_ppc64_pmu(void)returninit_generic_compat_pmu();}early_initcall(init_ppc64_pmu);++staticint__initpmu_setup(char*str)+{+unsignedlongval;++if(!early_cpu_has_feature(CPU_FTR_HVMODE))+return0;++pmu_override=true;++if(kstrtoul(str,0,&val))+val=0;++pmu_override_val=val;++return1;+}+__setup("pmu=",pmu_setup);+#endif