From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:13
ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
short), which enables simpler support of Type-2 hypervisors.
This extension allows the kernel to directly run at EL2, and
significantly reduces the number of system registers shared between
host and guest, reducing the overhead of virtualization.
In order to have the same kernel binary running on all versions of the
architecture, this series makes heavy use of runtime code patching.
The first 22 patches massage the KVM code to deal with VHE and enable
Linux to run at EL2. The last patch catches an ugly case when VHE
capable CPUs are paired with some of their less capable siblings. This
should never happen, but hey...
I have deliberately left out some of the more "advanced"
optimizations, as they are likely to distract the reviewer from the
core infrastructure, which is what I care about at the moment.
Note: GDB is currently busted on VHE systems, as it checks for version
6 on the debug architecture, while VHE is version 7. The
binutils people are on the case.
This has been tested on the FVP_Base_SLV-V8-A model, and based on
v4.5-rc3 + kvmarm/master. I've put a branch out on:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe
* From v3:
- Cleaned debug handling after comments from Catalin
- Added missing barriers in the panic patch
- Collected Rb and Acks from Christoffer and Catalin
* From v2:
- Added support for perf to count kernel events in EL2
- Added support for EL2 breakpoints
- Moved the VTCR_EL2 setup from assembly to C
- Made the fault handling easier to understand (hopefuly)
- Plenty of smaller fixups
* From v1:
- Full rewrite now that the World Switch is written in C code.
- Dropped the "early IRQ handling" for the moment.
Marc Zyngier (23):
arm/arm64: KVM: Add hook for C-based stage2 init
arm64: KVM: Switch to C-based stage2 init
arm/arm64: Add new is_kernel_in_hyp_mode predicate
arm64: Allow the arch timer to use the HYP timer
arm64: Add ARM64_HAS_VIRT_HOST_EXTN feature
arm64: KVM: Skip HYP setup when already running in HYP
arm64: KVM: VHE: Patch out use of HVC
arm64: KVM: VHE: Patch out kern_hyp_va
arm64: KVM: VHE: Introduce unified system register accessors
arm64: KVM: VHE: Differenciate host/guest sysreg save/restore
arm64: KVM: VHE: Split save/restore of registers shared between guest
and host
arm64: KVM: VHE: Use unified system register accessors
arm64: KVM: VHE: Enable minimal sysreg save/restore
arm64: KVM: VHE: Make __fpsimd_enabled VHE aware
arm64: KVM: VHE: Implement VHE activate/deactivate_traps
arm64: KVM: VHE: Use unified sysreg accessors for timer
arm64: KVM: VHE: Add fpsimd enabling on guest access
arm64: KVM: VHE: Add alternative panic handling
arm64: KVM: Move most of the fault decoding to C
arm64: perf: Count EL2 events if the kernel is running in HYP
arm64: hw_breakpoint: Allow EL2 breakpoints if running in HYP
arm64: VHE: Add support for running Linux in EL2 mode
arm64: Panic when VHE and non VHE CPUs coexist
arch/arm/include/asm/kvm_host.h | 4 +
arch/arm/include/asm/virt.h | 5 +
arch/arm/kvm/arm.c | 174 ++++++++++++++++++----------
arch/arm/kvm/mmu.c | 7 ++
arch/arm64/Kconfig | 13 +++
arch/arm64/include/asm/cpufeature.h | 3 +-
arch/arm64/include/asm/hw_breakpoint.h | 49 +++++---
arch/arm64/include/asm/kvm_arm.h | 6 +-
arch/arm64/include/asm/kvm_asm.h | 2 +
arch/arm64/include/asm/kvm_emulate.h | 3 +
arch/arm64/include/asm/kvm_host.h | 6 +
arch/arm64/include/asm/kvm_mmu.h | 12 +-
arch/arm64/include/asm/virt.h | 27 +++++
arch/arm64/kernel/asm-offsets.c | 3 -
arch/arm64/kernel/cpufeature.c | 11 ++
arch/arm64/kernel/head.S | 50 +++++++-
arch/arm64/kernel/perf_event.c | 14 ++-
arch/arm64/kernel/smp.c | 3 +
arch/arm64/kvm/hyp-init.S | 18 ---
arch/arm64/kvm/hyp.S | 7 ++
arch/arm64/kvm/hyp/Makefile | 1 +
arch/arm64/kvm/hyp/entry.S | 6 +
arch/arm64/kvm/hyp/hyp-entry.S | 109 ++++++------------
arch/arm64/kvm/hyp/hyp.h | 108 ++++++++++++++++--
arch/arm64/kvm/hyp/s2-setup.c | 44 +++++++
arch/arm64/kvm/hyp/switch.c | 202 +++++++++++++++++++++++++++++----
arch/arm64/kvm/hyp/sysreg-sr.c | 147 ++++++++++++++++--------
arch/arm64/kvm/hyp/timer-sr.c | 10 +-
drivers/clocksource/arm_arch_timer.c | 96 ++++++++++------
29 files changed, 842 insertions(+), 298 deletions(-)
create mode 100644 arch/arm64/kvm/hyp/s2-setup.c
--
2.1.4
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:15
As we're about to move the stage2 init to C code, introduce some
C hooks that will later be populated with arch-specific implementations.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm/include/asm/kvm_host.h | 4 ++++
arch/arm/kvm/arm.c | 1 +
arch/arm64/include/asm/kvm_host.h | 4 ++++
3 files changed, 9 insertions(+)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:19
With the ARMv8.1 VHE, the kernel can run in HYP mode, and thus
use the HYP timer instead of the normal guest timer in a mostly
transparent way, except for the interrupt line.
This patch reworks the arch timer code to allow the selection of
the HYP PPI, possibly falling back to the guest timer if not
available.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
drivers/clocksource/arm_arch_timer.c | 96 ++++++++++++++++++++++--------------
1 file changed, 59 insertions(+), 37 deletions(-)
@@ -460,7 +469,7 @@ static void __init arch_counter_register(unsigned type)/* Register the CP15 based counter if we have one */if(type&ARCH_CP15_TIMER){-if(IS_ENABLED(CONFIG_ARM64)||arch_timer_use_virtual)+if(IS_ENABLED(CONFIG_ARM64)||arch_timer_uses_ppi==VIRT_PPI)arch_timer_read_counter=arch_counter_get_cntvct;elsearch_timer_read_counter=arch_counter_get_cntpct;
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:22
With ARMv8.1 VHE extension, it will be possible to run the kernel
at EL2 (aka HYP mode). In order for the kernel to easily find out
where it is running, add a new predicate that returns whether or
not the kernel is in HYP mode.
For completeness, the 32bit code also get such a predicate (always
returning false) so that code common to both architecture (timers,
KVM) can use it transparently.
Acked-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm/include/asm/virt.h | 5 +++++
arch/arm64/include/asm/virt.h | 10 ++++++++++
2 files changed, 15 insertions(+)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:26
The kern_hyp_va macro is pretty meaninless with VHE, as there is
only one mapping - the kernel one.
In order to keep the code readable and efficient, use runtime
patching to replace the 'and' instruction used to compute the VA
with a 'nop'.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/kvm_mmu.h | 12 +++++++++++-
arch/arm64/kvm/hyp/hyp.h | 25 ++++++++++++++++++++++---
2 files changed, 33 insertions(+), 4 deletions(-)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:30
A handful of system registers are still shared between host and guest,
even while using VHE (tpidr*_el[01] and actlr_el1).
Also, some of the vcpu state (sp_el0, PC and PSTATE) must be
save/restored on entry/exit, as they are used on the host as well.
In order to facilitate the introduction of a VHE-specific sysreg
save/restore, make move the access to these registers to their
own save/restore functions.
No functional change.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kvm/hyp/sysreg-sr.c | 48 +++++++++++++++++++++++++++++-------------
1 file changed, 33 insertions(+), 15 deletions(-)
@@ -23,13 +23,29 @@#include"hyp.h"-/* ctxt is already in the HYP VA space */+/*+*Non-VHE:Bothhostandguestmustsaveeverything.+*+*VHE:Hostmustsavetpidr*_el[01],actlr_el1,sp0,pc,pstate,and+*guestmustsaveeverything.+*/++staticvoid__hyp_text__sysreg_save_common_state(structkvm_cpu_context*ctxt)+{+ctxt->sys_regs[ACTLR_EL1]=read_sysreg(actlr_el1);+ctxt->sys_regs[TPIDR_EL0]=read_sysreg(tpidr_el0);+ctxt->sys_regs[TPIDRRO_EL0]=read_sysreg(tpidrro_el0);+ctxt->sys_regs[TPIDR_EL1]=read_sysreg(tpidr_el1);+ctxt->gp_regs.regs.sp=read_sysreg(sp_el0);+ctxt->gp_regs.regs.pc=read_sysreg(elr_el2);+ctxt->gp_regs.regs.pstate=read_sysreg(spsr_el2);+}+staticvoid__hyp_text__sysreg_save_state(structkvm_cpu_context*ctxt){ctxt->sys_regs[MPIDR_EL1]=read_sysreg(vmpidr_el2);ctxt->sys_regs[CSSELR_EL1]=read_sysreg(csselr_el1);ctxt->sys_regs[SCTLR_EL1]=read_sysreg(sctlr_el1);-ctxt->sys_regs[ACTLR_EL1]=read_sysreg(actlr_el1);ctxt->sys_regs[CPACR_EL1]=read_sysreg(cpacr_el1);ctxt->sys_regs[TTBR0_EL1]=read_sysreg(ttbr0_el1);ctxt->sys_regs[TTBR1_EL1]=read_sysreg(ttbr1_el1);
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:33
We're now in a position where we can introduce VHE's minimal
save/restore, which is limited to the handful of shared sysregs.
Add the required alternative function calls that result in a
"do nothing" call on VHE, and the normal save/restore for non-VHE.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kvm/hyp/sysreg-sr.c | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
@@ -23,6 +23,9 @@#include"hyp.h"+/* Yes, this does nothing, on purpose */+staticvoid__hyp_text__sysreg_do_nothing(structkvm_cpu_context*ctxt){}+/**Non-VHE:Bothhostandguestmustsaveeverything.*
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:37
Running the kernel in HYP mode requires the HCR_E2H bit to be set
at all times, and the HCR_TGE bit to be set when running as a host
(and cleared when running as a guest). At the same time, the vector
must be set to the current role of the kernel (either host or
hypervisor), and a couple of system registers differ between VHE
and non-VHE.
We implement these by using another set of alternate functions
that get dynamically patched.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/kvm_arm.h | 3 +-
arch/arm64/include/asm/kvm_emulate.h | 3 ++
arch/arm64/kvm/hyp/switch.c | 53 ++++++++++++++++++++++++++++++++----
3 files changed, 52 insertions(+), 7 deletions(-)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:42
The fault decoding process (including computing the IPA in the case
of a permission fault) would be much better done in C code, as we
have a reasonable infrastructure to deal with the VHE/non-VHE
differences.
Let's move the whole thing to C, including the workaround for
erratum 834220, and just patch the odd ESR_EL2 access remaining
in hyp-entry.S.
Reviewed-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kernel/asm-offsets.c | 3 --
arch/arm64/kvm/hyp/hyp-entry.S | 69 +++------------------------------
arch/arm64/kvm/hyp/switch.c | 85 +++++++++++++++++++++++++++++++++++++++++
3 files changed, 90 insertions(+), 67 deletions(-)
@@ -105,72 +108,10 @@ el1_trap:cmpx2,#ESR_ELx_EC_FP_ASIMDb.eq__fpsimd_guest_restore-cmpx2,#ESR_ELx_EC_DABT_LOW-movx0,#ESR_ELx_EC_IABT_LOW-ccmpx2,x0,#4, ne-b.ne1f//Notanabortwecareabout--/*Thisisanabort.Checkforpermissionfault*/-alternative_if_notARM64_WORKAROUND_834220-andx2,x1,#ESR_ELx_FSC_TYPE-cmpx2,#FSC_PERM-b.ne1f//Notapermissionfault-alternative_else-nop//Usethepermissionfaultpathto-nop//checkforavalidS1translation,-nop//regardlessoftheESRvalue.-alternative_endif--/*-*CheckforStage-1pagetablewalk,whichisguaranteed-*togiveavalidHPFAR_EL2.-*/-tbnzx1,#7, 1f // S1PTW is set--/*PreservePAR_EL1*/-mrsx3,par_el1-stpx3,xzr,[sp,#-16]!--/*-*Permissionfault,HPFAR_EL2isinvalid.-*ResolvetheIPAthehardwayusingtheguestVA.-*Stage-1translationalreadyvalidatedthememoryaccessrights.-*Assuch,wecanusetheEL1translationregime,anddon't have-*todistinguishbetweenEL0andEL1access.-*/-mrsx2,far_el2-ats1e1r,x2-isb--/*Readresult*/-mrsx3,par_el1-ldpx0,xzr,[sp],#16 // Restore PAR_EL1 from the stack-msrpar_el1,x0-tbnzx3,#0, 3f // Bail out if we failed the translation-ubfxx3,x3,#12, #36 // Extract IPA-lslx3,x3,#4 // and present it like HPFAR-b2f--1:mrsx3,hpfar_el2-mrsx2,far_el2--2:mrsx0,tpidr_el2-strw1,[x0,#VCPU_ESR_EL2]-strx2,[x0,#VCPU_FAR_EL2]-strx3,[x0,#VCPU_HPFAR_EL2]-+mrsx0,tpidr_el2movx1,#ARM_EXCEPTION_TRAPb__guest_exit-/*-*Translationfailed.Justreturntotheguestand-*letitfaultagain.AnotherCPUisprobablyplaying-*behindourback.-*/-3:restore_x0_to_x3--eret-el1_irq:save_x0_to_x3mrsx0,tpidr_el2
@@ -149,6 +150,86 @@ static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu)__vgic_call_restore_state()(vcpu);}+staticbool__hyp_text__true_value(void)+{+returntrue;+}++staticbool__hyp_text__false_value(void)+{+returnfalse;+}++statichyp_alternate_select(__check_arm_834220,+__false_value,__true_value,+ARM64_WORKAROUND_834220);++staticbool__hyp_text__translate_far_to_hpfar(u64far,u64*hpfar)+{+u64par,tmp;++/*+*ResolvetheIPAthehardwayusingtheguestVA.+*+*Stage-1translationalreadyvalidatedthememoryaccess+*rights.Assuch,wecanusetheEL1translationregime,and+*don'thavetodistinguishbetweenEL0andEL1access.+*+*Wedoneedtosave/restorePAR_EL1though,aswehaven't+*savedtheguestcontextyet,andwemayreturnearly...+*/+par=read_sysreg(par_el1);+asmvolatile("at s1e1r, %0"::"r"(far));+isb();++tmp=read_sysreg(par_el1);+write_sysreg(par,par_el1);++if(unlikely(tmp&1))+returnfalse;/* Translation failed, back to guest */++/* Convert PAR to HPFAR format */+*hpfar=((tmp>>12)&((1UL<<36)-1))<<4;+returntrue;+}++staticbool__hyp_text__populate_fault_info(structkvm_vcpu*vcpu)+{+u64esr=read_sysreg_el2(esr);+u8ec=esr>>ESR_ELx_EC_SHIFT;+u64hpfar,far;++vcpu->arch.fault.esr_el2=esr;++if(ec!=ESR_ELx_EC_DABT_LOW&&ec!=ESR_ELx_EC_IABT_LOW)+returntrue;++far=read_sysreg_el2(far);++/*+*TheHPFARcanbeinvalidifthestage2faultdidnot+*happenduringastage1pagetablewalk(theESR_EL2.S1PTW+*bitisclear)andoneofthetwofollowingcasesaretrue:+*1.Thefaultwasduetoapermissionfault+*2.Theprocessorcarrieserrata834220+*+*Therefore,forallnonS1PTWfaultswhereweeitherhavea+*permissionfaultortheerrataworkaroundisenabled,we+*resolvetheIPAusingtheATinstruction.+*/+if(!(esr&ESR_ELx_S1PTW)&&+(__check_arm_834220()()||(esr&ESR_ELx_FSC_TYPE)==FSC_PERM)){+if(!__translate_far_to_hpfar(far,&hpfar))+returnfalse;+}else{+hpfar=read_sysreg(hpfar_el2);+}++vcpu->arch.fault.far_el2=far;+vcpu->arch.fault.hpfar_el2=hpfar;+returntrue;+}+staticint__hyp_text__guest_run(structkvm_vcpu*vcpu){structkvm_cpu_context*host_ctxt;
@@ -180,9 +261,13 @@ static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)__debug_restore_state(vcpu,kern_hyp_va(vcpu->arch.debug_ptr),guest_ctxt);/* Jump in the fire! */+again:exit_code=__guest_enter(vcpu,host_ctxt);/* And we're baaack! */+if(exit_code==ARM_EXCEPTION_TRAP&&!__populate_fault_info(vcpu))+gotoagain;+fp_enabled=__fpsimd_enabled();__sysreg_save_guest_state(guest_ctxt);
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:45
With VHE, we place kernel {watch,break}-points at EL2 to get things
like kgdb and "perf -e mem:..." working.
This requires a bit of repainting in the low-level encore/decode,
but is otherwise pretty simple.
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/hw_breakpoint.h | 49 +++++++++++++++++++++-------------
1 file changed, 31 insertions(+), 18 deletions(-)
On Thu, Feb 11, 2016 at 06:40:02PM +0000, Marc Zyngier wrote:
With VHE, we place kernel {watch,break}-points at EL2 to get things
like kgdb and "perf -e mem:..." working.
This requires a bit of repainting in the low-level encore/decode,
but is otherwise pretty simple.
Signed-off-by: Marc Zyngier <redacted>
To the best of my knowledge, this patch is fine ;)
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
From: Will Deacon <hidden> Date: 2016-02-15 17:46:54
On Thu, Feb 11, 2016 at 06:40:02PM +0000, Marc Zyngier wrote:
quoted hunk
With VHE, we place kernel {watch,break}-points at EL2 to get things
like kgdb and "perf -e mem:..." working.
This requires a bit of repainting in the low-level encore/decode,
but is otherwise pretty simple.
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/hw_breakpoint.h | 49 +++++++++++++++++++++-------------
1 file changed, 31 insertions(+), 18 deletions(-)
I don't think this is correct. We want to allow, for example, a userspace
watchpoint to fire thanks to something like put_user, so the encoding
really needs to build up the PMC field (like we do already), then orr in
the HMC field.
The "gotcha", which is similar to the PMU stuff, is that you can't have
HMC==1 (EL2) and PMC==2 (i.e. EL2 and EL0, but not EL1).
I *think* the conclusion is that you need AARCH64_BREAKPOINT_EL2 to look
like DBG_HMC_HYP | AARCH64_BREAKPOINT_EL1.
Will
From: Will Deacon <hidden> Date: 2016-02-15 19:07:45
On Mon, Feb 15, 2016 at 05:46:56PM +0000, Will Deacon wrote:
On Thu, Feb 11, 2016 at 06:40:02PM +0000, Marc Zyngier wrote:
quoted
With VHE, we place kernel {watch,break}-points at EL2 to get things
like kgdb and "perf -e mem:..." working.
This requires a bit of repainting in the low-level encore/decode,
but is otherwise pretty simple.
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/hw_breakpoint.h | 49 +++++++++++++++++++++-------------
1 file changed, 31 insertions(+), 18 deletions(-)
I don't think this is correct. We want to allow, for example, a userspace
watchpoint to fire thanks to something like put_user, so the encoding
really needs to build up the PMC field (like we do already), then orr in
the HMC field.
Hmm, I got my arm and my arm64 mixed up here. For the latter, we don't
actually support EL0+EL1 watchpoints, but I still think that the
{HMC,SSC,PMC} encoding of {1,00,xx} is cleaner.
Will
From: Marc Zyngier <hidden> Date: 2016-02-16 09:55:47
On 15/02/16 19:07, Will Deacon wrote:
On Mon, Feb 15, 2016 at 05:46:56PM +0000, Will Deacon wrote:
quoted
On Thu, Feb 11, 2016 at 06:40:02PM +0000, Marc Zyngier wrote:
quoted
With VHE, we place kernel {watch,break}-points at EL2 to get things
like kgdb and "perf -e mem:..." working.
This requires a bit of repainting in the low-level encore/decode,
but is otherwise pretty simple.
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/hw_breakpoint.h | 49 +++++++++++++++++++++-------------
1 file changed, 31 insertions(+), 18 deletions(-)
I don't think this is correct. We want to allow, for example, a userspace
watchpoint to fire thanks to something like put_user, so the encoding
really needs to build up the PMC field (like we do already), then orr in
the HMC field.
Hmm, I got my arm and my arm64 mixed up here. For the latter, we don't
actually support EL0+EL1 watchpoints, but I still think that the
{HMC,SSC,PMC} encoding of {1,00,xx} is cleaner.
Indeed. So with that in mind, I ended up with the following patch, which is much simpler than the previous one:
From: Marc Zyngier <hidden> Date: 2016-02-17 17:57:58
When the kernel is running in HYP (with VHE), it is necessary to
include EL2 events if the user requests counting kernel or
hypervisor events.
Reviewed-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kernel/perf_event.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
From: Will Deacon <hidden> Date: 2016-02-19 14:30:26
On Wed, Feb 17, 2016 at 05:57:39PM +0000, Marc Zyngier wrote:
When the kernel is running in HYP (with VHE), it is necessary to
include EL2 events if the user requests counting kernel or
hypervisor events.
Reviewed-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kernel/perf_event.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:40:50
Having both VHE and non-VHE capable CPUs in the same system
is likely to be a recipe for disaster.
If the boot CPU has VHE, but a secondary is not, we won't be
able to downgrade and run the kernel at EL1. Add CPU hotplug
to the mix, and this produces a terrifying mess.
Let's solve the problem once and for all. If you mix VHE and
non-VHE CPUs in the same system, you deserve to loose, and this
patch makes sure you don't get a chance.
This is implemented by storing the kernel execution level in
a global variable. Secondaries will park themselves in a
WFI loop if they observe a mismatch. Also, the primary CPU
will detect that the secondary CPU has died on a mismatched
execution level. Panic will follow.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/virt.h | 17 +++++++++++++++++
arch/arm64/kernel/head.S | 22 ++++++++++++++++++++++
arch/arm64/kernel/smp.c | 3 +++
3 files changed, 42 insertions(+)
@@ -60,6 +65,18 @@ static inline bool is_kernel_in_hyp_mode(void)returnel==CurrentEL_EL2;}+staticinlineboolis_kernel_mode_mismatched(void)+{+/*+*AmismatchedCPUwillhavewrittenitsownCurrentELin+*__run_cpu_mode[1](initiallysettozero)afterfailingto+*matchthevaluein__run_cpu_mode[0].Thus,anon-zero+*valuein__run_cpu_mode[1]isenoughtodetectthe+*pathologicalcase.+*/+return!!ACCESS_ONCE(__run_cpu_mode[1]);+}+/* The section containing the hypervisor text */externchar__hyp_text_start[];externchar__hyp_text_end[];
@@ -113,6 +113,9 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)pr_crit("CPU%u: failed to come online\n",cpu);ret=-EIO;}++if(is_kernel_mode_mismatched())+panic("CPU%u: incompatible execution level",cpu);}else{pr_err("CPU%u: failed to boot: %d\n",cpu,ret);}
From: Will Deacon <hidden> Date: 2016-02-15 17:26:50
On Thu, Feb 11, 2016 at 06:40:04PM +0000, Marc Zyngier wrote:
Having both VHE and non-VHE capable CPUs in the same system
is likely to be a recipe for disaster.
If the boot CPU has VHE, but a secondary is not, we won't be
able to downgrade and run the kernel at EL1. Add CPU hotplug
to the mix, and this produces a terrifying mess.
Let's solve the problem once and for all. If you mix VHE and
non-VHE CPUs in the same system, you deserve to loose, and this
patch makes sure you don't get a chance.
This is implemented by storing the kernel execution level in
a global variable. Secondaries will park themselves in a
WFI loop if they observe a mismatch. Also, the primary CPU
will detect that the secondary CPU has died on a mismatched
execution level. Panic will follow.
This should really be based on Suzuki's series for handling generic
mismatches:
http://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/401727.html
To avoid growing a dependency on something that's unlikely to make it
for 4.6, I'd be inclined to drop your homegrown checks altogether amd
help Suzuki with his series as a separate activity (i.e. it needn't be
a blocker imo).
Will
From: Marc Zyngier <hidden> Date: 2016-02-15 18:14:30
On 15/02/16 17:26, Will Deacon wrote:
On Thu, Feb 11, 2016 at 06:40:04PM +0000, Marc Zyngier wrote:
quoted
Having both VHE and non-VHE capable CPUs in the same system
is likely to be a recipe for disaster.
If the boot CPU has VHE, but a secondary is not, we won't be
able to downgrade and run the kernel at EL1. Add CPU hotplug
to the mix, and this produces a terrifying mess.
Let's solve the problem once and for all. If you mix VHE and
non-VHE CPUs in the same system, you deserve to loose, and this
patch makes sure you don't get a chance.
This is implemented by storing the kernel execution level in
a global variable. Secondaries will park themselves in a
WFI loop if they observe a mismatch. Also, the primary CPU
will detect that the secondary CPU has died on a mismatched
execution level. Panic will follow.
This should really be based on Suzuki's series for handling generic
mismatches:
http://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/401727.html
To avoid growing a dependency on something that's unlikely to make it
for 4.6, I'd be inclined to drop your homegrown checks altogether amd
help Suzuki with his series as a separate activity (i.e. it needn't be
a blocker imo).
That's fine, I'll drop that one - we can revisit it and fold it into
Suzuki's series.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
From: Marc Zyngier <hidden> Date: 2016-02-11 18:41:23
With ARMv8.1 VHE, the architecture is able to (almost) transparently
run the kernel at EL2, despite being written for EL1.
This patch takes care of the "almost" part, mostly preventing the kernel
from dropping from EL2 to EL1, and setting up the HYP configuration.
Reviewed-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/Kconfig | 13 +++++++++++++
arch/arm64/kernel/head.S | 28 +++++++++++++++++++++++++++-
2 files changed, 40 insertions(+), 1 deletion(-)
@@ -464,9 +465,27 @@ CPU_LE( bic x0, x0, #(3 << 24) ) // Clear the EE and E0E bits for EL1isbret+2:+#ifdef CONFIG_ARM64_VHE+/*+*CheckforVHEbeingpresent.FortherestoftheEL2setup,+*x2beingnon-zeroindicatesthatwedohaveVHE,andthatthe+*kernelisintendedtorun@EL2.+*/+mrsx2,id_aa64mmfr1_el1+ubfxx2,x2,#8, #4+#else+movx2,xzr+#endif+/*Hypconfiguration.*/-2:movx0,#(1 << 31) // 64-bit EL1+movx0,#HCR_RW // 64-bit EL1+cbzx2,set_hcr+orrx0,x0,#HCR_TGE // Enable Host Extensions+orrx0,x0,#HCR_E2H+set_hcr:msrhcr_el2,x0+isb/*Generictimers.*/mrsx0,cnthctl_el2
@@ -526,6 +545,13 @@ CPU_LE( movk x0, #0x30d0, lsl #16 ) // Clear EE and E0E on LE systems/*Stage-2translation*/msrvttbr_el2,xzr+cbzx2,install_el2_stub++movw20,#BOOT_CPU_MODE_EL2 // This CPU booted in EL2+isb+ret++install_el2_stub:/*Hypervisorstub*/adrpx0,__hyp_stub_vectorsaddx0,x0,#:lo12:__hyp_stub_vectors
From: Marc Zyngier <hidden> Date: 2016-02-17 17:58:53
With VHE, we place kernel {watch,break}-points at EL2 to get things
like kgdb and "perf -e mem:..." working.
This requires a bit of repainting in the low-level encore/decode,
but is otherwise pretty simple.
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/hw_breakpoint.h | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
From: Will Deacon <hidden> Date: 2016-02-19 14:30:43
On Wed, Feb 17, 2016 at 05:58:43PM +0000, Marc Zyngier wrote:
With VHE, we place kernel {watch,break}-points at EL2 to get things
like kgdb and "perf -e mem:..." working.
This requires a bit of repainting in the low-level encore/decode,
but is otherwise pretty simple.
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/hw_breakpoint.h | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:42:37
When the kernel is running in HYP (with VHE), it is necessary to
include EL2 events if the user requests counting kernel or
hypervisor events.
Reviewed-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kernel/perf_event.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
From: Will Deacon <hidden> Date: 2016-02-15 17:22:44
On Thu, Feb 11, 2016 at 06:40:01PM +0000, Marc Zyngier wrote:
quoted hunk
When the kernel is running in HYP (with VHE), it is necessary to
include EL2 events if the user requests counting kernel or
hypervisor events.
Reviewed-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kernel/perf_event.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
@@ -693,10 +694,15 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,return-EPERM;if(attr->exclude_user)config_base|=ARMV8_EXCLUDE_EL0;-if(attr->exclude_kernel)-config_base|=ARMV8_EXCLUDE_EL1;-if(!attr->exclude_hv)-config_base|=ARMV8_INCLUDE_EL2;+if(is_kernel_in_hyp_mode()){+if(!attr->exclude_kernel||!attr->exclude_hv)+config_base|=ARMV8_INCLUDE_EL2;
Hmm, so if userspace sets exclude_kernel but not exclude_user and
exclude_hv, what should we do? I'm slightly tempted to reject the
filter with -EINVAL...
Will
From: Marc Zyngier <hidden> Date: 2016-02-15 18:23:40
On 15/02/16 17:22, Will Deacon wrote:
On Thu, Feb 11, 2016 at 06:40:01PM +0000, Marc Zyngier wrote:
quoted
When the kernel is running in HYP (with VHE), it is necessary to
include EL2 events if the user requests counting kernel or
hypervisor events.
Reviewed-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kernel/perf_event.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
@@ -693,10 +694,15 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,return-EPERM;if(attr->exclude_user)config_base|=ARMV8_EXCLUDE_EL0;-if(attr->exclude_kernel)-config_base|=ARMV8_EXCLUDE_EL1;-if(!attr->exclude_hv)-config_base|=ARMV8_INCLUDE_EL2;+if(is_kernel_in_hyp_mode()){+if(!attr->exclude_kernel||!attr->exclude_hv)+config_base|=ARMV8_INCLUDE_EL2;
Hmm, so if userspace sets exclude_kernel but not exclude_user and
exclude_hv, what should we do? I'm slightly tempted to reject the
filter with -EINVAL...
I was angling for the minimum level of surprise for the user, but
I guess that the results are going to be troubling anyway.
How about something like this on top:
@@ -695,7 +695,10 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,if(attr->exclude_user)config_base|=ARMV8_EXCLUDE_EL0;if(is_kernel_in_hyp_mode()){-if(!attr->exclude_kernel||!attr->exclude_hv)+/* Demand that kernel and hv are consistent */+if(attr->exclude_kernel!=attr->exclude_hv)+return-EINVAL;+if(!attr->exclude_hv)config_base|=ARMV8_INCLUDE_EL2;}else{if(attr->exclude_kernel)
Thanks,
M.
--
Jazz is not dead. It just smells funny...
From: Will Deacon <hidden> Date: 2016-02-15 18:27:48
On Mon, Feb 15, 2016 at 06:23:35PM +0000, Marc Zyngier wrote:
quoted hunk
On 15/02/16 17:22, Will Deacon wrote:
quoted
On Thu, Feb 11, 2016 at 06:40:01PM +0000, Marc Zyngier wrote:
quoted
When the kernel is running in HYP (with VHE), it is necessary to
include EL2 events if the user requests counting kernel or
hypervisor events.
Reviewed-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kernel/perf_event.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
@@ -693,10 +694,15 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,return-EPERM;if(attr->exclude_user)config_base|=ARMV8_EXCLUDE_EL0;-if(attr->exclude_kernel)-config_base|=ARMV8_EXCLUDE_EL1;-if(!attr->exclude_hv)-config_base|=ARMV8_INCLUDE_EL2;+if(is_kernel_in_hyp_mode()){+if(!attr->exclude_kernel||!attr->exclude_hv)+config_base|=ARMV8_INCLUDE_EL2;
Hmm, so if userspace sets exclude_kernel but not exclude_user and
exclude_hv, what should we do? I'm slightly tempted to reject the
filter with -EINVAL...
I was angling for the minimum level of surprise for the user, but
I guess that the results are going to be troubling anyway.
How about something like this on top:
@@ -695,7 +695,10 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,if(attr->exclude_user)config_base|=ARMV8_EXCLUDE_EL0;if(is_kernel_in_hyp_mode()){-if(!attr->exclude_kernel||!attr->exclude_hv)+/* Demand that kernel and hv are consistent */+if(attr->exclude_kernel!=attr->exclude_hv)+return-EINVAL;+if(!attr->exclude_hv)config_base|=ARMV8_INCLUDE_EL2;}else{if(attr->exclude_kernel)
Looks like the right idea, and you can probably refactor things slightly
to avoid having two codepaths dealing with ARMV8_INCLUDE_EL2.
Note that I've got a couple of outstanding questions with the architects
that I should have answers to later in the week.
Will
From: Marc Zyngier <hidden> Date: 2016-02-11 18:43:29
As the kernel fully runs in HYP when VHE is enabled, we can
directly branch to the kernel's panic() implementation, and
not perform an exception return.
Add the alternative code to deal with this.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kvm/hyp/switch.c | 35 +++++++++++++++++++++++++++--------
1 file changed, 27 insertions(+), 8 deletions(-)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:43:59
Despite the fact that a VHE enabled kernel runs at EL2, it uses
CPACR_EL1 to trap FPSIMD access. Add the required alternative
code to re-enable guest FPSIMD access when it has trapped to
EL2.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kvm/hyp/entry.S | 6 ++++++
1 file changed, 6 insertions(+)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:44:27
As non-VHE and VHE have different ways to express the trapping of
FPSIMD registers to EL2, make __fpsimd_enabled a patchable predicate
and provide a VHE implementation.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/kvm_arm.h | 3 +++
arch/arm64/kvm/hyp/hyp.h | 5 +----
arch/arm64/kvm/hyp/switch.c | 19 +++++++++++++++++++
3 files changed, 23 insertions(+), 4 deletions(-)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:45:05
Use the recently introduced unified system register accessors for
those sysregs that behave differently depending on VHE being in
use or not.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kvm/hyp/sysreg-sr.c | 84 +++++++++++++++++++++---------------------
1 file changed, 42 insertions(+), 42 deletions(-)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:46:00
With ARMv8, host and guest share the same system register file,
making the save/restore procedure completely symetrical.
With VHE, host and guest now have different requirements, as they
use different sysregs.
In order to prepare for this, add split sysreg save/restore functions
for both host and guest. No functional changes yet.
Acked-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kvm/hyp/hyp.h | 6 ++++--
arch/arm64/kvm/hyp/switch.c | 10 +++++-----
arch/arm64/kvm/hyp/sysreg-sr.c | 24 ++++++++++++++++++++++--
3 files changed, 31 insertions(+), 9 deletions(-)
@@ -102,7 +102,7 @@ static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)host_ctxt=kern_hyp_va(vcpu->arch.host_cpu_context);guest_ctxt=&vcpu->arch.ctxt;-__sysreg_save_state(host_ctxt);+__sysreg_save_host_state(host_ctxt);__debug_cond_save_host_state(vcpu);__activate_traps(vcpu);
@@ -116,7 +116,7 @@ static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)*toCortex-A57erratum#852523.*/__sysreg32_restore_state(vcpu);-__sysreg_restore_state(guest_ctxt);+__sysreg_restore_guest_state(guest_ctxt);__debug_restore_state(vcpu,kern_hyp_va(vcpu->arch.debug_ptr),guest_ctxt);/* Jump in the fire! */
@@ -125,7 +125,7 @@ static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)fp_enabled=__fpsimd_enabled();-__sysreg_save_state(guest_ctxt);+__sysreg_save_guest_state(guest_ctxt);__sysreg32_save_state(vcpu);__timer_save_state(vcpu);__vgic_save_state(vcpu);
@@ -133,7 +133,7 @@ static int __hyp_text __guest_run(struct kvm_vcpu *vcpu)__deactivate_traps(vcpu);__deactivate_vm(vcpu);-__sysreg_restore_state(host_ctxt);+__sysreg_restore_host_state(host_ctxt);if(fp_enabled){__fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs);
@@ -165,7 +165,7 @@ void __hyp_text __noreturn __hyp_panic(void)host_ctxt=kern_hyp_va(vcpu->arch.host_cpu_context);__deactivate_traps(vcpu);__deactivate_vm(vcpu);-__sysreg_restore_state(host_ctxt);+__sysreg_restore_host_state(host_ctxt);}/* Call panic for real */
@@ -24,7 +24,7 @@#include"hyp.h"/* ctxt is already in the HYP VA space */-void__hyp_text__sysreg_save_state(structkvm_cpu_context*ctxt)+staticvoid__hyp_text__sysreg_save_state(structkvm_cpu_context*ctxt){ctxt->sys_regs[MPIDR_EL1]=read_sysreg(vmpidr_el2);ctxt->sys_regs[CSSELR_EL1]=read_sysreg(csselr_el1);
From: Marc Zyngier <hidden> Date: 2016-02-11 18:46:42
VHE brings its own bag of new system registers, or rather system
register accessors, as it define new ways to access both guest
and host system registers. For example, from the host:
- The host TCR_EL2 register is accessed using the TCR_EL1 accessor
- The guest TCR_EL1 register is accessed using the TCR_EL12 accessor
Obviously, this is confusing. A way to somehow reduce the complexity
of writing code for both ARMv8 and ARMv8.1 is to use a set of unified
accessors that will generate the right sysreg, depending on the mode
the CPU is running in. For example:
- read_sysreg_el1(tcr) will use TCR_EL1 on ARMv8, and TCR_EL12 on
ARMv8.1 with VHE.
- read_sysreg_el2(tcr) will use TCR_EL2 on ARMv8, and TCR_EL1 on
ARMv8.1 with VHE.
We end up with three sets of accessors ({read,write}_sysreg_el[012])
that can be directly used from C code. We take this opportunity to
also add the definition for the new VHE sysregs.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kvm/hyp/hyp.h | 72 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 72 insertions(+)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:47:01
With VHE, the host never issues an HVC instruction to get into the
KVM code, as we can simply branch there.
Use runtime code patching to simplify things a bit.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/kvm/hyp.S | 7 +++++++
arch/arm64/kvm/hyp/hyp-entry.S | 40 +++++++++++++++++++++++++++++++---------
2 files changed, 38 insertions(+), 9 deletions(-)
@@ -38,6 +38,34 @@ldpx0,x1,[sp],#16.endm+.macrodo_el2_call+/*+*Shuffletheparametersbeforecallingthefunction+*pointedtoinx0.Assumesparametersinx[1,2,3].+*/+subsp,sp,#16+strlr,[sp]+movlr,x0+movx0,x1+movx1,x2+movx2,x3+blrlr+ldrlr,[sp]+addsp,sp,#16+.endm++ENTRY(__vhe_hyp_call)+do_el2_call+/*+*Weusedtorelyonhavinganexceptionreturntoget+*animplicitisb.IntheE2Hcase,wedon't have it anymore.+*ratherthanchangingalltheleaffunctions,justdoithere+*beforereturningtotherestofthekernel.+*/+isb+ret+ENDPROC(__vhe_hyp_call)+el1_sync://GuesttrappedintoEL2save_x0_to_x3
@@ -58,19 +86,13 @@ el1_sync: // Guest trapped into EL2mrsx0,vbar_el2b2f-1:stplr,xzr,[sp,#-16]!-+1:/*-*ComputethefunctionaddressinEL2,andshuffletheparameters.+*PerformtheEL2call*/kern_hyp_vax0-movlr,x0-movx0,x1-movx1,x2-movx2,x3-blrlr+do_el2_call-ldplr,xzr,[sp],#162:eretel1_trap:
From: Marc Zyngier <hidden> Date: 2016-02-11 18:47:31
With the kernel running at EL2, there is no point trying to
configure page tables for HYP, as the kernel is already mapped.
Take this opportunity to refactor the whole init a bit, allowing
the various parts of the hypervisor bringup to be split across
multiple functions.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm/kvm/arm.c | 173 +++++++++++++++++++++++++++++++++++------------------
arch/arm/kvm/mmu.c | 7 +++
2 files changed, 121 insertions(+), 59 deletions(-)
@@ -967,6 +967,11 @@ long kvm_arch_vm_ioctl(struct file *filp,}}+staticvoidcpu_init_stage2(void*dummy)+{+__cpu_init_stage2();+}+staticvoidcpu_init_hyp_mode(void*dummy){phys_addr_tboot_pgd_ptr;
@@ -1036,6 +1041,82 @@ static inline void hyp_cpu_pm_init(void)}#endif+staticvoidteardown_common_resources(void)+{+free_percpu(kvm_host_cpu_state);+}++staticintinit_common_resources(void)+{+kvm_host_cpu_state=alloc_percpu(kvm_cpu_context_t);+if(!kvm_host_cpu_state){+kvm_err("Cannot allocate host CPU state\n");+return-ENOMEM;+}++return0;+}++staticintinit_subsystems(void)+{+interr;++/*+*InitHYPviewofVGIC+*/+err=kvm_vgic_hyp_init();+switch(err){+case0:+vgic_present=true;+break;+case-ENODEV:+case-ENXIO:+vgic_present=false;+break;+default:+returnerr;+}++/*+*InitHYParchitectedtimersupport+*/+err=kvm_timer_hyp_init();+if(err)+returnerr;++kvm_perf_init();+kvm_coproc_table_init();++return0;+}++staticvoidteardown_hyp_mode(void)+{+intcpu;++if(is_kernel_in_hyp_mode())+return;++free_hyp_pgds();+for_each_possible_cpu(cpu)+free_page(per_cpu(kvm_arm_hyp_stack_page,cpu));+}++staticintinit_vhe_mode(void)+{+/*+*ExecutetheinitcodeoneachCPU.+*/+on_each_cpu(cpu_init_stage2,NULL,1);++/* set size of VMID supported by CPU */+kvm_vmid_bits=kvm_get_vmid_bits();+kvm_info("%d-bit VMID\n",kvm_vmid_bits);++kvm_info("VHE mode initialized successfully\n");+return0;+}+/***InitsHyp-modeonallonlineCPUs*/
@@ -1066,7 +1147,7 @@ static int init_hyp_mode(void)stack_page=__get_free_page(GFP_KERNEL);if(!stack_page){err=-ENOMEM;-gotoout_free_stack_pages;+gotoout_err;}per_cpu(kvm_arm_hyp_stack_page,cpu)=stack_page;
@@ -1096,20 +1177,10 @@ static int init_hyp_mode(void)if(err){kvm_err("Cannot map hyp stack\n");-gotoout_free_mappings;+gotoout_err;}}-/*-*MapthehostCPUstructures-*/-kvm_host_cpu_state=alloc_percpu(kvm_cpu_context_t);-if(!kvm_host_cpu_state){-err=-ENOMEM;-kvm_err("Cannot allocate host CPU state\n");-gotoout_free_mappings;-}-for_each_possible_cpu(cpu){kvm_cpu_context_t*cpu_ctxt;
@@ -1118,7 +1189,7 @@ static int init_hyp_mode(void)if(err){kvm_err("Cannot map host CPU state: %d\n",err);-gotoout_free_context;+gotoout_err;}}
@@ -1127,34 +1198,22 @@ static int init_hyp_mode(void)*/on_each_cpu(cpu_init_hyp_mode,NULL,1);-/*-*InitHYPviewofVGIC-*/-err=kvm_vgic_hyp_init();-switch(err){-case0:-vgic_present=true;-break;-case-ENODEV:-case-ENXIO:-vgic_present=false;-break;-default:-gotoout_free_context;-}--/*-*InitHYParchitectedtimersupport-*/-err=kvm_timer_hyp_init();-if(err)-gotoout_free_context;-#ifndef CONFIG_HOTPLUG_CPUfree_boot_hyp_pgd();#endif-kvm_perf_init();+cpu_notifier_register_begin();++err=__register_cpu_notifier(&hyp_init_cpu_nb);++cpu_notifier_register_done();++if(err){+kvm_err("Cannot register HYP init CPU notifier (%d)\n",err);+gotoout_err;+}++hyp_cpu_pm_init();/* set size of VMID supported by CPU */kvm_vmid_bits=kvm_get_vmid_bits();
From: Marc Zyngier <hidden> Date: 2016-02-11 18:48:04
Add a new ARM64_HAS_VIRT_HOST_EXTN features to indicate that the
CPU has the ARMv8.1 VHE capability.
This will be used to trigger kernel patching in KVM.
Acked-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/cpufeature.h | 3 ++-
arch/arm64/kernel/cpufeature.c | 11 +++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
From: Will Deacon <hidden> Date: 2016-02-15 16:10:44
On Thu, Feb 11, 2016 at 06:39:46PM +0000, Marc Zyngier wrote:
quoted hunk
Add a new ARM64_HAS_VIRT_HOST_EXTN features to indicate that the
CPU has the ARMv8.1 VHE capability.
This will be used to trigger kernel patching in KVM.
Acked-by: Christoffer Dall <redacted>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/cpufeature.h | 3 ++-
arch/arm64/kernel/cpufeature.c | 11 +++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
From: Marc Zyngier <hidden> Date: 2016-02-11 18:48:55
There is no real need to leave the stage2 initialization as part
of the early HYP bootstrap, and we can easily postpone it to
the point where we can safely run C code.
This will help VHE, which doesn't need any of this bootstrap.
Reviewed-by: Christoffer Dall <redacted>
Signed-off-by: Marc Zyngier <redacted>
---
arch/arm64/include/asm/kvm_asm.h | 2 ++
arch/arm64/include/asm/kvm_host.h | 2 ++
arch/arm64/kvm/hyp-init.S | 18 ----------------
arch/arm64/kvm/hyp/Makefile | 1 +
arch/arm64/kvm/hyp/s2-setup.c | 44 +++++++++++++++++++++++++++++++++++++++
5 files changed, 49 insertions(+), 18 deletions(-)
create mode 100644 arch/arm64/kvm/hyp/s2-setup.c
From: Christoffer Dall <hidden> Date: 2016-02-11 19:06:23
On Thu, Feb 11, 2016 at 06:39:41PM +0000, Marc Zyngier wrote:
ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
short), which enables simpler support of Type-2 hypervisors.
This extension allows the kernel to directly run at EL2, and
significantly reduces the number of system registers shared between
host and guest, reducing the overhead of virtualization.
In order to have the same kernel binary running on all versions of the
architecture, this series makes heavy use of runtime code patching.
The first 22 patches massage the KVM code to deal with VHE and enable
Linux to run at EL2. The last patch catches an ugly case when VHE
capable CPUs are paired with some of their less capable siblings. This
should never happen, but hey...
I have deliberately left out some of the more "advanced"
optimizations, as they are likely to distract the reviewer from the
core infrastructure, which is what I care about at the moment.
Note: GDB is currently busted on VHE systems, as it checks for version
6 on the debug architecture, while VHE is version 7. The
binutils people are on the case.
This has been tested on the FVP_Base_SLV-V8-A model, and based on
v4.5-rc3 + kvmarm/master. I've put a branch out on:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe
I'm happy with this series as it stands, we just need someone to take a
look at that debug patch and be happy with it.
Thanks,
-Christoffer
From: Marc Zyngier <hidden> Date: 2016-02-11 19:10:55
On 11/02/16 19:07, Christoffer Dall wrote:
On Thu, Feb 11, 2016 at 06:39:41PM +0000, Marc Zyngier wrote:
quoted
ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
short), which enables simpler support of Type-2 hypervisors.
This extension allows the kernel to directly run at EL2, and
significantly reduces the number of system registers shared between
host and guest, reducing the overhead of virtualization.
In order to have the same kernel binary running on all versions of the
architecture, this series makes heavy use of runtime code patching.
The first 22 patches massage the KVM code to deal with VHE and enable
Linux to run at EL2. The last patch catches an ugly case when VHE
capable CPUs are paired with some of their less capable siblings. This
should never happen, but hey...
I have deliberately left out some of the more "advanced"
optimizations, as they are likely to distract the reviewer from the
core infrastructure, which is what I care about at the moment.
Note: GDB is currently busted on VHE systems, as it checks for version
6 on the debug architecture, while VHE is version 7. The
binutils people are on the case.
This has been tested on the FVP_Base_SLV-V8-A model, and based on
v4.5-rc3 + kvmarm/master. I've put a branch out on:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe
I'm happy with this series as it stands, we just need someone to take a
look at that debug patch and be happy with it.
I think Will is planning to have a go at it shortly. Once he and Catalin
are happy with the thing, I'll throw it into -next for a good shake.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
On Thu, Feb 11, 2016 at 07:10:50PM +0000, Marc Zyngier wrote:
On 11/02/16 19:07, Christoffer Dall wrote:
quoted
On Thu, Feb 11, 2016 at 06:39:41PM +0000, Marc Zyngier wrote:
quoted
ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
short), which enables simpler support of Type-2 hypervisors.
This extension allows the kernel to directly run at EL2, and
significantly reduces the number of system registers shared between
host and guest, reducing the overhead of virtualization.
In order to have the same kernel binary running on all versions of the
architecture, this series makes heavy use of runtime code patching.
The first 22 patches massage the KVM code to deal with VHE and enable
Linux to run at EL2. The last patch catches an ugly case when VHE
capable CPUs are paired with some of their less capable siblings. This
should never happen, but hey...
I have deliberately left out some of the more "advanced"
optimizations, as they are likely to distract the reviewer from the
core infrastructure, which is what I care about at the moment.
Note: GDB is currently busted on VHE systems, as it checks for version
6 on the debug architecture, while VHE is version 7. The
binutils people are on the case.
This has been tested on the FVP_Base_SLV-V8-A model, and based on
v4.5-rc3 + kvmarm/master. I've put a branch out on:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe
I'm happy with this series as it stands, we just need someone to take a
look at that debug patch and be happy with it.
I think Will is planning to have a go at it shortly. Once he and Catalin
are happy with the thing, I'll throw it into -next for a good shake.
Fine by me but it would be good for Will to have a look, especially on
the debug and perf stuff.
--
Catalin
From: Marc Zyngier <hidden> Date: 2016-02-15 10:34:25
On 15/02/16 10:23, Catalin Marinas wrote:
On Thu, Feb 11, 2016 at 07:10:50PM +0000, Marc Zyngier wrote:
quoted
On 11/02/16 19:07, Christoffer Dall wrote:
quoted
On Thu, Feb 11, 2016 at 06:39:41PM +0000, Marc Zyngier wrote:
quoted
ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
short), which enables simpler support of Type-2 hypervisors.
This extension allows the kernel to directly run at EL2, and
significantly reduces the number of system registers shared between
host and guest, reducing the overhead of virtualization.
In order to have the same kernel binary running on all versions of the
architecture, this series makes heavy use of runtime code patching.
The first 22 patches massage the KVM code to deal with VHE and enable
Linux to run at EL2. The last patch catches an ugly case when VHE
capable CPUs are paired with some of their less capable siblings. This
should never happen, but hey...
I have deliberately left out some of the more "advanced"
optimizations, as they are likely to distract the reviewer from the
core infrastructure, which is what I care about at the moment.
Note: GDB is currently busted on VHE systems, as it checks for version
6 on the debug architecture, while VHE is version 7. The
binutils people are on the case.
This has been tested on the FVP_Base_SLV-V8-A model, and based on
v4.5-rc3 + kvmarm/master. I've put a branch out on:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe
I'm happy with this series as it stands, we just need someone to take a
look at that debug patch and be happy with it.
I think Will is planning to have a go at it shortly. Once he and Catalin
are happy with the thing, I'll throw it into -next for a good shake.
Fine by me but it would be good for Will to have a look, especially on
the debug and perf stuff.
Definitely. To make it clear, I do not intend to put anything in -next
until all the relevant people has reviewed it, and at least given their Ack.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
From: Alex Bennée <hidden> Date: 2016-02-29 17:43:35
Marc Zyngier [off-list ref] writes:
ARMv8.1 comes with the "Virtualization Host Extension" (VHE for
short), which enables simpler support of Type-2 hypervisors.
This extension allows the kernel to directly run at EL2, and
significantly reduces the number of system registers shared between
host and guest, reducing the overhead of virtualization.
In order to have the same kernel binary running on all versions of the
architecture, this series makes heavy use of runtime code patching.
The first 22 patches massage the KVM code to deal with VHE and enable
Linux to run at EL2. The last patch catches an ugly case when VHE
capable CPUs are paired with some of their less capable siblings. This
should never happen, but hey...
I have deliberately left out some of the more "advanced"
optimizations, as they are likely to distract the reviewer from the
core infrastructure, which is what I care about at the moment.
Note: GDB is currently busted on VHE systems, as it checks for version
6 on the debug architecture, while VHE is version 7. The
binutils people are on the case.
I assume you are talking about ptrace debug here rather than guest
debug?
Anyway from a cursory inspection I didn't see anything to worry about
from the guest debug side which is unchanged.
Acked-by: Alex Benn?e <redacted>
This has been tested on the FVP_Base_SLV-V8-A model, and based on
v4.5-rc3 + kvmarm/master. I've put a branch out on:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/vhe
* From v3:
- Cleaned debug handling after comments from Catalin
- Added missing barriers in the panic patch
- Collected Rb and Acks from Christoffer and Catalin
* From v2:
- Added support for perf to count kernel events in EL2
- Added support for EL2 breakpoints
- Moved the VTCR_EL2 setup from assembly to C
- Made the fault handling easier to understand (hopefuly)
- Plenty of smaller fixups
* From v1:
- Full rewrite now that the World Switch is written in C code.
- Dropped the "early IRQ handling" for the moment.
Marc Zyngier (23):
arm/arm64: KVM: Add hook for C-based stage2 init
arm64: KVM: Switch to C-based stage2 init
arm/arm64: Add new is_kernel_in_hyp_mode predicate
arm64: Allow the arch timer to use the HYP timer
arm64: Add ARM64_HAS_VIRT_HOST_EXTN feature
arm64: KVM: Skip HYP setup when already running in HYP
arm64: KVM: VHE: Patch out use of HVC
arm64: KVM: VHE: Patch out kern_hyp_va
arm64: KVM: VHE: Introduce unified system register accessors
arm64: KVM: VHE: Differenciate host/guest sysreg save/restore
arm64: KVM: VHE: Split save/restore of registers shared between guest
and host
arm64: KVM: VHE: Use unified system register accessors
arm64: KVM: VHE: Enable minimal sysreg save/restore
arm64: KVM: VHE: Make __fpsimd_enabled VHE aware
arm64: KVM: VHE: Implement VHE activate/deactivate_traps
arm64: KVM: VHE: Use unified sysreg accessors for timer
arm64: KVM: VHE: Add fpsimd enabling on guest access
arm64: KVM: VHE: Add alternative panic handling
arm64: KVM: Move most of the fault decoding to C
arm64: perf: Count EL2 events if the kernel is running in HYP
arm64: hw_breakpoint: Allow EL2 breakpoints if running in HYP
arm64: VHE: Add support for running Linux in EL2 mode
arm64: Panic when VHE and non VHE CPUs coexist
arch/arm/include/asm/kvm_host.h | 4 +
arch/arm/include/asm/virt.h | 5 +
arch/arm/kvm/arm.c | 174 ++++++++++++++++++----------
arch/arm/kvm/mmu.c | 7 ++
arch/arm64/Kconfig | 13 +++
arch/arm64/include/asm/cpufeature.h | 3 +-
arch/arm64/include/asm/hw_breakpoint.h | 49 +++++---
arch/arm64/include/asm/kvm_arm.h | 6 +-
arch/arm64/include/asm/kvm_asm.h | 2 +
arch/arm64/include/asm/kvm_emulate.h | 3 +
arch/arm64/include/asm/kvm_host.h | 6 +
arch/arm64/include/asm/kvm_mmu.h | 12 +-
arch/arm64/include/asm/virt.h | 27 +++++
arch/arm64/kernel/asm-offsets.c | 3 -
arch/arm64/kernel/cpufeature.c | 11 ++
arch/arm64/kernel/head.S | 50 +++++++-
arch/arm64/kernel/perf_event.c | 14 ++-
arch/arm64/kernel/smp.c | 3 +
arch/arm64/kvm/hyp-init.S | 18 ---
arch/arm64/kvm/hyp.S | 7 ++
arch/arm64/kvm/hyp/Makefile | 1 +
arch/arm64/kvm/hyp/entry.S | 6 +
arch/arm64/kvm/hyp/hyp-entry.S | 109 ++++++------------
arch/arm64/kvm/hyp/hyp.h | 108 ++++++++++++++++--
arch/arm64/kvm/hyp/s2-setup.c | 44 +++++++
arch/arm64/kvm/hyp/switch.c | 202 +++++++++++++++++++++++++++++----
arch/arm64/kvm/hyp/sysreg-sr.c | 147 ++++++++++++++++--------
arch/arm64/kvm/hyp/timer-sr.c | 10 +-
drivers/clocksource/arm_arch_timer.c | 96 ++++++++++------
29 files changed, 842 insertions(+), 298 deletions(-)
create mode 100644 arch/arm64/kvm/hyp/s2-setup.c