This patch series enables the nested virtualization enlightenments for
SVM. This is very similar to the enlightenments for VMX except for the
fact that there is no enlightened VMCS. For SVM, VMCB is already an
architectural in-memory data structure.
Note: v5 is just a rebase on hyperv-next(5.13-rc1) and needed a rework
based on the patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush)
https://lore.kernel.org/lkml/20210305183123.3978098-1-seanjc@google.com/
The supported enlightenments are:
Enlightened TLB Flush: If this is enabled, ASID invalidations invalidate
only gva -> hpa entries. To flush entries derived from NPT, hyper-v
provided hypercalls (HvFlushGuestPhysicalAddressSpace or
HvFlushGuestPhysicalAddressList) should be used.
Enlightened MSR bitmap(TLFS 16.5.3): "When enabled, L0 hypervisor does
not monitor the MSR bitmaps for changes. Instead, the L1 hypervisor must
invalidate the corresponding clean field after making changes to one of
the MSR bitmaps."
Direct Virtual Flush(TLFS 16.8): The hypervisor exposes hypercalls
(HvFlushVirtualAddressSpace, HvFlushVirtualAddressSpaceEx,
HvFlushVirtualAddressList, and HvFlushVirtualAddressListEx) that allow
operating systems to more efficiently manage the virtual TLB. The L1
hypervisor can choose to allow its guest to use those hypercalls and
delegate the responsibility to handle them to the L0 hypervisor. This
requires the use of a partition assist page."
L2 Windows boot time was measured with and without the patch. Time was
measured from power on to the login screen and was averaged over a
consecutive 5 trials:
Without the patch: 42 seconds
With the patch: 29 seconds
--
Changes from v4
- Rebased on top of 5.13-rc1 and reworked based on the changes in the
patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush)
Changes from v3
- Included definitions for software/hypervisor reserved fields in SVM
architectural data structures.
- Consolidated Hyper-V specific code into svm_onhyperv.[ch] to reduce
the "ifdefs". This change applies only to SVM, VMX is not touched and
is not in the scope of this patch series.
Changes from v2:
- Refactored the Remote TLB Flush logic into separate hyperv specific
source files (kvm_onhyperv.[ch]).
- Reverted the VMCB Clean bits macro changes as it is no longer needed.
Changes from v1:
- Move the remote TLB flush related fields from kvm_vcpu_hv and kvm_hv
to kvm_vcpu_arch and kvm_arch.
- Modify the VMCB clean mask runtime based on whether L1 hypervisor
is running on Hyper-V or not.
- Detect Hyper-V nested enlightenments based on
HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS.
- Address other minor review comments.
---
Vineeth Pillai (7):
hyperv: Detect Nested virtualization support for SVM
hyperv: SVM enlightened TLB flush support flag
KVM: x86: hyper-v: Move the remote TLB flush logic out of vmx
KVM: SVM: Software reserved fields
KVM: SVM: hyper-v: Remote TLB flush for SVM
KVM: SVM: hyper-v: Enlightened MSR-Bitmap support
KVM: SVM: hyper-v: Direct Virtual Flush support
arch/x86/include/asm/hyperv-tlfs.h | 9 ++
arch/x86/include/asm/kvm_host.h | 9 ++
arch/x86/include/asm/svm.h | 9 +-
arch/x86/include/uapi/asm/svm.h | 3 +
arch/x86/kernel/cpu/mshyperv.c | 10 ++-
arch/x86/kvm/Makefile | 9 ++
arch/x86/kvm/kvm_onhyperv.c | 93 +++++++++++++++++++++
arch/x86/kvm/kvm_onhyperv.h | 32 +++++++
arch/x86/kvm/svm/svm.c | 14 ++++
arch/x86/kvm/svm/svm.h | 22 ++++-
arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++
arch/x86/kvm/svm/svm_onhyperv.h | 129 +++++++++++++++++++++++++++++
arch/x86/kvm/vmx/vmx.c | 105 +----------------------
arch/x86/kvm/vmx/vmx.h | 9 --
arch/x86/kvm/x86.c | 9 ++
15 files changed, 384 insertions(+), 119 deletions(-)
create mode 100644 arch/x86/kvm/kvm_onhyperv.c
create mode 100644 arch/x86/kvm/kvm_onhyperv.h
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.h
--
2.25.1
Previously, to detect nested virtualization enlightenment support,
we were using HV_X64_ENLIGHTENED_VMCS_RECOMMENDED feature bit of
HYPERV_CPUID_ENLIGHTMENT_INFO.EAX CPUID as docuemented in TLFS:
"Bit 14: Recommend a nested hypervisor using the enlightened VMCS
interface. Also indicates that additional nested enlightenments
may be available (see leaf 0x4000000A)".
Enlightened VMCS, however, is an Intel only feature so the above
detection method doesn't work for AMD. So, use the
HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS.EAX CPUID information ("The
maximum input value for hypervisor CPUID information.") and this
works for both AMD and Intel.
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/kernel/cpu/mshyperv.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
Bit 22 of HYPERV_CPUID_FEATURES.EDX is specific to SVM and specifies
support for enlightened TLB flush. With this enlightenment enabled,
ASID invalidations flushes only gva->hpa entries. To flush TLB entries
derived from NPT, hypercalls should be used
(HvFlushGuestPhysicalAddressSpace or HvFlushGuestPhysicalAddressList)
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/include/asm/hyperv-tlfs.h | 9 +++++++++
1 file changed, 9 insertions(+)
Currently the remote TLB flush logic is specific to VMX.
Move it to a common place so that SVM can use it as well.
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/include/asm/kvm_host.h | 9 +++
arch/x86/kvm/Makefile | 5 ++
arch/x86/kvm/kvm_onhyperv.c | 93 ++++++++++++++++++++++++++++
arch/x86/kvm/kvm_onhyperv.h | 32 ++++++++++
arch/x86/kvm/vmx/vmx.c | 105 +-------------------------------
arch/x86/kvm/vmx/vmx.h | 9 ---
arch/x86/kvm/x86.c | 9 +++
7 files changed, 150 insertions(+), 112 deletions(-)
create mode 100644 arch/x86/kvm/kvm_onhyperv.c
create mode 100644 arch/x86/kvm/kvm_onhyperv.h
SVM added support for certain reserved fields to be used by
software or hypervisor. Add the following reserved fields:
- VMCB offset 0x3e0 - 0x3ff
- Clean bit 31
- SVM intercept exit code 0xf0000000
Later patches will make use of this for supporting Hyper-V
nested virtualization enhancements.
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/include/asm/svm.h | 9 +++++++--
arch/x86/include/uapi/asm/svm.h | 3 +++
arch/x86/kvm/svm/svm.h | 17 +++++++++++++++--
3 files changed, 25 insertions(+), 4 deletions(-)
Enlightened MSR-Bitmap as per TLFS:
"The L1 hypervisor may collaborate with the L0 hypervisor to make MSR
accesses more efficient. It can enable enlightened MSR bitmaps by setting
the corresponding field in the enlightened VMCS to 1. When enabled, L0
hypervisor does not monitor the MSR bitmaps for changes. Instead, the L1
hypervisor must invalidate the corresponding clean field after making
changes to one of the MSR bitmaps."
Enable this for SVM.
Related VMX changes:
commit ceef7d10dfb6 ("KVM: x86: VMX: hyper-v: Enlightened MSR-Bitmap support")
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/kvm/svm/svm.c | 3 +++
arch/x86/kvm/svm/svm.h | 5 +++++
arch/x86/kvm/svm/svm_onhyperv.h | 27 +++++++++++++++++++++++++++
3 files changed, 35 insertions(+)
From Hyper-V TLFS:
"The hypervisor exposes hypercalls (HvFlushVirtualAddressSpace,
HvFlushVirtualAddressSpaceEx, HvFlushVirtualAddressList, and
HvFlushVirtualAddressListEx) that allow operating systems to more
efficiently manage the virtual TLB. The L1 hypervisor can choose to
allow its guest to use those hypercalls and delegate the responsibility
to handle them to the L0 hypervisor. This requires the use of a
partition assist page."
Add the Direct Virtual Flush support for SVM.
Related VMX changes:
commit 6f6a657c9998 ("KVM/Hyper-V/VMX: Add direct tlb flush support")
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/kvm/Makefile | 4 ++++
arch/x86/kvm/svm/svm.c | 2 ++
arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++++++++++++++++++++++++++
arch/x86/kvm/svm/svm_onhyperv.h | 36 +++++++++++++++++++++++++++++
4 files changed, 83 insertions(+)
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c
From: Michael Kelley <hidden> Date: 2021-06-08 17:00:07
From: Vineeth Pillai <redacted> Sent: Thursday, June 3, 2021 8:15 AM
Previously, to detect nested virtualization enlightenment support,
we were using HV_X64_ENLIGHTENED_VMCS_RECOMMENDED feature bit of
HYPERV_CPUID_ENLIGHTMENT_INFO.EAX CPUID as docuemented in TLFS:
s/docuemented/documented/
quoted hunk
"Bit 14: Recommend a nested hypervisor using the enlightened VMCS
interface. Also indicates that additional nested enlightenments
may be available (see leaf 0x4000000A)".
Enlightened VMCS, however, is an Intel only feature so the above
detection method doesn't work for AMD. So, use the
HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS.EAX CPUID information ("The
maximum input value for hypervisor CPUID information.") and this
works for both AMD and Intel.
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/kernel/cpu/mshyperv.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
Nit: Drop the colon after "Nested features". Current code isn't very consistent
but I'm trying to establish the pattern of "Hyper-V:" followed by names and
values, with multiple name/value pairs separated by a comma.
+ ms_hyperv.nested_features);
}
/*
--
2.25.1
Nits notwithstanding,
Reviewed-by: Michael Kelley <redacted>
From: Michael Kelley <hidden> Date: 2021-06-08 17:04:56
From: Vineeth Pillai <redacted> Sent: Thursday, June 3, 2021 8:15 AM
Bit 22 of HYPERV_CPUID_FEATURES.EDX is specific to SVM and specifies
support for enlightened TLB flush. With this enlightenment enabled,
ASID invalidations flushes only gva->hpa entries. To flush TLB entries
derived from NPT, hypercalls should be used
Nit: Isn't this "must be used"? "Should be used" sounds slightly optional,
and I don't think that's the case.
From Hyper-V TLFS:
"The hypervisor exposes hypercalls (HvFlushVirtualAddressSpace,
HvFlushVirtualAddressSpaceEx, HvFlushVirtualAddressList, and
HvFlushVirtualAddressListEx) that allow operating systems to more
efficiently manage the virtual TLB. The L1 hypervisor can choose to
allow its guest to use those hypercalls and delegate the responsibility
to handle them to the L0 hypervisor. This requires the use of a
partition assist page."
Add the Direct Virtual Flush support for SVM.
Related VMX changes:
commit 6f6a657c9998 ("KVM/Hyper-V/VMX: Add direct tlb flush support")
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/kvm/Makefile | 4 ++++
arch/x86/kvm/svm/svm.c | 2 ++
arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++++++++++++++++++++++++++
arch/x86/kvm/svm/svm_onhyperv.h | 36 +++++++++++++++++++++++++++++
4 files changed, 83 insertions(+)
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c
I would've avoided re-using 'hv_enable_direct_tlbflush()' name which we
already have in vmx. In fact, in the spirit of this patch, I'd suggest
we create arch/x86/kvm/vmx/vmx_onhyperv.c and move the existing
hv_enable_direct_tlbflush() there. We can then re-name it to e.g.
vmx_enable_hv_direct_tlbflush()
so the one introduced by this patch will be
svm_enable_hv_direct_tlbflush()
Currently the remote TLB flush logic is specific to VMX.
Move it to a common place so that SVM can use it as well.
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/include/asm/kvm_host.h | 9 +++
arch/x86/kvm/Makefile | 5 ++
arch/x86/kvm/kvm_onhyperv.c | 93 ++++++++++++++++++++++++++++
arch/x86/kvm/kvm_onhyperv.h | 32 ++++++++++
arch/x86/kvm/vmx/vmx.c | 105 +-------------------------------
arch/x86/kvm/vmx/vmx.h | 9 ---
arch/x86/kvm/x86.c | 9 +++
7 files changed, 150 insertions(+), 112 deletions(-)
create mode 100644 arch/x86/kvm/kvm_onhyperv.c
create mode 100644 arch/x86/kvm/kvm_onhyperv.h
Super-nitpick: I'd suggest adding /* __ARCH_X86_KVM_KVM_ONHYPERV_H__ */
to the second '#endif' and /* IS_ENABLED(CONFIG_HYPERV) */ to '#else'
and the first one: files/functions tend to grow and it becomes hard to
see where the particular '#endif/#else' belongs.
Super-nitpick: I'd suggest adding /* __ARCH_X86_KVM_KVM_ONHYPERV_H__ */
to the second '#endif' and /* IS_ENABLED(CONFIG_HYPERV) */ to '#else'
and the first one: files/functions tend to grow and it becomes hard to
see where the particular '#endif/#else' belongs.
Done, thanks. I've also changed the #if to just "#ifdef CONFIG_HYPERV",
since IS_ENABLED is only needed in C statements.
Paolo
I would've avoided re-using 'hv_enable_direct_tlbflush()' name which we
already have in vmx. In fact, in the spirit of this patch, I'd suggest
we create arch/x86/kvm/vmx/vmx_onhyperv.c and move the existing
hv_enable_direct_tlbflush() there. We can then re-name it to e.g.
vmx_enable_hv_direct_tlbflush()
so the one introduced by this patch will be
svm_enable_hv_direct_tlbflush()
I did the rename, and agree with creating a similar file that is split
off vmx.c.
Paolo
From: Paolo Bonzini <pbonzini@redhat.com> Date: 2021-06-10 15:17:13
On 03/06/21 17:14, Vineeth Pillai wrote:
This patch series enables the nested virtualization enlightenments for
SVM. This is very similar to the enlightenments for VMX except for the
fact that there is no enlightened VMCS. For SVM, VMCB is already an
architectural in-memory data structure.
Note: v5 is just a rebase on hyperv-next(5.13-rc1) and needed a rework
based on the patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush)
https://lore.kernel.org/lkml/20210305183123.3978098-1-seanjc@google.com/
The supported enlightenments are:
Enlightened TLB Flush: If this is enabled, ASID invalidations invalidate
only gva -> hpa entries. To flush entries derived from NPT, hyper-v
provided hypercalls (HvFlushGuestPhysicalAddressSpace or
HvFlushGuestPhysicalAddressList) should be used.
Enlightened MSR bitmap(TLFS 16.5.3): "When enabled, L0 hypervisor does
not monitor the MSR bitmaps for changes. Instead, the L1 hypervisor must
invalidate the corresponding clean field after making changes to one of
the MSR bitmaps."
Direct Virtual Flush(TLFS 16.8): The hypervisor exposes hypercalls
(HvFlushVirtualAddressSpace, HvFlushVirtualAddressSpaceEx,
HvFlushVirtualAddressList, and HvFlushVirtualAddressListEx) that allow
operating systems to more efficiently manage the virtual TLB. The L1
hypervisor can choose to allow its guest to use those hypercalls and
delegate the responsibility to handle them to the L0 hypervisor. This
requires the use of a partition assist page."
L2 Windows boot time was measured with and without the patch. Time was
measured from power on to the login screen and was averaged over a
consecutive 5 trials:
Without the patch: 42 seconds
With the patch: 29 seconds
--
Changes from v4
- Rebased on top of 5.13-rc1 and reworked based on the changes in the
patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush)
Changes from v3
- Included definitions for software/hypervisor reserved fields in SVM
architectural data structures.
- Consolidated Hyper-V specific code into svm_onhyperv.[ch] to reduce
the "ifdefs". This change applies only to SVM, VMX is not touched and
is not in the scope of this patch series.
Changes from v2:
- Refactored the Remote TLB Flush logic into separate hyperv specific
source files (kvm_onhyperv.[ch]).
- Reverted the VMCB Clean bits macro changes as it is no longer needed.
Changes from v1:
- Move the remote TLB flush related fields from kvm_vcpu_hv and kvm_hv
to kvm_vcpu_arch and kvm_arch.
- Modify the VMCB clean mask runtime based on whether L1 hypervisor
is running on Hyper-V or not.
- Detect Hyper-V nested enlightenments based on
HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS.
- Address other minor review comments.
---
Vineeth Pillai (7):
hyperv: Detect Nested virtualization support for SVM
hyperv: SVM enlightened TLB flush support flag
KVM: x86: hyper-v: Move the remote TLB flush logic out of vmx
KVM: SVM: Software reserved fields
KVM: SVM: hyper-v: Remote TLB flush for SVM
KVM: SVM: hyper-v: Enlightened MSR-Bitmap support
KVM: SVM: hyper-v: Direct Virtual Flush support
arch/x86/include/asm/hyperv-tlfs.h | 9 ++
arch/x86/include/asm/kvm_host.h | 9 ++
arch/x86/include/asm/svm.h | 9 +-
arch/x86/include/uapi/asm/svm.h | 3 +
arch/x86/kernel/cpu/mshyperv.c | 10 ++-
arch/x86/kvm/Makefile | 9 ++
arch/x86/kvm/kvm_onhyperv.c | 93 +++++++++++++++++++++
arch/x86/kvm/kvm_onhyperv.h | 32 +++++++
arch/x86/kvm/svm/svm.c | 14 ++++
arch/x86/kvm/svm/svm.h | 22 ++++-
arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++
arch/x86/kvm/svm/svm_onhyperv.h | 129 +++++++++++++++++++++++++++++
arch/x86/kvm/vmx/vmx.c | 105 +----------------------
arch/x86/kvm/vmx/vmx.h | 9 --
arch/x86/kvm/x86.c | 9 ++
15 files changed, 384 insertions(+), 119 deletions(-)
create mode 100644 arch/x86/kvm/kvm_onhyperv.c
create mode 100644 arch/x86/kvm/kvm_onhyperv.h
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.h
Super-nitpick: I'd suggest adding /* __ARCH_X86_KVM_KVM_ONHYPERV_H__ */
to the second '#endif' and /* IS_ENABLED(CONFIG_HYPERV) */ to '#else'
and the first one: files/functions tend to grow and it becomes hard to
see where the particular '#endif/#else' belongs.
Done, thanks. I've also changed the #if to just "#ifdef CONFIG_HYPERV",
since IS_ENABLED is only needed in C statements.
kvm/queue fails to compile and I blame this change:
In file included from arch/x86/kvm/svm/svm_onhyperv.c:16:
arch/x86/kvm/svm/svm_onhyperv.h: In function ‘svm_hv_hardware_setup’:
arch/x86/kvm/svm/svm_onhyperv.h:56:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function); did you mean ‘svm_flush_tlb’?
56 | svm_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
| ^~~~~~~~~~~~~~~~~~~
| svm_flush_tlb
arch/x86/kvm/svm/svm_onhyperv.h:56:34: note: each undeclared identifier is reported only once for each function it appears in
arch/x86/kvm/svm/svm_onhyperv.h:58:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function)
58 | hv_remote_flush_tlb_with_range;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [scripts/Makefile.build:272: arch/x86/kvm/svm/svm_onhyperv.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from arch/x86/kvm/svm/svm.c:47:
arch/x86/kvm/svm/svm_onhyperv.h: In function ‘svm_hv_hardware_setup’:
arch/x86/kvm/svm/svm_onhyperv.h:56:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function); did you mean ‘svm_flush_tlb’?
56 | svm_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
| ^~~~~~~~~~~~~~~~~~~
| svm_flush_tlb
arch/x86/kvm/svm/svm_onhyperv.h:56:34: note: each undeclared identifier is reported only once for each function it appears in
arch/x86/kvm/svm/svm_onhyperv.h:58:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function)
58 | hv_remote_flush_tlb_with_range;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [scripts/Makefile.build:272: arch/x86/kvm/svm/svm.o] Error 1
arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’:
arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function)
7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
| ^~~~~~~~~~~~~~~~~~~
arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in
arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function)
7754 | hv_remote_flush_tlb_with_range;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(Note: CONFIG_HYPERV can be 'm'.)
The following:
index 96da53edfe83..1c67abf2eba9 100644
On Thu, 2021-06-10 at 17:17 +0200, Paolo Bonzini wrote:
On 03/06/21 17:14, Vineeth Pillai wrote:
quoted
This patch series enables the nested virtualization enlightenments for
SVM. This is very similar to the enlightenments for VMX except for the
fact that there is no enlightened VMCS. For SVM, VMCB is already an
architectural in-memory data structure.
Note: v5 is just a rebase on hyperv-next(5.13-rc1) and needed a rework
based on the patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush)
https://lore.kernel.org/lkml/20210305183123.3978098-1-seanjc@google.com/
The supported enlightenments are:
Enlightened TLB Flush: If this is enabled, ASID invalidations invalidate
only gva -> hpa entries. To flush entries derived from NPT, hyper-v
provided hypercalls (HvFlushGuestPhysicalAddressSpace or
HvFlushGuestPhysicalAddressList) should be used.
Enlightened MSR bitmap(TLFS 16.5.3): "When enabled, L0 hypervisor does
not monitor the MSR bitmaps for changes. Instead, the L1 hypervisor must
invalidate the corresponding clean field after making changes to one of
the MSR bitmaps."
Direct Virtual Flush(TLFS 16.8): The hypervisor exposes hypercalls
(HvFlushVirtualAddressSpace, HvFlushVirtualAddressSpaceEx,
HvFlushVirtualAddressList, and HvFlushVirtualAddressListEx) that allow
operating systems to more efficiently manage the virtual TLB. The L1
hypervisor can choose to allow its guest to use those hypercalls and
delegate the responsibility to handle them to the L0 hypervisor. This
requires the use of a partition assist page."
L2 Windows boot time was measured with and without the patch. Time was
measured from power on to the login screen and was averaged over a
consecutive 5 trials:
Without the patch: 42 seconds
With the patch: 29 seconds
--
Changes from v4
- Rebased on top of 5.13-rc1 and reworked based on the changes in the
patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush)
Changes from v3
- Included definitions for software/hypervisor reserved fields in SVM
architectural data structures.
- Consolidated Hyper-V specific code into svm_onhyperv.[ch] to reduce
the "ifdefs". This change applies only to SVM, VMX is not touched and
is not in the scope of this patch series.
Changes from v2:
- Refactored the Remote TLB Flush logic into separate hyperv specific
source files (kvm_onhyperv.[ch]).
- Reverted the VMCB Clean bits macro changes as it is no longer needed.
Changes from v1:
- Move the remote TLB flush related fields from kvm_vcpu_hv and kvm_hv
to kvm_vcpu_arch and kvm_arch.
- Modify the VMCB clean mask runtime based on whether L1 hypervisor
is running on Hyper-V or not.
- Detect Hyper-V nested enlightenments based on
HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS.
- Address other minor review comments.
---
Vineeth Pillai (7):
hyperv: Detect Nested virtualization support for SVM
hyperv: SVM enlightened TLB flush support flag
KVM: x86: hyper-v: Move the remote TLB flush logic out of vmx
KVM: SVM: Software reserved fields
KVM: SVM: hyper-v: Remote TLB flush for SVM
KVM: SVM: hyper-v: Enlightened MSR-Bitmap support
KVM: SVM: hyper-v: Direct Virtual Flush support
arch/x86/include/asm/hyperv-tlfs.h | 9 ++
arch/x86/include/asm/kvm_host.h | 9 ++
arch/x86/include/asm/svm.h | 9 +-
arch/x86/include/uapi/asm/svm.h | 3 +
arch/x86/kernel/cpu/mshyperv.c | 10 ++-
arch/x86/kvm/Makefile | 9 ++
arch/x86/kvm/kvm_onhyperv.c | 93 +++++++++++++++++++++
arch/x86/kvm/kvm_onhyperv.h | 32 +++++++
arch/x86/kvm/svm/svm.c | 14 ++++
arch/x86/kvm/svm/svm.h | 22 ++++-
arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++
arch/x86/kvm/svm/svm_onhyperv.h | 129 +++++++++++++++++++++++++++++
arch/x86/kvm/vmx/vmx.c | 105 +----------------------
arch/x86/kvm/vmx/vmx.h | 9 --
arch/x86/kvm/x86.c | 9 ++
15 files changed, 384 insertions(+), 119 deletions(-)
create mode 100644 arch/x86/kvm/kvm_onhyperv.c
create mode 100644 arch/x86/kvm/kvm_onhyperv.h
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.h
Queued, thanks.
Paolo
Hi!
This patch series causes a build failure here:
arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’:
arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function)
7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
| ^~~~~~~~~~~~~~~~~~~
arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in
arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function)
7754 | hv_remote_flush_tlb_with_range;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Also this:
arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’:
arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function)
7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
| ^~~~~~~~~~~~~~~~~~~
arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in
arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function)
7754 | hv_remote_flush_tlb_with_range;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Best regards,
Maxim Levitsky
On Thu, 2021-06-10 at 17:17 +0200, Paolo Bonzini wrote:
quoted
On 03/06/21 17:14, Vineeth Pillai wrote:
quoted
This patch series enables the nested virtualization enlightenments for
SVM. This is very similar to the enlightenments for VMX except for the
fact that there is no enlightened VMCS. For SVM, VMCB is already an
architectural in-memory data structure.
Note: v5 is just a rebase on hyperv-next(5.13-rc1) and needed a rework
based on the patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush)
https://lore.kernel.org/lkml/20210305183123.3978098-1-seanjc@google.com/
The supported enlightenments are:
Enlightened TLB Flush: If this is enabled, ASID invalidations invalidate
only gva -> hpa entries. To flush entries derived from NPT, hyper-v
provided hypercalls (HvFlushGuestPhysicalAddressSpace or
HvFlushGuestPhysicalAddressList) should be used.
Enlightened MSR bitmap(TLFS 16.5.3): "When enabled, L0 hypervisor does
not monitor the MSR bitmaps for changes. Instead, the L1 hypervisor must
invalidate the corresponding clean field after making changes to one of
the MSR bitmaps."
Direct Virtual Flush(TLFS 16.8): The hypervisor exposes hypercalls
(HvFlushVirtualAddressSpace, HvFlushVirtualAddressSpaceEx,
HvFlushVirtualAddressList, and HvFlushVirtualAddressListEx) that allow
operating systems to more efficiently manage the virtual TLB. The L1
hypervisor can choose to allow its guest to use those hypercalls and
delegate the responsibility to handle them to the L0 hypervisor. This
requires the use of a partition assist page."
L2 Windows boot time was measured with and without the patch. Time was
measured from power on to the login screen and was averaged over a
consecutive 5 trials:
Without the patch: 42 seconds
With the patch: 29 seconds
--
Changes from v4
- Rebased on top of 5.13-rc1 and reworked based on the changes in the
patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush)
Changes from v3
- Included definitions for software/hypervisor reserved fields in SVM
architectural data structures.
- Consolidated Hyper-V specific code into svm_onhyperv.[ch] to reduce
the "ifdefs". This change applies only to SVM, VMX is not touched and
is not in the scope of this patch series.
Changes from v2:
- Refactored the Remote TLB Flush logic into separate hyperv specific
source files (kvm_onhyperv.[ch]).
- Reverted the VMCB Clean bits macro changes as it is no longer needed.
Changes from v1:
- Move the remote TLB flush related fields from kvm_vcpu_hv and kvm_hv
to kvm_vcpu_arch and kvm_arch.
- Modify the VMCB clean mask runtime based on whether L1 hypervisor
is running on Hyper-V or not.
- Detect Hyper-V nested enlightenments based on
HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS.
- Address other minor review comments.
---
Vineeth Pillai (7):
hyperv: Detect Nested virtualization support for SVM
hyperv: SVM enlightened TLB flush support flag
KVM: x86: hyper-v: Move the remote TLB flush logic out of vmx
KVM: SVM: Software reserved fields
KVM: SVM: hyper-v: Remote TLB flush for SVM
KVM: SVM: hyper-v: Enlightened MSR-Bitmap support
KVM: SVM: hyper-v: Direct Virtual Flush support
arch/x86/include/asm/hyperv-tlfs.h | 9 ++
arch/x86/include/asm/kvm_host.h | 9 ++
arch/x86/include/asm/svm.h | 9 +-
arch/x86/include/uapi/asm/svm.h | 3 +
arch/x86/kernel/cpu/mshyperv.c | 10 ++-
arch/x86/kvm/Makefile | 9 ++
arch/x86/kvm/kvm_onhyperv.c | 93 +++++++++++++++++++++
arch/x86/kvm/kvm_onhyperv.h | 32 +++++++
arch/x86/kvm/svm/svm.c | 14 ++++
arch/x86/kvm/svm/svm.h | 22 ++++-
arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++
arch/x86/kvm/svm/svm_onhyperv.h | 129 +++++++++++++++++++++++++++++
arch/x86/kvm/vmx/vmx.c | 105 +----------------------
arch/x86/kvm/vmx/vmx.h | 9 --
arch/x86/kvm/x86.c | 9 ++
15 files changed, 384 insertions(+), 119 deletions(-)
create mode 100644 arch/x86/kvm/kvm_onhyperv.c
create mode 100644 arch/x86/kvm/kvm_onhyperv.h
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.h
Queued, thanks.
Paolo
Hi!
This patch series causes a build failure here:
arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’:
arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function)
7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
| ^~~~~~~~~~~~~~~~~~~
arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in
arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function)
7754 | hv_remote_flush_tlb_with_range;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Also this:
arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’:
arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function)
7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb;
| ^~~~~~~~~~~~~~~~~~~
arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in
arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function)
7754 | hv_remote_flush_tlb_with_range;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
And that's how I learnt that 1) IS_ENABLED != #ifdef for modules 2)
CONFIG_HYPERV=m is possible. Sorry Vineeth for clobbering your
perfectly fine patch!
Paolo
We've stubmled upon this multiple times already. Initially, the whole
Hyper-V support code was a module (what's now in drivers/hv/) but then
some core functionallity was moved out to arch/x86/ but we didn't add a
new config back then. Still suffering :-)
Ideally, we would want to have
CONFIG_HYPERV_GUEST=y/n for what's in arch/x86/ (just like CONFIG_KVM_GUEST)
CONFIG_HYPERV_VMBUS=y/n/m for what's in drivers/hv
--
Vitaly
From Hyper-V TLFS:
"The hypervisor exposes hypercalls (HvFlushVirtualAddressSpace,
HvFlushVirtualAddressSpaceEx, HvFlushVirtualAddressList, and
HvFlushVirtualAddressListEx) that allow operating systems to more
efficiently manage the virtual TLB. The L1 hypervisor can choose to
allow its guest to use those hypercalls and delegate the responsibility
to handle them to the L0 hypervisor. This requires the use of a
partition assist page."
Add the Direct Virtual Flush support for SVM.
Related VMX changes:
commit 6f6a657c9998 ("KVM/Hyper-V/VMX: Add direct tlb flush support")
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/kvm/Makefile | 4 ++++
arch/x86/kvm/svm/svm.c | 2 ++
arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++++++++++++++++++++++++++
arch/x86/kvm/svm/svm_onhyperv.h | 36 +++++++++++++++++++++++++++++
4 files changed, 83 insertions(+)
create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c
This blows up in testing when no Hyper-V context was created on a vCPU,
e.g. when running KVM selftests (to_hv_vcpu(vcpu) is NULL when no
Hyper-V emulation features were requested on a vCPU but
svm_hv_update_vp_id() is called unconditionally by svm_vcpu_run()).
I'll be sending a patch to fix the immediate issue but I was wondering
why we need to call svm_hv_update_vp_id() from svm_vcpu_run() as VP
index is unlikely to change; we can probably just call it from
kvm_hv_set_msr() instead.
This blows up in testing when no Hyper-V context was created on a vCPU,
e.g. when running KVM selftests (to_hv_vcpu(vcpu) is NULL when no
Hyper-V emulation features were requested on a vCPU but
svm_hv_update_vp_id() is called unconditionally by svm_vcpu_run()).
I'll be sending a patch to fix the immediate issue but I was wondering
why we need to call svm_hv_update_vp_id() from svm_vcpu_run() as VP
index is unlikely to change; we can probably just call it from
kvm_hv_set_msr() instead.
Thanks a lot for catching this.
I think you are right, updating at kvm_hv_set_msr() makes sense. I was
following the vmx logic where it also sets the vp_id in vmx_vcpu_run. But it
calls a wrapper "kvm_hv_get_vpindex" which actually checks if hv_vcpu is not
null before the assignment. I should have used that instead, my mistake.
I will look a bit more into it and send out a patch for vmx and svm
after little
more investigation.
Thanks,
Vineeth
From: Yosry Ahmed <hidden> Date: 2025-09-11 22:35:42
On Thu, Jun 03, 2021 at 03:14:37PM +0000, Vineeth Pillai wrote:
quoted hunk
SVM added support for certain reserved fields to be used by
software or hypervisor. Add the following reserved fields:
- VMCB offset 0x3e0 - 0x3ff
- Clean bit 31
- SVM intercept exit code 0xf0000000
Later patches will make use of this for supporting Hyper-V
nested virtualization enhancements.
Signed-off-by: Vineeth Pillai <redacted>
---
arch/x86/include/asm/svm.h | 9 +++++++--
arch/x86/include/uapi/asm/svm.h | 3 +++
arch/x86/kvm/svm/svm.h | 17 +++++++++++++++--
3 files changed, 25 insertions(+), 4 deletions(-)
@@ -110,6 +110,9 @@#define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff+/* Exit code reserved for hypervisor/software use */+#define SVM_EXIT_SW 0xf0000000
Apologies for reviving this 2021 thread, but it seems like the APM says
in Table C-1. SVM Intercept Codes that the host reserved value is
F000_000h.
APM typo or wrong KVM definition?
From: Tom Lendacky <thomas.lendacky@amd.com> Date: 2025-09-12 13:09:24
On 9/11/25 17:35, Yosry Ahmed wrote:
On Thu, Jun 03, 2021 at 03:14:37PM +0000, Vineeth Pillai wrote:
Apologies for reviving this 2021 thread, but it seems like the APM says
in Table C-1. SVM Intercept Codes that the host reserved value is
F000_000h.
APM typo or wrong KVM definition?