On Mon, Jan 06, 2025 at 07:49:15PM +0000, Michael Kelley wrote:

From: Stanislav Kinsburskii <redacted> Sent: Monday, January 6, 2025 11:19 AM

quoted

On Mon, Jan 06, 2025 at 06:18:51PM +0000, Michael Kelley wrote:

quoted

From: Stanislav Kinsburskii <redacted> Sent: Monday, January 6, 2025 9:23 AM

quoted

On Fri, Jan 03, 2025 at 10:08:05PM +0000, Michael Kelley wrote:

quoted

From: Stanislav Kinsburskii <redacted> Sent: Friday, January

3, 2025 11:20 AM

quoted

quoted

On Mon, Dec 30, 2024 at 10:09:39AM -0800, Roman Kisel wrote:

quoted

Due to the hypercall page not being allocated in the VTL mode,
the code resorts to using a part of the input page.

Allocate the hypercall output page in the VTL mode thus enabling
it to use it for output and share code with dom0.

Signed-off-by: Roman Kisel <redacted>
---
 drivers/hv/hv_common.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/hv/hv_common.c b/drivers/hv/hv_common.c
index c6ed3ba4bf61..c983cfd4d6c0 100644
--- a/drivers/hv/hv_common.c
+++ b/drivers/hv/hv_common.c

@@ -340,7 +340,7 @@ int __init hv_common_init(void)
 	BUG_ON(!hyperv_pcpu_input_arg);

 	/* Allocate the per-CPU state for output arg for root */
-	if (hv_root_partition) {
+	if (hv_root_partition || IS_ENABLED(CONFIG_HYPERV_VTL_MODE)) {

This check doesn't look nice.
First of all, IS_ENABLED(CONFIG_HYPERV_VTL_MODE) doesn't mean that this
particular kernel is being booted in VTL other that VTL0.

Actually, it does mean that. Kernels built with CONFIG_HYPERV_VTL_MODE=y
will not run as a normal guest in VTL 0. See the third paragraph of the
"help" section for HYPERV_VTL_MODE in drivers/hv/Kconfig.

Thanks for pointing to this piece.

This limitation looks aritificial to me and as VTL support in Linux is
currently being extended beyond Underhill support, keeping this
restriction makes some further development in scope of LVBS support
complicated and error prone due to potential ABI mismatches between
Linux kernels in different VTLs.

IOW, making the same kernel properly bootable (or - worse - explicitly
un-bootable) in different VTLs is a more robust way in the long run.

The reason for the limitation is the sequencing of early Hyper-V-related
initialization steps. Knowing at runtime whether you are running at
VTL0 or some other VTL requires making a hypercall in get_vtl().
Unfortunately, the machinery for making a hypercall (setting the guest
OS ID, and allocating the x86 hypercall page) is established relatively late
during initialization, in hyperv_init(). But running in other than VTL0
requires the initializations that are done in hv_vtl_init_platform(), which
must be done much earlier. There's no clear way out of this conundrum
purely on the Linux guest side.

To solve the conundrum on x86, one possibility to consider is having
Hyper-V make HV_REGISTER_VSM_VP_STATUS available as a synthetic
MSR, which can be read without making a hypercall. This register could be
read in ms_hyperv_init_platform() to know if running at VTL0 or not.
Using synthetic MSRs is how other aspects of early Hyper-V-related
initialization is done in Linux on x86.

I think there's some discussion on the x86 sequencing issues on LKML
from when the VTL code was first added. I was part of that discussion, but
don't remember all the details. There might additional issues raised in
that discussion.

The sequencing issues would also need to be sorted out on the arm64
side, as they are different from x86. We don't have an early Hyper-V
specific hook like ms_hyperv_init_platform() on the arm64 side, so
that might be problem. But on the flip side, we also don't have the
x86-specific messiness that hv_vtl_init_platform() handles. Also,
there are no synthetic MSRs on arm64, so register accesses always
use hypercalls, but there's no hypercall page needed. On balance, I
think getting VTL stuff initialized on arm64 will be easier, but I'm not sure.

Michael

Thank you for summarizing this up. This aligns with my understanding.

Since VTL1 firmware is a payload for VTL0 kernel, one of my proposals to
the original message was to explicitly notify the kernel it's running in
VTL1 via command line argument and/or DT.

OK, yes.  Rather than non-VTL0 behavior being a kernel build-time decision,
make it a kernel boot-line based decision. A runtime decision based on
detecting the VTL is hard as described above.

I don't immediately see an initialization sequence problem in using a kernel
boot line parameter to specify running in non-VTL0. I'm unsure if DT would
be available at the time ms_hyperv_init_platform() runs and decides to call
hv_vtl_init_platform(). Have you looked to see?

In current implementation, DT is not available at the moment of
hypervisor platform init on x86, which leaves only the command line
option for now.

But I guess given the overall DT struggle on x86, command line option is
a least intrusive solution out of these two.

Thanks,
Stas

Michael