On Mon, May 17, 2021 at 05:27:49PM +0000, Michael Kelley wrote:

From: Mark Rutland <mark.rutland@arm.com> Sent: Monday, May 17, 2021 6:08 AM

quoted

On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:

quoted

From: Mark Rutland <mark.rutland@arm.com> Sent: Friday, May 14, 2021 5:37 AM

quoted

On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:

quoted

Add architecture specific definitions and functions needed
by the architecture independent Hyper-V clocksource driver.
Update the Hyper-V clocksource driver to be initialized
on ARM64.

Previously we've said that for a clocksource we must use the architected
counter, since that's necessary for things like the VDSO to work
correctly and efficiently.

Given that, I'm a bit confused that we're registering a per-cpu
clocksource that is in part based on the architected counter. Likewise,
I don't entirely follow why it's necessary to PV the clock_event_device.

Are the architected counter and timer reliable without this PV
infrastructure? Why do we need to PV either of those?

Thanks,
Mark.

For the clocksource, we have a requirement to live migrate VMs
between Hyper-V hosts running on hardware that may have different
arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
requirement).  The Hyper-V virtualization does scaling to handle the
frequency difference.  And yes, there's a tradeoff with vDSO not
working, though we have an out-of-tree vDSO implementation that
we can use when necessary.

Just to be clear, the vDSO is *one example* of something that won't
function correctly. More generally, because this undermines core
architectural guarantees, it requires more invasive changes (e.g. we'd
have to weaken the sanity checks, and not use the counter in things like
kexec paths), impacts any architectural features tied to the generic
timer/counter (e.g. the event stream, SPE and tracing, future features),
and means that other SW (e.g. bootloaders and other EFI applications)
are unlikley to function correctly in this environment.

I am very much not keen on trying to PV this.

What does the guest see when it reads CNTFRQ_EL0? Does this match the
real HW value (and can this change over time)? Or is this entirely
synthetic?

What do the ACPI tables look like in the guest? Is there a GTDT table at
all?

How does the counter event stream behave?

Are there other architectural features which Hyper-V does not implement
for a guest?

Is there anything else that may change across a migration? e.g. MIDR?
MPIDR? Any of the ID registers?

The ARMv8 architectural system counter and associated registers are visible
and functional in a VM on Hyper-V.   The "arch_sys_counter" clocksource is
instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
and a Linux guest on Hyper-V runs fine with this clocksource.  Low level code
like bootloaders and EFI applications work normally.

That's good to hear!

One potential issue worth noting is that as those pieces of software are
unlikely to handle counter frequency changes reliably, and so may not
behave correctly if live-migrated.

The Hyper-V virtualization provides another Linux clocksource that is an
overlay on the arch counter and that provides time consistency across a live
migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
but the Hyper-V team believes they can make it functional.  I have not
explored with them the live migration implications of things beyond time
consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.

Would a summary of your point be that live migration across hardware
with different arch counter frequencies is likely to not be possible with
100% fidelity because of these other dependencies on the arch counter
frequency?  (hence the fixed 1 GHz frequency in ARM v8.6)

Yes.

In addition, there are a larger set of things necessarily exposed to VMs
that mean that live migration isn't all that practical except betweenm
identical machines (where the counter frequency should be identical),
and the timer frequency might just be the canary in the coalmine. For
example, the cache properties enumerated in CTR_EL0 cannot necessarily
be emulated on another machine.

quoted

quoted

For clockevents, the only timer interrupt that Hyper-V provides
in a guest VM is its virtualized "STIMER" interrupt.  There's no
virtualization of the ARM arch timer in the guest.

I think that is rather unfortunate, given it's a core architectural
feature. Is it just the interrupt that's missing? i.e. does all the
PE-local functionality behave as the architecture requires?

Right off the bat, I don't know about timer-related PE-local
functionality as it's not exercised in a Linux VM on Hyper-V that is
using STIMER-based clockevents.  I'll explore with the Hyper-V
team.  My impression is that enabling the ARM arch timer in a
guest VM is more work for Hyper-V than just wiring up an
interrupt.

Thanks for chasing this up!

Mark.