From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Also move the Hyper-V reference TSC initialization much earlier in the boot
process so no discontinuity is observed when pv_ops.time.sched_clock
calculates its offset. This earlier initialization requires that the Hyper-V TSC
page be allocated statically instead of with vmalloc(), so fixup the references
to the TSC page and the method of getting its physical address.
Tianyu Lan (2):
clocksource/Hyper-v: Allocate Hyper-V tsc page statically
clocksource/Hyper-V: Add Hyper-V specific sched clock function
arch/x86/entry/vdso/vma.c | 2 +-
arch/x86/hyperv/hv_init.c | 2 --
arch/x86/kernel/cpu/mshyperv.c | 8 ++++++++
drivers/clocksource/hyperv_timer.c | 34 ++++++++++++++++------------------
include/asm-generic/mshyperv.h | 1 +
5 files changed, 26 insertions(+), 21 deletions(-)
--
2.14.5
From: Tianyu Lan <redacted>
This is to prepare to add Hyper-V sched clock callback and move
Hyper-V reference TSC initialization much earlier in the boot
process when timestamp is 0. So no discontinuity is observed
when pv_ops.time.sched_clock to calculate its offset. This earlier
initialization requires that the Hyper-V TSC page be allocated
statically instead of with vmalloc(), so fixup the references
to the TSC page and the method of getting its physical address.
Signed-off-by: Tianyu Lan <redacted>
---
arch/x86/entry/vdso/vma.c | 2 +-
drivers/clocksource/hyperv_timer.c | 12 ++++--------
2 files changed, 5 insertions(+), 9 deletions(-)
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets pv_ops.time.
sched_clock to read the Hyper-V reference TSC value that is scaled and adjusted
to be continuous.
Also move the Hyper-V reference TSC initialization much earlier in the boot
process so no discontinuity is observed when pv_ops.time.sched_clock
calculates its offset.
Signed-off-by: Tianyu Lan <redacted>
---
arch/x86/hyperv/hv_init.c | 2 --
arch/x86/kernel/cpu/mshyperv.c | 8 ++++++++
drivers/clocksource/hyperv_timer.c | 22 ++++++++++++----------
include/asm-generic/mshyperv.h | 1 +
4 files changed, 21 insertions(+), 12 deletions(-)
@@ -298,8 +299,9 @@ static bool __init hv_init_tsc_clocksource(void)hv_set_clocksource_vdso(hyperv_cs_tsc);clocksource_register_hz(&hyperv_cs_tsc,NSEC_PER_SEC/100);-/* sched_clock_register is needed on ARM64 but is a no-op on x86 */-sched_clock_register(read_hv_sched_clock_tsc,64,HV_CLOCK_HZ);+hv_sched_clock_offset=hyperv_cs->read(hyperv_cs);+hv_setup_sched_clock(read_hv_sched_clock_tsc);+returntrue;}#else
@@ -329,7 +331,7 @@ void __init hv_init_clocksource(void)hyperv_cs=&hyperv_cs_msr;clocksource_register_hz(&hyperv_cs_msr,NSEC_PER_SEC/100);-/* sched_clock_register is needed on ARM64 but is a no-op on x86 */-sched_clock_register(read_hv_sched_clock_msr,64,HV_CLOCK_HZ);+hv_sched_clock_offset=hyperv_cs->read(hyperv_cs);+hv_setup_sched_clock(read_hv_sched_clock_msr);}EXPORT_SYMBOL_GPL(hv_init_clocksource);
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Hypervisor can, in theory, disable TSC page and then we're forced to use
MSR-based clocksource but using it as sched_clock() can be very slow,
I'm afraid.
On the other hand, what we have now is probably worse: TSC can,
actually, jump backwards (e.g. on migration) and we're breaking the
requirements for sched_clock().
--
Vitaly
From: Peter Zijlstra <peterz@infradead.org> Date: 2019-07-29 11:09:43
On Mon, Jul 29, 2019 at 12:59:26PM +0200, Vitaly Kuznetsov wrote:
lantianyu1986@gmail.com writes:
quoted
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Hypervisor can, in theory, disable TSC page and then we're forced to use
MSR-based clocksource but using it as sched_clock() can be very slow,
I'm afraid.
On the other hand, what we have now is probably worse: TSC can,
actually, jump backwards (e.g. on migration) and we're breaking the
requirements for sched_clock().
That (obviously) also breaks the requirements for using TSC as
clocksource.
IOW, it breaks the entire purpose of having TSC in the first place.
On Mon, Jul 29, 2019 at 12:59:26PM +0200, Vitaly Kuznetsov wrote:
quoted
lantianyu1986@gmail.com writes:
quoted
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Hypervisor can, in theory, disable TSC page and then we're forced to use
MSR-based clocksource but using it as sched_clock() can be very slow,
I'm afraid.
On the other hand, what we have now is probably worse: TSC can,
actually, jump backwards (e.g. on migration) and we're breaking the
requirements for sched_clock().
That (obviously) also breaks the requirements for using TSC as
clocksource.
IOW, it breaks the entire purpose of having TSC in the first place.
Currently, we mark raw TSC as unstable when running on Hyper-V (see
88c9281a9fba6), 'TSC page' (which is TSC * scale + offset) is being used
instead. The problem is that 'TSC page' can be disabled by the
hypervisor and in that case the only remaining clocksource is MSR-based
(slow).
--
Vitaly
From: Tianyu Lan <hidden> Date: 2019-07-30 13:41:31
Hi Vitaly & Peter:
Thanks for your review.
On Mon, Jul 29, 2019 at 8:13 PM Vitaly Kuznetsov [off-list ref] wrote:
Peter Zijlstra [off-list ref] writes:
quoted
On Mon, Jul 29, 2019 at 12:59:26PM +0200, Vitaly Kuznetsov wrote:
quoted
lantianyu1986@gmail.com writes:
quoted
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Hypervisor can, in theory, disable TSC page and then we're forced to use
MSR-based clocksource but using it as sched_clock() can be very slow,
I'm afraid.
On the other hand, what we have now is probably worse: TSC can,
actually, jump backwards (e.g. on migration) and we're breaking the
requirements for sched_clock().
That (obviously) also breaks the requirements for using TSC as
clocksource.
IOW, it breaks the entire purpose of having TSC in the first place.
Currently, we mark raw TSC as unstable when running on Hyper-V (see
88c9281a9fba6), 'TSC page' (which is TSC * scale + offset) is being used
instead. The problem is that 'TSC page' can be disabled by the
hypervisor and in that case the only remaining clocksource is MSR-based
(slow).
Yes, that will be slow if Hyper-V doesn't expose hv tsc page and
kernel uses MSR based
clocksource. Each MSR read will trigger one VM-EXIT. This also happens on other
hypervisors (e,g, KVM doesn't expose KVM clock). Hypervisor should
take this into
account and determine which clocksource should be exposed or not.
--
Best regards
Tianyu Lan
From: Michael Kelley <hidden> Date: 2019-08-12 18:40:04
From: lantianyu1986@gmail.com <redacted> Sent: Monday, July 29, 2019 12:53 AM
This is to prepare to add Hyper-V sched clock callback and move
Hyper-V reference TSC initialization much earlier in the boot
process when timestamp is 0. So no discontinuity is observed
when pv_ops.time.sched_clock to calculate its offset. This earlier
initialization requires that the Hyper-V TSC page be allocated
statically instead of with vmalloc(), so fixup the references
to the TSC page and the method of getting its physical address.
I'd suggest tweaking the commit message wording a bit:
Prepare to add Hyper-V sched clock callback and move Hyper-V
Reference TSC initialization much earlier in the boot process. Earlier
initialization is needed so that it happens while the timestamp value
is still 0 and no discontinuity in the timestamp will occur when
pv_ops.time.sched_clock calculates its offset. The earlier
initialization requires that the Hyper-V TSC page be allocated
statically instead of with vmalloc(), so fixup the references
to the TSC page and the method of getting its physical address.
The and'ing with PAGE_MASK isn't needed. You've set up tsc_pg
to ensure it is page aligned, so there's no need to mask out any
low order bits. That's why the previous code didn't do the masking
either.
/*
* The Hyper-V TLFS specifies to preserve the value of reserved
--
2.14.5
From: Michael Kelley <hidden> Date: 2019-08-12 18:42:22
From: Tianyu Lan <redacted> Sent: Monday, July 29, 2019 12:53 AM
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets pv_ops.time.
sched_clock to read the Hyper-V reference TSC value that is scaled and adjusted
to be continuous.
Also move the Hyper-V reference TSC initialization much earlier in the boot
process so no discontinuity is observed when pv_ops.time.sched_clock
calculates its offset.
Signed-off-by: Tianyu Lan <redacted>
---
arch/x86/hyperv/hv_init.c | 2 --
arch/x86/kernel/cpu/mshyperv.c | 8 ++++++++
drivers/clocksource/hyperv_timer.c | 22 ++++++++++++----------
include/asm-generic/mshyperv.h | 1 +
4 files changed, 21 insertions(+), 12 deletions(-)
From: Michael Kelley <hidden> Date: 2019-08-12 19:22:32
From: Tianyu Lan <redacted> Sent: Tuesday, July 30, 2019 6:41 AM
On Mon, Jul 29, 2019 at 8:13 PM Vitaly Kuznetsov [off-list ref] wrote:
quoted
Peter Zijlstra [off-list ref] writes:
quoted
On Mon, Jul 29, 2019 at 12:59:26PM +0200, Vitaly Kuznetsov wrote:
quoted
lantianyu1986@gmail.com writes:
quoted
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Hypervisor can, in theory, disable TSC page and then we're forced to use
MSR-based clocksource but using it as sched_clock() can be very slow,
I'm afraid.
On the other hand, what we have now is probably worse: TSC can,
actually, jump backwards (e.g. on migration) and we're breaking the
requirements for sched_clock().
That (obviously) also breaks the requirements for using TSC as
clocksource.
IOW, it breaks the entire purpose of having TSC in the first place.
Currently, we mark raw TSC as unstable when running on Hyper-V (see
88c9281a9fba6), 'TSC page' (which is TSC * scale + offset) is being used
instead. The problem is that 'TSC page' can be disabled by the
hypervisor and in that case the only remaining clocksource is MSR-based
(slow).
Yes, that will be slow if Hyper-V doesn't expose hv tsc page and
kernel uses MSR based
clocksource. Each MSR read will trigger one VM-EXIT. This also happens on other
hypervisors (e,g, KVM doesn't expose KVM clock). Hypervisor should
take this into
account and determine which clocksource should be exposed or not.
We've confirmed with the Hyper-V team that the TSC page is always available
on Hyper-V 2016 and later, and on Hyper-V 2012 R2 when the physical
hardware presents an InvariantTSC. But the Linux Kconfig's are set up so
the TSC page is not used for 32-bit guests -- all clock reads are synthetic MSR
reads. For 32-bit, this set of changes will add more overhead because the
sched clock reads will now be MSR reads.
I would be inclined to fix the problem, even with the perf hit on 32-bit Linux.
I don’t have any data on 32-bit Linux being used in a Hyper-V guest, but it's not
supported in Azure so usage is pretty small. The alternative would be to continue
to use the raw TSC value on 32-bit, even with the risk of a discontinuity in case of
live migration or similar scenarios.
Michael
From: Tianyu Lan <redacted> Sent: Tuesday, July 30, 2019 6:41 AM
quoted
On Mon, Jul 29, 2019 at 8:13 PM Vitaly Kuznetsov [off-list ref] wrote:
quoted
Peter Zijlstra [off-list ref] writes:
quoted
On Mon, Jul 29, 2019 at 12:59:26PM +0200, Vitaly Kuznetsov wrote:
quoted
lantianyu1986@gmail.com writes:
quoted
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Hypervisor can, in theory, disable TSC page and then we're forced to use
MSR-based clocksource but using it as sched_clock() can be very slow,
I'm afraid.
On the other hand, what we have now is probably worse: TSC can,
actually, jump backwards (e.g. on migration) and we're breaking the
requirements for sched_clock().
That (obviously) also breaks the requirements for using TSC as
clocksource.
IOW, it breaks the entire purpose of having TSC in the first place.
Currently, we mark raw TSC as unstable when running on Hyper-V (see
88c9281a9fba6), 'TSC page' (which is TSC * scale + offset) is being used
instead. The problem is that 'TSC page' can be disabled by the
hypervisor and in that case the only remaining clocksource is MSR-based
(slow).
Yes, that will be slow if Hyper-V doesn't expose hv tsc page and
kernel uses MSR based
clocksource. Each MSR read will trigger one VM-EXIT. This also happens on other
hypervisors (e,g, KVM doesn't expose KVM clock). Hypervisor should
take this into
account and determine which clocksource should be exposed or not.
We've confirmed with the Hyper-V team that the TSC page is always available
on Hyper-V 2016 and later, and on Hyper-V 2012 R2 when the physical
hardware presents an InvariantTSC.
Currently we check that TSC page is valid on every read and it seems
this is redundant, right? It is either available on boot or not. I can
only imagine migrating a VM to a non-InvariantTSC host when Hyper-V will
likely disable the page (and we can get reenlightenment notification
then).
But the Linux Kconfig's are set up so
the TSC page is not used for 32-bit guests -- all clock reads are synthetic MSR
reads. For 32-bit, this set of changes will add more overhead because the
sched clock reads will now be MSR reads.
I would be inclined to fix the problem, even with the perf hit on 32-bit Linux.
I don’t have any data on 32-bit Linux being used in a Hyper-V guest, but it's not
supported in Azure so usage is pretty small. The alternative would be to continue
to use the raw TSC value on 32-bit, even with the risk of a discontinuity in case of
live migration or similar scenarios.
The issue needs fixing, I agree, however using MSR based clocksource as
sched clock may give us too big of a performance hit (not sure who cares
about 32 bit guest performance nowadays but still). What stops us from
enabling TSC page for 32 bit guests if it is available?
--
Vitaly
From: Michael Kelley <hidden> Date: 2019-08-20 14:32:49
From: Vitaly Kuznetsov <vkuznets@redhat.com> Sent: Tuesday, August 13, 2019 1:34 AM
Michael Kelley [off-list ref] writes:
quoted
From: Tianyu Lan <redacted> Sent: Tuesday, July 30, 2019 6:41 AM
quoted
On Mon, Jul 29, 2019 at 8:13 PM Vitaly Kuznetsov [off-list ref] wrote:
quoted
Peter Zijlstra [off-list ref] writes:
quoted
On Mon, Jul 29, 2019 at 12:59:26PM +0200, Vitaly Kuznetsov wrote:
quoted
lantianyu1986@gmail.com writes:
quoted
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Hypervisor can, in theory, disable TSC page and then we're forced to use
MSR-based clocksource but using it as sched_clock() can be very slow,
I'm afraid.
On the other hand, what we have now is probably worse: TSC can,
actually, jump backwards (e.g. on migration) and we're breaking the
requirements for sched_clock().
That (obviously) also breaks the requirements for using TSC as
clocksource.
IOW, it breaks the entire purpose of having TSC in the first place.
Currently, we mark raw TSC as unstable when running on Hyper-V (see
88c9281a9fba6), 'TSC page' (which is TSC * scale + offset) is being used
instead. The problem is that 'TSC page' can be disabled by the
hypervisor and in that case the only remaining clocksource is MSR-based
(slow).
Yes, that will be slow if Hyper-V doesn't expose hv tsc page and
kernel uses MSR based
clocksource. Each MSR read will trigger one VM-EXIT. This also happens on other
hypervisors (e,g, KVM doesn't expose KVM clock). Hypervisor should
take this into
account and determine which clocksource should be exposed or not.
We've confirmed with the Hyper-V team that the TSC page is always available
on Hyper-V 2016 and later, and on Hyper-V 2012 R2 when the physical
hardware presents an InvariantTSC.
Currently we check that TSC page is valid on every read and it seems
this is redundant, right? It is either available on boot or not. I can
only imagine migrating a VM to a non-InvariantTSC host when Hyper-V will
likely disable the page (and we can get reenlightenment notification
then).
I think Hyper-V can have brief intervals when the TSC page is not valid, so
the code checks for the "sequence" value being zero. Otherwise, yes, it
should always be there or not be there. Is there some other validity
check on every read that you are thinking of?
quoted
But the Linux Kconfig's are set up so
the TSC page is not used for 32-bit guests -- all clock reads are synthetic MSR
reads. For 32-bit, this set of changes will add more overhead because the
sched clock reads will now be MSR reads.
I would be inclined to fix the problem, even with the perf hit on 32-bit Linux.
I don’t have any data on 32-bit Linux being used in a Hyper-V guest, but it's not
supported in Azure so usage is pretty small. The alternative would be to continue
to use the raw TSC value on 32-bit, even with the risk of a discontinuity in case of
live migration or similar scenarios.
The issue needs fixing, I agree, however using MSR based clocksource as
sched clock may give us too big of a performance hit (not sure who cares
about 32 bit guest performance nowadays but still). What stops us from
enabling TSC page for 32 bit guests if it is available?
I talked to KY Srinivasan for any history about TSC page on 32-bit. He said
there was no technical reason not to implement it, but our focus was always
64-bit Linux, so the 32-bit was much less important. Also, on 32-bit Linux,
the required 64x64 multiply and shift is more complex and takes more
more cycles (compare 32-bit implementation of mul_u64_u64_shr vs.
the 64-bit implementation), so the win over a MSR read is less. I
don't know of any actual measurements being made to compare vs.
MSR read.
Michael
From: Vitaly Kuznetsov <vkuznets@redhat.com> Sent: Tuesday, August 13, 2019 1:34 AM
quoted
Michael Kelley [off-list ref] writes:
quoted
From: Tianyu Lan <redacted> Sent: Tuesday, July 30, 2019 6:41 AM
quoted
On Mon, Jul 29, 2019 at 8:13 PM Vitaly Kuznetsov [off-list ref] wrote:
quoted
Peter Zijlstra [off-list ref] writes:
quoted
On Mon, Jul 29, 2019 at 12:59:26PM +0200, Vitaly Kuznetsov wrote:
quoted
lantianyu1986@gmail.com writes:
quoted
From: Tianyu Lan <redacted>
Hyper-V guests use the default native_sched_clock() in pv_ops.time.sched_clock
on x86. But native_sched_clock() directly uses the raw TSC value, which
can be discontinuous in a Hyper-V VM. Add the generic hv_setup_sched_clock()
to set the sched clock function appropriately. On x86, this sets
pv_ops.time.sched_clock to read the Hyper-V reference TSC value that is
scaled and adjusted to be continuous.
Hypervisor can, in theory, disable TSC page and then we're forced to use
MSR-based clocksource but using it as sched_clock() can be very slow,
I'm afraid.
On the other hand, what we have now is probably worse: TSC can,
actually, jump backwards (e.g. on migration) and we're breaking the
requirements for sched_clock().
That (obviously) also breaks the requirements for using TSC as
clocksource.
IOW, it breaks the entire purpose of having TSC in the first place.
Currently, we mark raw TSC as unstable when running on Hyper-V (see
88c9281a9fba6), 'TSC page' (which is TSC * scale + offset) is being used
instead. The problem is that 'TSC page' can be disabled by the
hypervisor and in that case the only remaining clocksource is MSR-based
(slow).
Yes, that will be slow if Hyper-V doesn't expose hv tsc page and
kernel uses MSR based
clocksource. Each MSR read will trigger one VM-EXIT. This also happens on other
hypervisors (e,g, KVM doesn't expose KVM clock). Hypervisor should
take this into
account and determine which clocksource should be exposed or not.
We've confirmed with the Hyper-V team that the TSC page is always available
on Hyper-V 2016 and later, and on Hyper-V 2012 R2 when the physical
hardware presents an InvariantTSC.
Currently we check that TSC page is valid on every read and it seems
this is redundant, right? It is either available on boot or not. I can
only imagine migrating a VM to a non-InvariantTSC host when Hyper-V will
likely disable the page (and we can get reenlightenment notification
then).
I think Hyper-V can have brief intervals when the TSC page is not valid, so
the code checks for the "sequence" value being zero. Otherwise, yes, it
should always be there or not be there. Is there some other validity
check on every read that you are thinking of?
No, it's this one. In case these 'invalidity periods' are real there's
nothing to improve in the current code.
quoted
quoted
But the Linux Kconfig's are set up so
the TSC page is not used for 32-bit guests -- all clock reads are synthetic MSR
reads. For 32-bit, this set of changes will add more overhead because the
sched clock reads will now be MSR reads.
I would be inclined to fix the problem, even with the perf hit on 32-bit Linux.
I don’t have any data on 32-bit Linux being used in a Hyper-V guest, but it's not
supported in Azure so usage is pretty small. The alternative would be to continue
to use the raw TSC value on 32-bit, even with the risk of a discontinuity in case of
live migration or similar scenarios.
The issue needs fixing, I agree, however using MSR based clocksource as
sched clock may give us too big of a performance hit (not sure who cares
about 32 bit guest performance nowadays but still). What stops us from
enabling TSC page for 32 bit guests if it is available?
I talked to KY Srinivasan for any history about TSC page on 32-bit. He said
there was no technical reason not to implement it, but our focus was always
64-bit Linux, so the 32-bit was much less important. Also, on 32-bit Linux,
the required 64x64 multiply and shift is more complex and takes more
more cycles (compare 32-bit implementation of mul_u64_u64_shr vs.
the 64-bit implementation), so the win over a MSR read is less. I
don't know of any actual measurements being made to compare vs.
MSR read.
VMExit is 1000 CPU cycles or so, I would guess that TSC page
calculations are better. Let me try to build 32bit kernel and do some
quick measurements.
--
Vitaly
I talked to KY Srinivasan for any history about TSC page on 32-bit. He said
there was no technical reason not to implement it, but our focus was always
64-bit Linux, so the 32-bit was much less important. Also, on 32-bit Linux,
the required 64x64 multiply and shift is more complex and takes more
more cycles (compare 32-bit implementation of mul_u64_u64_shr vs.
the 64-bit implementation), so the win over a MSR read is less. I
don't know of any actual measurements being made to compare vs.
MSR read.
VMExit is 1000 CPU cycles or so, I would guess that TSC page
calculations are better. Let me try to build 32bit kernel and do some
quick measurements.
So I tried and the difference is HUGE.
For in-kernel clocksource reads (like sched_clock()), the testing code
was:
before = rdtsc_ordered();
for (i = 0; i < 1000; i++)
(void)read_hv_sched_clock_msr();
after = rdtsc_ordered();
printk("MSR based clocksource: %d cycles\n", ((u32)(after - before))/1000);
before = rdtsc_ordered();
for (i = 0; i < 1000; i++)
(void)read_hv_sched_clock_tsc();
after = rdtsc_ordered();
printk("TSC page clocksource: %d cycles\n", ((u32)(after - before))/1000);
The result (WS2016) is:
[ 1.101910] MSR based clocksource: 3361 cycles
[ 1.105224] TSC page clocksource: 49 cycles
For userspace reads the absolute difference is even bigger as TSC page
gives us functional vDSO:
Testing code:
before = rdtsc();
for (i = 0; i < COUNT; i++)
clock_gettime(CLOCK_REALTIME, &tp);
after = rdtsc();
printf("%d\n", (after - before)/COUNT);
Result:
TSC page:
# ./gettime_cycles
131
MSR:
# ./gettime_cycles
5664
With all that I see no reason for us to not enable TSC page on 32bit,
even if the number of users is negligible, this will allow us to get rid
of ugly #ifdef CONFIG_HYPERV_TSCPAGE in the code.
I'll send a patch for discussion.
--
Vitaly