Hi Ionut,
Thanks for the details. It was indeed an informative reading. Let me
add a couple notes below.
Thanks,
Yury
On Wed, Aug 12, 2026 at 10:45:56PM +0300, Ionut Nechita (Sunlight Linux) wrote:
On Wed, Aug 12, 2026 at 11:10:21AM +0530, Shrikanth Hegde wrote:
quoted
This patch series represents the result of multiple iterations,
redesigns and community feedback.
Nice work, and thanks for the very readable cover letter.
I looked at this from the KVM and Xen guest angle rather than from
PowerVM, since that is what I run. Three observations below. All of
them are from code inspection only -- I have not measured any of this,
so please treat the numbers as arithmetic rather than as results.
Code references are against next-20260812, which already carries your
base commit f2c2ba7219e5, so the series applies there directly.
1) Default thresholds are unreachable on common QEMU command lines
==================================================================
get_system_cpus() returns num_possible_cpus(), and steal_governor_loop()
divides by it:
delta_ns = max_t(u64, div_u64(delta_ns * get_system_cpus(), 10000), 1);
steal_ratio = div64_u64(delta_steal, delta_ns);
On x86 the possible map is sized by topology_init_possible_cpus()
(arch/x86/kernel/cpu/topology.c) from assigned + disabled CPUs, where
"disabled" is incremented by topo_register_apic() for every APIC that is
registered but not present.
QEMU emits exactly such MADT entries for the range [smp, maxcpus), and
acpi_is_processor_usable() in arch/x86/kernel/acpi/boot.c documents that
this is deliberate:
/*
* QEMU expects legacy "Enabled=0" LAPIC entries to be counted as
* usable in order to support CPU hotplug in guests.
*/
So those vCPUs are registered as usable, land in nr_disabled_cpus, and
end up in the possible map. A guest started with
-smp 4,maxcpus=32
has num_possible_cpus() == 32 while only 4 vCPUs ever run. The steal
ratio is then diluted 8x, and the default thresholds of 5% / 2% become
40% / 16% of the steal that is actually observable. The governor never
leaves the "do nothing" window, and the only symptom is that nothing
happens.
Xen PV guests are affected in the same way, and often more strongly,
because the possible map there tends to be sized for vCPU hotplug.
I understand from the v9 changelog why possible CPUs were chosen -- it
keeps the accumulated steal monotonic across hotplug, which is a real
property worth having. The documentation does describe the effect and
gives a worked example for recomputing the thresholds by hand. But for
a mechanism whose whole premise is that every VM on the host opts in
with the same policy, requiring each operator to first derive their own
thresholds seems likely to translate into low real-world adoption.
Some options, roughly in increasing order of intrusiveness:
- emit a pr_info() (or pr_warn()) at module init when
num_possible_cpus() significantly exceeds num_online_cpus(), naming
the ratio and the effective thresholds. Cheap, and turns a silent
no-op into something diagnosable.
- scale the thresholds by num_possible_cpus() / num_online_cpus() at
init, so the documented defaults keep their intended meaning.
- keep summing steal over the possible mask, as today, but divide by
the online count, and handle the hotplug discontinuity by resetting
the sg_ctx.steal baseline from a hotplug notifier.
I do not have a strong preference among these, and the first one alone
would already be a large improvement.
I asked the same question on v9 iteration, but the problem is still
there. I don't think that VM with a big number of offlined CPUs is
something that needs to bring admin's attention with pr_info(). The
solution should be found withing the driver.
I thought about saving the steal time for each CPU in an array, so on
the next iteration we count steal time from the same subset of CPUs,
regardless of hot plugging.
Your approach with num_possible_cpus() / num_online_cpus() scale looks
nicer, if it works. Definitely need a testing for both. Resetting the
steal baseline on each hotplug is another option, but to me the least
attractive.
This problem must be resolved this way or another before we merge the
driver.
2) Nothing stops the driver from folding Xen dom0
=================================================
dom0 accounts steal time like any other domain -- xen_time_setup_guest()
in arch/x86/xen/time.c wires up pv_steal_clock unconditionally, with no
feature negotiation and no privileged-domain exemption.
So loading steal_governor in dom0 makes it shrink its own preferred mask
under contention. That is precisely when the blkback and netback
threads serving every other guest need CPU, and the driver has no notion
that this domain is different from the ones it is trying to be polite
towards. The effect would be host-wide, not confined to the domain that
loaded the module.
Given that Kconfig carries "default m", the module is built on any
distro kernel with PARAVIRT=y, which includes dom0 kernels. It is one
modprobe away from being loaded there, quite plausibly by someone who
read the documentation's advice to enable it uniformly across all VMs.
A xen_initial_domain() check that refuses to load, or at minimum a loud
warning, seems worth having. More generally it may be worth stating in
the documentation that the driver is meant for guests only -- the same
argument applies to a KVM host that is itself running nested guests.
Juergen and the virtualization list are already on Cc -- I would value
their view on this one in particular.
3) Core granularity degenerates to single vCPUs on KVM and Xen
==============================================================
decrease_preferred_cpus() and increase_preferred_cpus() step by
topology_sibling_cpumask(), which matches PowerVM, where the hypervisor
schedules whole cores.
On Xen PV that mask is always the CPU itself, by design. The header
comment of arch/x86/xen/smp_pv.c is explicit about both the behaviour
and the reason for it:
/*
* Because virtual CPUs can be scheduled onto any real CPU, there's
* no useful topology information for the kernel to make use of. As
* a result, all CPUs are treated as if they're single-core and
* single-threaded.
*/
A plain "-smp N" under QEMU gives one thread per core and one core per
socket, with the same result. So on both hypervisors the step size is
one vCPU, and convergence to a folded state takes proportionally longer
than the PowerVM numbers would suggest.
I do not think this is a correctness problem -- per-vCPU is arguably the
right granularity there, for exactly the reason the Xen comment gives.
The case that does not work as intended is the opposite one: a guest
given a synthetic topology such as
-smp 16,sockets=1,cores=8,threads=2
will fold what it believes is a core, but those two vCPU threads need
not be co-located on any host core, so nothing in particular is freed.
On 2 and 3.
If the governor will become a practical tool for admins, I believe
there will be the more corner cases, the more adoption the driver
gets.
That's why I suggested the modular architecture for the governor. Now
we have 2 options:
1. Keep the driver as a single module, and add those checks for DOM 0
and per-core vs per-thread behavior.
2. Rename this driver to powervm_steal_governor, and let KVM and XEN
engineers implement their own drivers.
The options are not mutually exclusive. The earlier versions of the
driver had a 'core' part and hooks for different arches.
At this point, I believe, keeping the driver simple is the most
important goal for making it upstreamable. When it's merged, it
will be much easier to add more functionality and tuning for
different VMs and architectures.
So mainly a documentation request: a sentence in
Documentation/driver-api/steal-governor.rst noting that the core-level
step assumes the guest topology reflects host scheduling granularity,
and that on KVM and Xen it commonly does not. It would also explain to
readers why convergence looks slower there than in your PowerVM numbers.
I agree, this should be documented.
--
I would be happy to run this on x86 KVM and on Xen PV guests if that is
useful -- the series has no Xen coverage that I can see, and points 1
and 3 above would both show up there first.
I didn't test the patches myself yet, mostly because configuring VMs
is a separate topic. If you share a script that configures an
environment with paravirtual VMs that run some payload that overcommit
the vCPUs, it would be a valuable addition to the series.
Andrea Righi with his virtme is already in CC. It's a very friendly
and highly configurable wrapper around qemu that allows to compile a
and run kernel with a couple commands. Andrea, can the virtme prepare
such an environment? For testing we need:
- several paravirtual VMs, so that the number of vCPUs exceeds the
number of CPUs, and
- some payload running on them to overcommit the vCPUs, triggering
the steal time counter.
Thanks,
Yury