[PATCH 0/6] kvm: powerpc halt polling updates

This patch series makes some updates and bug fixes to the powerpc kvm-hv
halt polling code.

The first two patches are concerned with exporting the generic kvm module
parameter variables and accessing these from the powerpc specific code.

The third patch fixes a bug where changing the global max halt polling
interval module parameter can sometimes have no effect.

The fourth patch decreases the default global max halt polling interval
to something more sensible.

The fifth patch contains generic fixups with no functional effect.

The last patch adds halt polling documentation.

Suraj Jitindar Singh (6):
  kvm: export kvm module parameter variables
  powerpc/kvm: Use generic kvm module parameters in kvm-hv
  powerpc/kvm: Add check for module parameter halt_poll_ns
  powerpc/kvm: Decrease the powerpc default halt poll max value
  powerpc/kvm: Comment style and print format fixups
  doc/kvm: Add halt polling documentation

 Documentation/virtual/kvm/00-INDEX         |   2 +
 Documentation/virtual/kvm/halt-polling.txt | 127 +++++++++++++++++++++++++++++
 arch/powerpc/include/asm/kvm_host.h        |   2 +-
 arch/powerpc/kvm/book3s_hv.c               |  33 ++------
 arch/powerpc/kvm/trace_hv.h                |   2 +-
 include/linux/kvm_host.h                   |   4 +
 virt/kvm/kvm_main.c                        |   9 +-
 7 files changed, 149 insertions(+), 30 deletions(-)
 create mode 100644 Documentation/virtual/kvm/halt-polling.txt

-- 
2.5.5

The kvm module has the parameters halt_poll_ns, halt_poll_ns_grow, and
halt_poll_ns_shrink. Halt polling was recently added to the powerpc kvm-hv
module and these parameters were essentially duplicated for that. There is
no benefit to this duplication and it can lead to confusion when trying to
tune halt polling.

Thus move the definition of these variables to kvm_host.h and export them.
This will allow the kvm-hv module to use the same module parameters by
accessing these variables, which will be implemented in the next patch,
meaning that they will no longer be duplicated.

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 include/linux/kvm_host.h | 4 ++++
 virt/kvm/kvm_main.c      | 9 ++++++---
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 01c0b9c..29b500a 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h

@@ -1107,6 +1107,10 @@ static inline bool kvm_check_request(int req, struct kvm_vcpu *vcpu)
 
 extern bool kvm_rebooting;
 
+extern unsigned int halt_poll_ns;
+extern unsigned int halt_poll_ns_grow;
+extern unsigned int halt_poll_ns_shrink;
+
 struct kvm_device {
 	struct kvm_device_ops *ops;
 	struct kvm *kvm;

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 81dfc73..675d7b5 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c

@@ -70,16 +70,19 @@ MODULE_AUTHOR("Qumranet");
 MODULE_LICENSE("GPL");
 
 /* Architectures should define their poll value according to the halt latency */
-static unsigned int halt_poll_ns = KVM_HALT_POLL_NS_DEFAULT;
+unsigned int halt_poll_ns = KVM_HALT_POLL_NS_DEFAULT;
 module_param(halt_poll_ns, uint, S_IRUGO | S_IWUSR);
+EXPORT_SYMBOL_GPL(halt_poll_ns);
 
 /* Default doubles per-vcpu halt_poll_ns. */
-static unsigned int halt_poll_ns_grow = 2;
+unsigned int halt_poll_ns_grow = 2;
 module_param(halt_poll_ns_grow, uint, S_IRUGO | S_IWUSR);
+EXPORT_SYMBOL_GPL(halt_poll_ns_grow);
 
 /* Default resets per-vcpu halt_poll_ns . */
-static unsigned int halt_poll_ns_shrink;
+unsigned int halt_poll_ns_shrink;
 module_param(halt_poll_ns_shrink, uint, S_IRUGO | S_IWUSR);
+EXPORT_SYMBOL_GPL(halt_poll_ns_shrink);
 
 /*
  * Ordering of locks:

-- 
2.5.5

The previous patch exported the variables which back the module parameters
of the generic kvm module. Now use these variables in the kvm-hv module
so that any change to the generic module parameters will also have the
same effect for the kvm-hv module. This removes the duplication of the
kvm module parameters which was redundant and should reduce confusion when
tuning them.

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 arch/powerpc/kvm/book3s_hv.c | 29 ++++++-----------------------
 1 file changed, 6 insertions(+), 23 deletions(-)

diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 3686471..daad638 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c

@@ -104,23 +104,6 @@ module_param_cb(h_ipi_redirect, &module_param_ops, &h_ipi_redirect,
 MODULE_PARM_DESC(h_ipi_redirect, "Redirect H_IPI wakeup to a free host core");
 #endif
 
-/* Maximum halt poll interval defaults to KVM_HALT_POLL_NS_DEFAULT */
-static unsigned int halt_poll_max_ns = KVM_HALT_POLL_NS_DEFAULT;
-module_param(halt_poll_max_ns, uint, S_IRUGO | S_IWUSR);
-MODULE_PARM_DESC(halt_poll_max_ns, "Maximum halt poll time in ns");
-
-/* Factor by which the vcore halt poll interval is grown, default is to double
- */
-static unsigned int halt_poll_ns_grow = 2;
-module_param(halt_poll_ns_grow, int, S_IRUGO);
-MODULE_PARM_DESC(halt_poll_ns_grow, "Factor halt poll time is grown by");
-
-/* Factor by which the vcore halt poll interval is shrunk, default is to reset
- */
-static unsigned int halt_poll_ns_shrink;
-module_param(halt_poll_ns_shrink, int, S_IRUGO);
-MODULE_PARM_DESC(halt_poll_ns_shrink, "Factor halt poll time is shrunk by");
-
 static void kvmppc_end_cede(struct kvm_vcpu *vcpu);
 static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu);
 

@@ -2544,8 +2527,8 @@ static void grow_halt_poll_ns(struct kvmppc_vcore *vc)
 	else
 		vc->halt_poll_ns *= halt_poll_ns_grow;
 
-	if (vc->halt_poll_ns > halt_poll_max_ns)
-		vc->halt_poll_ns = halt_poll_max_ns;
+	if (vc->halt_poll_ns > halt_poll_ns)
+		vc->halt_poll_ns = halt_poll_ns;
 }
 
 static void shrink_halt_poll_ns(struct kvmppc_vcore *vc)

@@ -2655,15 +2638,15 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
 	}
 
 	/* Adjust poll time */
-	if (halt_poll_max_ns) {
+	if (halt_poll_ns) {
 		if (block_ns <= vc->halt_poll_ns)
 			;
 		/* We slept and blocked for longer than the max halt time */
-		else if (vc->halt_poll_ns && block_ns > halt_poll_max_ns)
+		else if (vc->halt_poll_ns && block_ns > halt_poll_ns)
 			shrink_halt_poll_ns(vc);
 		/* We slept and our poll time is too small */
-		else if (vc->halt_poll_ns < halt_poll_max_ns &&
-				block_ns < halt_poll_max_ns)
+		else if (vc->halt_poll_ns < halt_poll_ns &&
+				block_ns < halt_poll_ns)
 			grow_halt_poll_ns(vc);
 	} else
 		vc->halt_poll_ns = 0;

-- 
2.5.5

The kvm module parameter halt_poll_ns defines the global maximum halt
polling interval and can be dynamically changed by writing to the
/sys/module/kvm/parameters/halt_poll_ns sysfs file. However in kvm-hv
this module parameter value is only ever checked when we grow the current
polling interval for the given vcore. This means that if we decrease the
halt_poll_ns value below the current polling interval we won't see any
effect unless we try to grow the polling interval above the new max at some
point or it happens to be shrunk below the halt_poll_ns value.

Update the halt polling code so that we always check for a new module param
value of halt_poll_ns and set the current halt polling interval to it if
it's currently greater than the new max. This means that it's redundant to
also perform this check in the grow_halt_poll_ns() function now.

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 arch/powerpc/kvm/book3s_hv.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index daad638..6503a63 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c

@@ -2526,9 +2526,6 @@ static void grow_halt_poll_ns(struct kvmppc_vcore *vc)
 		vc->halt_poll_ns = 10000;
 	else
 		vc->halt_poll_ns *= halt_poll_ns_grow;
-
-	if (vc->halt_poll_ns > halt_poll_ns)
-		vc->halt_poll_ns = halt_poll_ns;
 }
 
 static void shrink_halt_poll_ns(struct kvmppc_vcore *vc)

@@ -2648,6 +2645,8 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc)
 		else if (vc->halt_poll_ns < halt_poll_ns &&
 				block_ns < halt_poll_ns)
 			grow_halt_poll_ns(vc);
+		if (vc->halt_poll_ns > halt_poll_ns)
+			vc->halt_poll_ns = halt_poll_ns;
 	} else
 		vc->halt_poll_ns = 0;
 

-- 
2.5.5

KVM_HALT_POLL_NS_DEFAULT is an arch specific constant which sets the
default value of the halt_poll_ns kvm module parameter which determines
the global maximum halt polling interval.

The current value for powerpc is 500000 (500us) which means that any
repetitive workload with a period of less than that can drive the cpu
usage to 100% where it may have been mostly idle without halt polling.
This presents the possibility of a large increase in power usage with
a comparatively small performance benefit.

Reduce the default to 10000 (10us) and a user can tune this themselves
to set their affinity for halt polling based on the trade off between power
and performance which they are willing to make.

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 arch/powerpc/include/asm/kvm_host.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 28350a2..037b6a1 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h

@@ -48,7 +48,7 @@
 #ifdef CONFIG_KVM_MMIO
 #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
 #endif
-#define KVM_HALT_POLL_NS_DEFAULT 500000
+#define KVM_HALT_POLL_NS_DEFAULT 10000	/* 10 us */
 
 /* These values are internal and can be increased later */
 #define KVM_NR_IRQCHIPS          1

-- 
2.5.5

Fix comment block to match kernel comment style.

Fix print format from signed to unsigned.

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 arch/powerpc/kvm/book3s_hv.c | 3 ++-
 arch/powerpc/kvm/trace_hv.h  | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 6503a63..431950d 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c

@@ -2536,7 +2536,8 @@ static void shrink_halt_poll_ns(struct kvmppc_vcore *vc)
 		vc->halt_poll_ns /= halt_poll_ns_shrink;
 }
 
-/* Check to see if any of the runnable vcpus on the vcore have pending
+/*
+ * Check to see if any of the runnable vcpus on the vcore have pending
  * exceptions or are no longer ceded
  */
 static int kvmppc_vcore_check_block(struct kvmppc_vcore *vc)

diff --git a/arch/powerpc/kvm/trace_hv.h b/arch/powerpc/kvm/trace_hv.h
index fb21990..ebc6dd4 100644
--- a/arch/powerpc/kvm/trace_hv.h
+++ b/arch/powerpc/kvm/trace_hv.h

@@ -449,7 +449,7 @@ TRACE_EVENT(kvmppc_vcore_wakeup,
 		__entry->tgid   = current->tgid;
 	),
 
-	TP_printk("%s time %lld ns, tgid=%d",
+	TP_printk("%s time %llu ns, tgid=%d",
 		__entry->waited ? "wait" : "poll",
 		__entry->ns, __entry->tgid)
 );

-- 
2.5.5

There is currently no documentation about the halt polling capabilities
of the kvm module. Add some documentation describing the mechanism as well
as the module parameters to all better understanding of how halt polling
should be used and the effect of tuning the module parameters.

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 Documentation/virtual/kvm/00-INDEX         |   2 +
 Documentation/virtual/kvm/halt-polling.txt | 127 +++++++++++++++++++++++++++++
 2 files changed, 129 insertions(+)
 create mode 100644 Documentation/virtual/kvm/halt-polling.txt

diff --git a/Documentation/virtual/kvm/00-INDEX b/Documentation/virtual/kvm/00-INDEX
index fee9f2b..69fe1a8 100644
--- a/Documentation/virtual/kvm/00-INDEX
+++ b/Documentation/virtual/kvm/00-INDEX

@@ -6,6 +6,8 @@ cpuid.txt
 	- KVM-specific cpuid leaves (x86).
 devices/
 	- KVM_CAP_DEVICE_CTRL userspace API.
+halt-polling.txt
+	- notes on halt-polling
 hypercalls.txt
 	- KVM hypercalls.
 locking.txt

diff --git a/Documentation/virtual/kvm/halt-polling.txt b/Documentation/virtual/kvm/halt-polling.txt
new file mode 100644
index 0000000..4a84183
--- /dev/null
+++ b/Documentation/virtual/kvm/halt-polling.txt

@@ -0,0 +1,127 @@
+The KVM halt polling system
+===========================
+
+The KVM halt polling system provides a feature within KVM whereby the latency
+of a guest can, under some circumstances, be reduced by polling in the host
+for some time period after the guest has elected to no longer run by cedeing.
+That is, when a guest vcpu has ceded, or in the case of powerpc when all of the
+vcpus of a single vcore have ceded, the host kernel polls for wakeup conditions
+before giving up the cpu to the scheduler in order to let something else run.
+
+Polling provides a latency advantage in cases where the guest can be run again
+very quickly by at least saving us a trip through the scheduler, normally on
+the order of a few micro-seconds, although performance benefits are workload
+dependant. In the event that no wakeup source arrives during the polling
+interval or some other task on the runqueue is runnable the scheduler is
+invoked. Thus halt polling is especially useful on workloads with very short
+wakeup periods where the time spent halt polling is minimised and the time
+savings of not invoking the scheduler are distinguishable.
+
+The generic halt polling code is implemented in:
+
+	virt/kvm/kvm_main.c: kvm_vcpu_block()
+
+The powerpc kvm-hv specific case is implemented in:
+
+	arch/powerpc/kvm/book3s_hv.c: kvmppc_vcore_blocked()
+
+Halt Polling Interval
+=====================
+
+The maximum time for which to poll before invoking the scheduler, referred to
+as the halt polling interval, is increased and decreased based on the perceived
+effectiveness of the polling in an attempt to limit pointless polling.
+This value is stored in either the vcpu struct:
+
+	kvm_vcpu->halt_poll_ns
+
+or in the case of powerpc kvm-hv, in the vcore struct:
+
+	kvmppc_vcore->halt_poll_ns
+
+Thus this is a per vcpu (or vcore) value.
+
+During polling if a wakeup source is received within the halt polling interval,
+the interval is left unchanged. In the event that a wakeup source isn't
+received during the polling interval (and thus schedule is invoked) there are
+two options, either the polling interval and total block time[0] were less than
+the global max polling interval (see module params below), or the total block
+time was greater than the global max polling interval.
+
+In the event that both the polling interval and total block time were less than
+the global max polling interval then the polling interval can be increased in
+the hope that next time during the longer polling interval the wake up source
+will be received while the host is polling and the latency benefits will be
+received. The polling interval is grown in the function grow_halt_poll_ns() and
+is multiplied by the module parameter halt_poll_ns_grow.
+
+In the event that the total block time was greater than the global max polling
+interval then the host will never poll for long enough (limited by the global
+max) to wakeup during the polling interval so it may as well be shrunk in order
+to avoid pointless polling. The polling interval is shrunk in the function
+shrink_halt_poll_ns() and is divided by the module parameter
+halt_poll_ns_shrink, or set to 0 iff halt_poll_ns_shrink == 0.
+
+It is worth noting that this adjustment process attempts to hone in on some
+steady state polling interval but will only really do a good job for wakeups
+which come at an approximately constant rate, otherwise there will be constant
+adjustment of the polling interval.
+
+[0] total block time: the time between when the halt polling function is
+		      invoked and a wakeup source received (irrespective of
+		      whether the scheduler is invoked within that function).
+
+Module Parameters
+=================
+
+The kvm module has 3 tuneable module parameters to adjust the global max
+polling interval as well as the rate at which the polling interval is grown and
+shrunk. These variables are defined in include/linux/kvm_host.h and as module
+parameters in virt/kvm/kvm_main.c, or arch/powerpc/kvm/book3s_hv.c in the
+powerpc kvm-hv case.
+
+Module Parameter    |	     Description	      |	     Default Value
+--------------------------------------------------------------------------------
+halt_poll_ns	    | The global max polling interval | KVM_HALT_POLL_NS_DEFAULT
+		    | which defines the ceiling value |
+		    | of the polling interval for     | (per arch value)
+		    | each vcpu. 		      |
+--------------------------------------------------------------------------------
+halt_poll_ns_grow   | The value by which the halt     |	2
+		    | polling interval is multiplied  |
+		    | in the grow_halt_poll_ns()      |
+		    | function.			      |
+--------------------------------------------------------------------------------
+halt_poll_ns_shrink | The value by which the halt     |	0
+		    | polling interval is divided in  |
+		    | the shrink_halt_poll_ns()	      |
+		    | function.			      |
+--------------------------------------------------------------------------------
+
+These module parameters can be set from the debugfs files in:
+
+	/sys/module/kvm/parameters/
+
+Note: that these module parameters are system wide values and are not able to
+      be tuned on a per vm basis.
+
+Further Notes
+=============
+
+- Care should be taken when setting the halt_poll_ns module parameter as a
+large value has the potential to drive the cpu usage to 100% on a machine which
+would be almost entirely idle otherwise. This is because even if a guest has
+wakeups during which very little work is done and which are quite far apart, if
+the period is shorter than the global max polling interval (halt_poll_ns) then
+the host will always poll for the entire block time and thus cpu utilisation
+will go to 100%.
+
+- Halt polling essentially presents a trade off between power usage and latency
+and the module parameters should be used to tune the affinity for this. Idle
+cpu time is essentially converted to host kernel time with the aim of decreasing
+latency when entering the guest.
+
+- Halt polling will only be conducted by the host when no other tasks are
+runnable on that cpu, otherwise the polling will cease immediately and
+schedule will be invoked to allow that other task to run. Thus this doesn't
+allow a guest to denial of service the cpu.

-- 
2.5.5

2016-10-14 8:53 GMT+08:00 Suraj Jitindar Singh [off-list ref]:

There is currently no documentation about the halt polling capabilities
of the kvm module. Add some documentation describing the mechanism as well
as the module parameters to all better understanding of how halt polling
should be used and the effect of tuning the module parameters.

How about replace "halt-polling" by "Adaptive halt-polling"? Btw,
thanks for your docs.

Regards,
Wanpeng Li

On Fri, 2016-10-14 at 09:16 +0800, Wanpeng Li wrote:

2016-10-14 8:53 GMT+08:00 Suraj Jitindar Singh <sjitindarsingh@gmail.
com>:

quoted

There is currently no documentation about the halt polling
capabilities
of the kvm module. Add some documentation describing the mechanism
as well
as the module parameters to all better understanding of how halt
polling
should be used and the effect of tuning the module parameters.

How about replace "halt-polling" by "Adaptive halt-polling"? Btw,

Yeah that's slightly more descriptive I guess

thanks for your docs.

Regards,
Wanpeng Li

quoted

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
---
 Documentation/virtual/kvm/00-INDEX         |   2 +
 Documentation/virtual/kvm/halt-polling.txt | 127
+++++++++++++++++++++++++++++
 2 files changed, 129 insertions(+)
 create mode 100644 Documentation/virtual/kvm/halt-polling.txt

diff --git a/Documentation/virtual/kvm/00-INDEX

b/Documentation/virtual/kvm/00-INDEX
index fee9f2b..69fe1a8 100644

--- a/Documentation/virtual/kvm/00-INDEX
+++ b/Documentation/virtual/kvm/00-INDEX

@@ -6,6 +6,8 @@ cpuid.txt

        - KVM-specific cpuid leaves (x86).
 devices/
        - KVM_CAP_DEVICE_CTRL userspace API.
+halt-polling.txt
+       - notes on halt-polling
 hypercalls.txt
        - KVM hypercalls.
 locking.txt

diff --git a/Documentation/virtual/kvm/halt-polling.txt

b/Documentation/virtual/kvm/halt-polling.txt
new file mode 100644
index 0000000..4a84183

--- /dev/null
+++ b/Documentation/virtual/kvm/halt-polling.txt

@@ -0,0 +1,127 @@
+The KVM halt polling system
+===========================
+
+The KVM halt polling system provides a feature within KVM whereby

the latency
+of a guest can, under some circumstances, be reduced by polling in
the host
+for some time period after the guest has elected to no longer run
by cedeing.
+That is, when a guest vcpu has ceded, or in the case of powerpc
when all of the
+vcpus of a single vcore have ceded, the host kernel polls for
wakeup conditions
+before giving up the cpu to the scheduler in order to let
something else run.
+
+Polling provides a latency advantage in cases where the guest can
be run again
+very quickly by at least saving us a trip through the scheduler,
normally on
+the order of a few micro-seconds, although performance benefits
are workload
+dependant. In the event that no wakeup source arrives during the
polling
+interval or some other task on the runqueue is runnable the
scheduler is
+invoked. Thus halt polling is especially useful on workloads with
very short
+wakeup periods where the time spent halt polling is minimised and
the time
+savings of not invoking the scheduler are distinguishable.
+
+The generic halt polling code is implemented in:
+
+       virt/kvm/kvm_main.c: kvm_vcpu_block()
+
+The powerpc kvm-hv specific case is implemented in:
+
+       arch/powerpc/kvm/book3s_hv.c: kvmppc_vcore_blocked()
+
+Halt Polling Interval
+=====================
+
+The maximum time for which to poll before invoking the scheduler,
referred to
+as the halt polling interval, is increased and decreased based on
the perceived
+effectiveness of the polling in an attempt to limit pointless
polling.
+This value is stored in either the vcpu struct:
+
+       kvm_vcpu->halt_poll_ns
+
+or in the case of powerpc kvm-hv, in the vcore struct:
+
+       kvmppc_vcore->halt_poll_ns
+
+Thus this is a per vcpu (or vcore) value.
+
+During polling if a wakeup source is received within the halt
polling interval,
+the interval is left unchanged. In the event that a wakeup source
isn't
+received during the polling interval (and thus schedule is
invoked) there are
+two options, either the polling interval and total block time[0]
were less than
+the global max polling interval (see module params below), or the
total block
+time was greater than the global max polling interval.
+
+In the event that both the polling interval and total block time
were less than
+the global max polling interval then the polling interval can be
increased in
+the hope that next time during the longer polling interval the
wake up source
+will be received while the host is polling and the latency
benefits will be
+received. The polling interval is grown in the function
grow_halt_poll_ns() and
+is multiplied by the module parameter halt_poll_ns_grow.
+
+In the event that the total block time was greater than the global
max polling
+interval then the host will never poll for long enough (limited by
the global
+max) to wakeup during the polling interval so it may as well be
shrunk in order
+to avoid pointless polling. The polling interval is shrunk in the
function
+shrink_halt_poll_ns() and is divided by the module parameter
+halt_poll_ns_shrink, or set to 0 iff halt_poll_ns_shrink == 0.
+
+It is worth noting that this adjustment process attempts to hone
in on some
+steady state polling interval but will only really do a good job
for wakeups
+which come at an approximately constant rate, otherwise there will
be constant
+adjustment of the polling interval.
+
+[0] total block time: the time between when the halt polling
function is
+                     invoked and a wakeup source received
(irrespective of
+                     whether the scheduler is invoked within that
function).
+
+Module Parameters
+=================
+
+The kvm module has 3 tuneable module parameters to adjust the
global max
+polling interval as well as the rate at which the polling interval
is grown and
+shrunk. These variables are defined in include/linux/kvm_host.h
and as module
+parameters in virt/kvm/kvm_main.c, or arch/powerpc/kvm/book3s_hv.c
in the
+powerpc kvm-hv case.
+
+Module
Parameter    |       Description              |      Default Value
+----------------------------------------------------------------
----------------
+halt_poll_ns       | The global max polling interval |
KVM_HALT_POLL_NS_DEFAULT
+                   | which defines the ceiling value |
+                   | of the polling interval for     | (per arch
value)
+                   | each vcpu.                      |
+----------------------------------------------------------------
----------------
+halt_poll_ns_grow   | The value by which the halt     |        2
+                   | polling interval is multiplied  |
+                   | in the grow_halt_poll_ns()      |
+                   | function.                       |
+----------------------------------------------------------------
----------------
+halt_poll_ns_shrink | The value by which the halt     |        0
+                   | polling interval is divided in  |
+                   | the shrink_halt_poll_ns()       |
+                   | function.                       |
+----------------------------------------------------------------
----------------
+
+These module parameters can be set from the debugfs files in:
+
+       /sys/module/kvm/parameters/
+
+Note: that these module parameters are system wide values and are
not able to
+      be tuned on a per vm basis.
+
+Further Notes
+=============
+
+- Care should be taken when setting the halt_poll_ns module
parameter as a
+large value has the potential to drive the cpu usage to 100% on a
machine which
+would be almost entirely idle otherwise. This is because even if a
guest has
+wakeups during which very little work is done and which are quite
far apart, if
+the period is shorter than the global max polling interval
(halt_poll_ns) then
+the host will always poll for the entire block time and thus cpu
utilisation
+will go to 100%.
+
+- Halt polling essentially presents a trade off between power
usage and latency
+and the module parameters should be used to tune the affinity for
this. Idle
+cpu time is essentially converted to host kernel time with the aim
of decreasing
+latency when entering the guest.
+
+- Halt polling will only be conducted by the host when no other
tasks are
+runnable on that cpu, otherwise the polling will cease immediately
and
+schedule will be invoked to allow that other task to run. Thus
this doesn't
+allow a guest to denial of service the cpu.
--
2.5.5

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Fri, Oct 14, 2016 at 11:53:18AM +1100, Suraj Jitindar Singh wrote:

This patch series makes some updates and bug fixes to the powerpc kvm-hv
halt polling code.

The first two patches are concerned with exporting the generic kvm module
parameter variables and accessing these from the powerpc specific code.

The third patch fixes a bug where changing the global max halt polling
interval module parameter can sometimes have no effect.

The fourth patch decreases the default global max halt polling interval
to something more sensible.

The fifth patch contains generic fixups with no functional effect.

The last patch adds halt polling documentation.

Suraj Jitindar Singh (6):
  kvm: export kvm module parameter variables
  powerpc/kvm: Use generic kvm module parameters in kvm-hv
  powerpc/kvm: Add check for module parameter halt_poll_ns
  powerpc/kvm: Decrease the powerpc default halt poll max value
  powerpc/kvm: Comment style and print format fixups
  doc/kvm: Add halt polling documentation

 Documentation/virtual/kvm/00-INDEX         |   2 +
 Documentation/virtual/kvm/halt-polling.txt | 127 +++++++++++++++++++++++++++++
 arch/powerpc/include/asm/kvm_host.h        |   2 +-
 arch/powerpc/kvm/book3s_hv.c               |  33 ++------
 arch/powerpc/kvm/trace_hv.h                |   2 +-
 include/linux/kvm_host.h                   |   4 +
 virt/kvm/kvm_main.c                        |   9 +-
 7 files changed, 149 insertions(+), 30 deletions(-)
 create mode 100644 Documentation/virtual/kvm/halt-polling.txt

-- 
2.5.5

Hi Suraj,

I've given this set a quick test and it seems to work fine. I used a repetitive
wakeup, using a nanosleep loop in guest userspace (with real time prio), and I
was able to cause halt polling to switch on and off as I adjusted halt_poll_ns.
I think the new default value is much better: halt polling started (e.g. CPU
utilization rose to 100%) once CPU utilization had already risen to about 75%.

Cheers,
Sam.

Tested-by: Sam Bobroff <redacted>

On Fri, 14 Oct 2016 11:53:18 +1100
Suraj Jitindar Singh [off-list ref] wrote:

This patch series makes some updates and bug fixes to the powerpc kvm-hv
halt polling code.

The first two patches are concerned with exporting the generic kvm module
parameter variables and accessing these from the powerpc specific code.

The third patch fixes a bug where changing the global max halt polling
interval module parameter can sometimes have no effect.

The fourth patch decreases the default global max halt polling interval
to something more sensible.

The fifth patch contains generic fixups with no functional effect.

The last patch adds halt polling documentation.

I want to enable polling idle in Linux for SPLPAR/KVM as we do for
dedicated mode. Essentially the guest OS will spin for a small time
before ceding.

There will be a lot of interaction between this and halt polling. I
think guest polling may still be worthwhile if you have halt polling in
the host, although it might be less effective. We should set up some
performance testing with various guest/host parameters and see what
works best. What have you been testing with so far?

Thanks,
Nick