[RFC PATCH] Increase in idle power with schedutil

This patch adds driver callback for fast_switch and below observations
on schedutil governor are done with this patch.

In POWER8 there is a regression observed with schedutil compared to
ondemand. With schedutil the frequency is not ramping down and is
mostly stuck at max frequency during idle . This is because of the
watchdog timer, an RT task which is fired every 4 seconds which
results in requesting max frequency.

In a completely idle system, when there are no processes running apart
from few short running housekeeping tasks (like watchdog) the system is
stuck at max frequency due to 'cpufreq_trigger_update()'

static inline void cpufreq_trigger_update(u64 time)
{
        cpufreq_update_util(time, ULONG_MAX, 0);
}

If there is no noise apart from the watchdog timer the cpu is held at
max frequency for no good reason. On a 16 core system I can see an
increase in 20% idle power with schedutil compared to ondemand
governor.

Below is the trace with 'sched:sched_switch' and 'power:cpu_frequency'
events. Here the watchdog timer that runs for a very small period is
requesting Pmax and this gets triggered regularly.

<idle>-0  19059.992912: sched_switch: prev_comm=swapper/16  prev_state=R
				==> next_comm=watchdog/16 
watchdog/16-107 19059.992914: cpu_frequency: state=4322000 cpu_id=16
watchdog/16-107 19059.992915: sched_switch: prev_comm=watchdog/16 prev_state=S
	 			==> next_comm=swapper/16 

However adding a cpufreq hook in pick_next_task_idle() to decrease the
frequency helped to reduce the problem.

static inline void cpufreq_trigger_idle(u64 time)
{
       cpufreq_update_util(time, 0, 1);
}

This might not be the right fix for the problem, however this thread
is reporting the other short-comings of cpufreq_trigger_update().

Shilpasri G Bhat (1):
  cpufreq: powernv: Add fast_switch callback

 drivers/cpufreq/powernv-cpufreq.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

-- 
1.9.3

Add fast_switch driver callback to support frequency update in
interrupt context while using schedutil governor. Changing frequency
in interrupt context will remove the jitter on the workloads which can
be seen when a kworker thread is used for the changing the frequency.

Signed-off-by: Shilpasri G Bhat <redacted>
---
 drivers/cpufreq/powernv-cpufreq.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/drivers/cpufreq/powernv-cpufreq.c b/drivers/cpufreq/powernv-cpufreq.c
index 54c4536..4553eb6 100644
--- a/drivers/cpufreq/powernv-cpufreq.c
+++ b/drivers/cpufreq/powernv-cpufreq.c

@@ -678,6 +678,8 @@ static int powernv_cpufreq_cpu_init(struct cpufreq_policy *policy)
 	for (i = 0; i < threads_per_core; i++)
 		cpumask_set_cpu(base + i, policy->cpus);
 
+	policy->fast_switch_possible = true;
+
 	kn = kernfs_find_and_get(policy->kobj.sd, throttle_attr_grp.name);
 	if (!kn) {
 		int ret;

@@ -854,6 +856,24 @@ static void powernv_cpufreq_stop_cpu(struct cpufreq_policy *policy)
 	del_timer_sync(&gpstates->timer);
 }
 
+static unsigned int powernv_fast_switch(struct cpufreq_policy *policy,
+					unsigned int target_freq)
+{
+	int index;
+	struct powernv_smp_call_data freq_data;
+
+	cpufreq_frequency_table_target(policy, policy->freq_table,
+				       target_freq,
+				       CPUFREQ_RELATION_C, &index);
+	if (index < 0 || index >= powernv_pstate_info.nr_pstates)
+		return CPUFREQ_ENTRY_INVALID;
+	freq_data.pstate_id = powernv_freqs[index].driver_data;
+	freq_data.gpstate_id = powernv_freqs[index].driver_data;
+	set_pstate(&freq_data);
+
+	return pstate_id_to_freq(-index);
+}
+
 static struct cpufreq_driver powernv_cpufreq_driver = {
 	.name		= "powernv-cpufreq",
 	.flags		= CPUFREQ_CONST_LOOPS,

@@ -861,6 +881,7 @@ static struct cpufreq_driver powernv_cpufreq_driver = {
 	.exit		= powernv_cpufreq_cpu_exit,
 	.verify		= cpufreq_generic_frequency_table_verify,
 	.target_index	= powernv_cpufreq_target_index,
+	.fast_switch	= powernv_fast_switch,
 	.get		= powernv_cpufreq_get,
 	.stop_cpu	= powernv_cpufreq_stop_cpu,
 	.attr		= powernv_cpu_freq_attr,

-- 
1.9.3

On Wed, May 18, 2016 at 2:53 PM, Shilpasri G Bhat
[off-list ref] wrote:

This patch adds driver callback for fast_switch and below observations
on schedutil governor are done with this patch.

In POWER8 there is a regression observed with schedutil compared to
ondemand. With schedutil the frequency is not ramping down and is
mostly stuck at max frequency during idle . This is because of the
watchdog timer, an RT task which is fired every 4 seconds which
results in requesting max frequency.

Well, yes, that would be problematic.

I guess the Steve Muckle's cross-CPU utilization updates series might
help (you can find it in the linux-pm patchwork).

In a completely idle system, when there are no processes running apart
from few short running housekeeping tasks (like watchdog) the system is
stuck at max frequency due to 'cpufreq_trigger_update()'

static inline void cpufreq_trigger_update(u64 time)
{
        cpufreq_update_util(time, ULONG_MAX, 0);
}

If there is no noise apart from the watchdog timer the cpu is held at
max frequency for no good reason. On a 16 core system I can see an
increase in 20% idle power with schedutil compared to ondemand
governor.

Below is the trace with 'sched:sched_switch' and 'power:cpu_frequency'
events. Here the watchdog timer that runs for a very small period is
requesting Pmax and this gets triggered regularly.

<idle>-0  19059.992912: sched_switch: prev_comm=swapper/16  prev_state=R
                                ==> next_comm=watchdog/16
watchdog/16-107 19059.992914: cpu_frequency: state=4322000 cpu_id=16
watchdog/16-107 19059.992915: sched_switch: prev_comm=watchdog/16 prev_state=S
                                ==> next_comm=swapper/16

However adding a cpufreq hook in pick_next_task_idle() to decrease the
frequency helped to reduce the problem.

static inline void cpufreq_trigger_idle(u64 time)
{
       cpufreq_update_util(time, 0, 1);
}

This might not be the right fix for the problem, however this thread
is reporting the other short-comings of cpufreq_trigger_update().

On Wed, May 18, 2016 at 2:53 PM, Shilpasri G Bhat
[off-list ref] wrote:

Add fast_switch driver callback to support frequency update in
interrupt context while using schedutil governor. Changing frequency
in interrupt context will remove the jitter on the workloads which can
be seen when a kworker thread is used for the changing the frequency.

Signed-off-by: Shilpasri G Bhat <redacted>

This looks simple enough. :-)

A couple of comments, though.

According to the discussion I had with Peter some time ago, this
should be RELATION_L or you may end up using a frequency that's not
sufficient to meet a deadline somewhere.

Also cpufreq_frequency_table_target() is somewhat heavy-weight
especially if the table is known to be sorted (which I guess is the
case).

On Wed, May 18, 2016 at 11:11:51PM +0200, Rafael J. Wysocki wrote:

On Wed, May 18, 2016 at 2:53 PM, Shilpasri G Bhat
[off-list ref] wrote:

quoted

This patch adds driver callback for fast_switch and below observations
on schedutil governor are done with this patch.

In POWER8 there is a regression observed with schedutil compared to
ondemand. With schedutil the frequency is not ramping down and is
mostly stuck at max frequency during idle . This is because of the
watchdog timer, an RT task which is fired every 4 seconds which
results in requesting max frequency.

Well, yes, that would be problematic.

Right; we need to come up with something for RT tasks; but what happens
if you disable the watchdog? This should be entirely doable and might
give a better comparison.

On Thu, May 19, 2016 at 1:40 PM, Peter Zijlstra [off-list ref] wrote:

On Wed, May 18, 2016 at 11:11:51PM +0200, Rafael J. Wysocki wrote:

quoted

On Wed, May 18, 2016 at 2:53 PM, Shilpasri G Bhat
[off-list ref] wrote:

quoted

This patch adds driver callback for fast_switch and below observations
on schedutil governor are done with this patch.

In POWER8 there is a regression observed with schedutil compared to
ondemand. With schedutil the frequency is not ramping down and is
mostly stuck at max frequency during idle . This is because of the
watchdog timer, an RT task which is fired every 4 seconds which
results in requesting max frequency.

Well, yes, that would be problematic.

Right; we need to come up with something for RT tasks;

I think we need the hints thing for that to be able to distinguish
between RT and the rest.

Also in this particular case it looks like an RT task is the only task
that wakes up often enough and we don't drop the frequency when going
idle.  Do we need a hook somewhere in the idle path?

Hi,

On 05/19/2016 05:10 PM, Peter Zijlstra wrote:

On Wed, May 18, 2016 at 11:11:51PM +0200, Rafael J. Wysocki wrote:

quoted

On Wed, May 18, 2016 at 2:53 PM, Shilpasri G Bhat
[off-list ref] wrote:

quoted

This patch adds driver callback for fast_switch and below observations
on schedutil governor are done with this patch.

In POWER8 there is a regression observed with schedutil compared to
ondemand. With schedutil the frequency is not ramping down and is
mostly stuck at max frequency during idle . This is because of the
watchdog timer, an RT task which is fired every 4 seconds which
results in requesting max frequency.

Well, yes, that would be problematic.

Right; we need to come up with something for RT tasks; but what happens
if you disable the watchdog? This should be entirely doable and might
give a better comparison.

Below are the comparisons by disabling watchdog.
Both schedutil and ondemand have a similar ramp-down trend. And in both the
cases I can see that frequency of the cpu is not reduced in deterministic
fashion. In a observation window of 30 seconds after running a workload I can
see that the frequency is not ramped down on some cpus in the system and are
idling at max frequency.

Below are the sample trace showcasing the frequency request when the cpu enters
idle with schedutil.
<...>-3528  7650.011010: cpu_frequency: state=4322000 cpu_id=120
<...>-3528  7650.027540: sched_switch: prev_comm=ppc64_cpu prev_state=x ==>
			next_comm=swapper/120
<idle>-0    7650.035017: cpu_frequency: state=4322000 cpu_id=120
<idle>-0    7729.683536: cpu_frequency: state=4322000 cpu_id=120
<idle>-0    7729.683552: sched_switch: prev_comm=swapper/120 prev_state=R ==>
			next_comm=kworker/120:1
kworker/120  7729.683565: sched_switch: prev_comm=kworker/120:1 prev_state=S ==>
			 next_comm=swapper/120

However ondemand governor(with watchdog enabled) benefits from the noise created
by watchdog timer and is able to brig down the frequency.

Thanks and Regards,
Shilpa