On Mon, Dec 30, 2013 at 12:10:10PM +0000, Morten Rasmussen wrote:

On Sun, Dec 22, 2013 at 04:28:22PM +0000, mark gross wrote:

quoted

On Fri, Dec 20, 2013 at 04:45:40PM +0000, Morten Rasmussen wrote:

quoted

Hi,

One of the requests from the scheduler maintainers at the Energy-aware
Scheduling workshop at Kernel Summit this year was to provide plain text
descriptions of use-cases (workloads) and system topologies. To get that
moving I have written some short texts about some use-cases. In addition
I described a list of issues that today prevent mainly the scheduler
from achieving a good energy/performance balance in common use-cases.
The follow-up emails are structured as follows:

1-6:	Current issues related to energy/performance balance.

We have seen some of these issues as well.  Still from my point of view (which
may not be the most well informed) most of my issues are related to bad choices
on task migrations.

Thanks for sharing your view. In my opinion, all of these issues relate
to task migration choices in one way or another. Lack of knowledge about
the power topology, frequency scaling, and different types of cores
(e.g. big.LITTLE) lead to bad migration choices.

quoted

quoted

7-10:	Use-cases (overall behaviour and energy/performance goals)

I really like your break down of the use cases.  I like the Android focus as
well.  However; can we get some similar workload break downs for representive
data center workloads from other folks?

I don't have much insight into data center workloads, so I was hoping
for input from other folks.

quoted

quoted

11:	DVFS example (for reference)

I'm hoping that this provides some of the background for why I'm
interested in improving energy-awareness in the scheduler. I'm aware
that the use-cases and issues/wishlist don't cover everyone's area of
interest. Input is needed to fix that.

Comments and input are appreciated.

What is missing is more data or modeling tying the SoC charactoristics to
scheduling choices.  You have some (energy per instruction at different
P-states) but there are a lot more topological differences that are important
for proper scheduler choices.  Specifically shared L2's between some cores and
not others, or shared power rails, or if the cores are hyper threaded, or if
there are mutliple sockets.

Agree. This is the missing power topology information in the scheduler.
Power domain information (power rail sharing), including the cost of
waking up the first cpu and additional cpus in the domain, is required.
I guess multi-socket can be modelled that way too?

Most aspects of power management is implementation dependent on ARM, but
a typical big.LITTLE system looks like this:

      little  big
cpu   0   1   2   3
L1   |-| |-| |-| |-|
L2   |-----| |-----|

Two clusters (cpu groups), one little and one big. Cluster shared L2
cache. cpus have (depending on implementation) per cpu C-states and
deeper C-states apply to the entire cluster including the L2. P-states
often apply to the entire cluster (cpu 0-1 and 2-3 in this example).
Clusters may have 1-4 cpus each and doesn't have to be the same for all
clusters (e.g. ARM TC2 is 2+3).

Is that fundamentally different from the systems that you are working
on?

This is not so different for the SoC's I've interacted with.  I'v seen a few
more variants  (as does the ARM SoC's)

One of the newer intel CPU's has a cache picture that looks like a cut and paste
of yours (minus the little/big):

 cpu   0   1   2   3
 L1   |-| |-| |-| |-|
 L2   |-----| |-----|

An older CPU looks just like this but adds SMT to each CPU core.

I was hoping that we could come up with a fairly simplistic energy model
that could guide the scheduling decisions based on data provided by the
vendor. I would start we something very simple and see far we can get
and which data that is necessary.

I keep flip flopping in my mind over what is more important.  Energy modeling or
latency performance measuring.

I mean, one way to look at the world is given a workload with minimal latency
and throughput expectations we need deliver those first and then optimize power.

With poor load balancing we do not deliver on performance expectations typically
in the areas of latencies.  Note, Linux does well on throughput IMO because that
is easier to measure with kstats and other sampling.

what sorts of missing thing are needed to measure and understand when wrong
choices are getting made?  What basic information do we need to capture to know
if we are doing a good job or not?

--mark

Morten