Hi Vincent,

On Mon, Dec 02, 2019 at 02:45:42PM +0100 Vincent Guittot wrote:

On Mon, 2 Dec 2019 at 05:02, Dave Chinner [off-list ref] wrote:

...

quoted

So, we can fiddle with workqueues, but it doesn't address the
underlying issue that the scheduler appears to be migrating
non-bound tasks off a busy CPU too easily....

The root cause of the problem is that the sched_wakeup_granularity_ns
is in the same range or higher than load balance period. As Peter
explained, This make the kworker waiting for the CPU for several load
period and a transient unbalanced state becomes a stable one that the
scheduler to fix. With default value, the scheduler doesn't try to
migrate any task.

There are actually two issues here.   With the high wakeup granularity
we get the user task actively migrated. This causes the significant
performance hit Ming was showing. With the fast wakeup_granularity
(or smaller IOs - 512 instead of 4k) we get, instead, the user task
migrated at wakeup to a new CPU for every IO completion.

This is the 11k migrations per sec doing 11k iops.  In this test it
is not by itself causing the measured performance issue. It generally
flips back and forth between 2 cpus for large periods. I think it is
crossing cache boundaries at times (but I have not looked closely
at the traces compared to the topology, yet).

The active balances are what really hurts in thie case but I agree
that seems to be a tuning problem.


Cheers,
Phil

Then, I agree that having an ack close to the request makes sense but
forcing it on the exact same CPU is too restrictive IMO. Being able to
use another CPU on the same core should not harm the performance and
may even improve it. And that may still be the case while CPUs share
their cache.

quoted

-Dave.

[*] Pay attention to the WQ_POWER_EFFICIENT definition for a work
queue: it's designed for interrupt routines that defer work via work
queues to avoid doing work on otherwise idle CPUs. It does this by
turning the per-cpu wq into an unbound wq so that work gets
scheduled on a non-idle CPUs in preference to the local idle CPU
which can then remain in low power states.

That's the exact opposite of what using WQ_UNBOUND ends up doing in
this IO completion context: it pushes the work out over idle CPUs
rather than keeping them confined on the already busy CPUs where CPU
affinity allows the work to be done quickly. So while WQ_UNBOUND
avoids the user task being migrated frequently, it results in the
work being spread around many more CPUs and we burn more power to do
the same work.

--
Dave Chinner
david@fromorbit.com

--