Re: False waker detection in BFQ

Il giorno 25 ago 2021, alle ore 18:43, Jan Kara [off-list ref] ha scritto:

On Mon 23-08-21 18:06:18, Jan Kara wrote:

quoted

On Mon 23-08-21 15:58:25, Paolo Valente wrote:

quoted

quoted

Currently I'm running wider set of benchmarks for the patches to see
whether I didn't regress anything else. If not, I'll post the patches to
the list.

Any news?

It took a while for all those benchmarks to run. Overall results look sane,
I'm just verifying by hand now whether some of the localized regressions
(usually specific to a particular fs+machine config) are due to a measurement
noise or real regressions...

OK, so after some manual analysis I've found out that dbench indeed becomes
more noisy with my changes for high numbers of processes. I'm leaving for
vacation soon so I will not be probably able to debug it before I leave but
let me ask you one thing: The problematic change seems to be mostly a
revert of 7cc4ffc55564 ("block, bfq: put reqs of waker and woken in
dispatch list") and I'm currently puzzled why it has such an effect. What
I've found out is that 7cc4ffc55564 results in IO of higher priority
process being injected into the time slice of lower priority process and
thus there's always only single busy queue (of the lower priority process)
and thus higher priority process queue never gets scheduled. As a result
higher priority IO always competes with lower priority IO and there's no
service differentiation (we get 50/50 split of throughput between the
processes despite different IO priorities).

I need a little help here.  Since high-priority I/O is immediately
injected, I wonder why it does not receive all the bandwidth it
demands.  Maybe, from your analysis, you have an answer.  Perhaps it
happens because:
1) high-priority I/O is FIFO-queued with lower-priority I/O in the
   dispatch list?
or
2) immediate injection prevents idling from being performed in favor
   of high-priority I/O?

 And this scenario shows that
always injecting IO of waker/wakee isn't desirable, especially in a way as
done in 7cc4ffc55564 where injected IO isn't accounted within BFQ at all
(which easily allows for service degradation unnoticed by BFQ).

Not sure that accounting would help high-priority I/O in your scenario.

 That's why
I've basically reverted that commit on the ground that on next dispatch we
call bfq_select_queue() which will see waker/wakee has IO to do and can
decide to inject the IO anyway. We do more CPU work but the IO pattern
should be similar. But apparently I was wrong :)

For the pattern to be similar, I guess that, when new high-priority
I/O arrives, this I/O should preempt lower-priority I/O.
Unfortunately, this is not always the case, depending on other
parameters.  Waker/wakee I/O is guaranteed to be injected only when the
in-service queue has no I/O.

At any rate, probably we can solve this puzzle by just analyzing a
trace in which you detect a loss of throughput without 7cc4ffc55564.
The best case would be one with the minimum possible number of
threads, to get a simpler trace.

I just wanted to bounce
this off of you if you have any suggestion what to look for or any tips
regarding why 7cc4ffc55564 apparently achieves much more reliable request
injection for dbench.

I hope my considerations above help a little bit.

Thanks,
Paolo

								Honza
-- 
Jan Kara [off-list ref]
SUSE Labs, CR