On Thu, Mar 08, 2012 at 03:42:52PM -0500, Jeff Moyer wrote:

So now we're back to figuring out how to tell how long I/O will take?
If writeback is issuing random access I/Os to spinning media, you can
bet it might be a while.  Today, you could lower nr_requests to some
obscenely small number to improve worst-case latency.  I thought there
was some talk about improving the intelligence of writeback in this
regard, but it's a tough problem, especially given that writeback isn't
the only cook in the kitchen.

... and it gets worse if there is any kind of I/O prioritization going
on via ionice(), or (as was the case in our example) I/O cgroups were
being used to provide proportional I/O rate controls.  I don't think
it's realistic to assume the writeback code can predict how long I/O
will take when it does a submission.

BTW, I'd have to check (having not looked at the application code in
depth; the bug was primarily solved by bisection and reverting the
problem commit) but I'm not entirely sure the thread doing the write
was calling fsync(); the main issue as I understand things was that
the application wasn't expecting the write(2) system call would block
unexpectedly for long periods of time while doing small buffered,
appending I/O's.  (Again, for the kind of work that distributed
systems do, 99th percentile latency is important!)

						- Ted