From: Jeff Moyer <hidden> Date: 2012-03-08 20:42:52
Chris Mason [off-list ref] writes:
On Thu, Mar 08, 2012 at 12:20:26PM -0800, Boaz Harrosh wrote:
quoted
I think I understand this one. It's do to the sync nature introduced
by page_waiting in mkwrite.
Pages go from dirty to writeback for a few reasons. Background
writeout, or O_DIRECT or someone running sync
background writeout shouldn't be queueing up so much work that
synchronous writeout has a 2 second delay.
So now we're back to figuring out how to tell how long I/O will take?
If writeback is issuing random access I/Os to spinning media, you can
bet it might be a while. Today, you could lower nr_requests to some
obscenely small number to improve worst-case latency. I thought there
was some talk about improving the intelligence of writeback in this
regard, but it's a tough problem, especially given that writeback isn't
the only cook in the kitchen.
Cheers,
Jeff
From: Chris Mason <hidden> Date: 2012-03-08 20:55:48
On Thu, Mar 08, 2012 at 03:42:52PM -0500, Jeff Moyer wrote:
Chris Mason [off-list ref] writes:
quoted
On Thu, Mar 08, 2012 at 12:20:26PM -0800, Boaz Harrosh wrote:
quoted
I think I understand this one. It's do to the sync nature introduced
by page_waiting in mkwrite.
Pages go from dirty to writeback for a few reasons. Background
writeout, or O_DIRECT or someone running sync
background writeout shouldn't be queueing up so much work that
synchronous writeout has a 2 second delay.
So now we're back to figuring out how to tell how long I/O will take?
If writeback is issuing random access I/Os to spinning media, you can
bet it might be a while. Today, you could lower nr_requests to some
obscenely small number to improve worst-case latency. I thought there
was some talk about improving the intelligence of writeback in this
regard, but it's a tough problem, especially given that writeback isn't
the only cook in the kitchen.
I'm not against Ted's original idea, but I'd hate to miss the chance to
blame Jens for the block layer making filesystems slow.
Since there is a reliable way to reproduce, it makes sense to track it
down a little more instead of papering over the wait_on_page_writeback
call.
-chris
On Thu, Mar 08, 2012 at 03:42:52PM -0500, Jeff Moyer wrote:
So now we're back to figuring out how to tell how long I/O will take?
If writeback is issuing random access I/Os to spinning media, you can
bet it might be a while. Today, you could lower nr_requests to some
obscenely small number to improve worst-case latency. I thought there
was some talk about improving the intelligence of writeback in this
regard, but it's a tough problem, especially given that writeback isn't
the only cook in the kitchen.
... and it gets worse if there is any kind of I/O prioritization going
on via ionice(), or (as was the case in our example) I/O cgroups were
being used to provide proportional I/O rate controls. I don't think
it's realistic to assume the writeback code can predict how long I/O
will take when it does a submission.
BTW, I'd have to check (having not looked at the application code in
depth; the bug was primarily solved by bisection and reverting the
problem commit) but I'm not entirely sure the thread doing the write
was calling fsync(); the main issue as I understand things was that
the application wasn't expecting the write(2) system call would block
unexpectedly for long periods of time while doing small buffered,
appending I/O's. (Again, for the kind of work that distributed
systems do, 99th percentile latency is important!)
- Ted
From: Chris Mason <hidden> Date: 2012-03-08 21:21:06
On Thu, Mar 08, 2012 at 04:12:21PM -0500, Ted Ts'o wrote:
On Thu, Mar 08, 2012 at 03:42:52PM -0500, Jeff Moyer wrote:
quoted
So now we're back to figuring out how to tell how long I/O will take?
If writeback is issuing random access I/Os to spinning media, you can
bet it might be a while. Today, you could lower nr_requests to some
obscenely small number to improve worst-case latency. I thought there
was some talk about improving the intelligence of writeback in this
regard, but it's a tough problem, especially given that writeback isn't
the only cook in the kitchen.
... and it gets worse if there is any kind of I/O prioritization going
on via ionice(), or (as was the case in our example) I/O cgroups were
being used to provide proportional I/O rate controls. I don't think
it's realistic to assume the writeback code can predict how long I/O
will take when it does a submission.
cgroups do make it much harder because it could be a simple IO priority
inversion. The latencies are just going to be a fact of life for now
and the best choice is to skip the stable pages.
-chris
From: Dave Chinner <david@fromorbit.com> Date: 2012-03-09 08:11:17
On Thu, Mar 08, 2012 at 04:20:54PM -0500, Chris Mason wrote:
On Thu, Mar 08, 2012 at 04:12:21PM -0500, Ted Ts'o wrote:
quoted
On Thu, Mar 08, 2012 at 03:42:52PM -0500, Jeff Moyer wrote:
quoted
So now we're back to figuring out how to tell how long I/O will take?
If writeback is issuing random access I/Os to spinning media, you can
bet it might be a while. Today, you could lower nr_requests to some
obscenely small number to improve worst-case latency. I thought there
was some talk about improving the intelligence of writeback in this
regard, but it's a tough problem, especially given that writeback isn't
the only cook in the kitchen.
... and it gets worse if there is any kind of I/O prioritization going
on via ionice(), or (as was the case in our example) I/O cgroups were
being used to provide proportional I/O rate controls. I don't think
it's realistic to assume the writeback code can predict how long I/O
will take when it does a submission.
cgroups do make it much harder because it could be a simple IO priority
inversion. The latencies are just going to be a fact of life for now
and the best choice is to skip the stable pages.
They have always been a fact of life - just ask anyone that has to
deal with deterministic or "real-time" IO applications.
Unpredictable IO path latencies are not a new problem, and it
doesn't take stable pages to cause sigificant holdoffs in the
writing to a file. For example: writeback triggers triggers delayed
allocation, which locks the extent map and then blocks behind an
allocation already in progress or has to do IO to read in freespace
metadata. The next write comes along from another thread/process and
it has to map a new page and that now blocks on the extent map lock
and won't progress until the delayed allocation in progress
completes....
IO latencies are pretty much unavoidable, so the best thing to do is
to write applications that care about latency to minimise it's
impact as much as possible. Simple techniques like double buffering
and async IO dispatch techniques to decouple the IO stream from the
process/threads that are doing real work are the usual ways of
dealing with this problem.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com