--- On Wed, 8/20/08, Dave Chinner <david@fromorbit.com> wrote:

Ok, I thought it might be the tiny log, but it didn't
improve anything
here when increased the log size, or the log buffer size.

Looking at the block trace, I think elevator merging is
somewhat busted. I'm
seeing adjacent I/Os being dispatched without having been
merged.  e.g:

[snip]

Also, CFQ appears to not be merging WRITE_SYNC bios or
issuing them
with any urgency.  The result of this is that it stalls the
XFS
transaction subsystem by capturing all the log buffers in
the
elevator and not issuing them. e.g.:

[snip]

The I/Os are merged, but there's still that 700ms delay
before dispatch.
i was looking at this a while back but didn't get to
finishing it off.
i.e.:

http://oss.sgi.com/archives/xfs/2008-01/msg00151.html
http://oss.sgi.com/archives/xfs/2008-01/msg00152.html

I'll have a bit more of a look at this w.r.t to
compilebench performance,
because it seems like a similar set of problems that I was
seeing back
then...

I concur your observation, esp. w.r.t. XFS and CFQ clashing:

http://gus3.typepad.com/i_am_therefore_i_think/2008/07/finding-the-fas.html

CFQ is the default on most Linux systems AFAIK; for decent XFS performance one needs to switch to "noop" or "deadline". I wasn't sure if it was broken code, or simply base assumptions in conflict (XFS vs. CFQ). Your log output sheds light on the matter for me, thanks.