On Thu, 4 Oct 2007 10:10:02 -0400 (EDT), Justin Piszcz wrote:

Also, did performance just go to crap one day or was it gradual?

IIRC I just noticed one day that firefox and vim was stalling. That was
back in February/March I think. At the time the server was running a
2.6.18 kernel, since then I've tried a few kernels in between that and
currently 2.6.23-rc9

Something seems to be periodically causing a lot of activity that
max's out the stripe_cache for a few seconds (when I was trying
to look with blktrace, it seemed pdflush was doing a lot of activity
during this time). 
 
What I had noticed just recently was when I was the only one doing IO
on the server (no NFS running and I was logged in at the console) even
just patching the kernel was crawling to a halt.

Justin.

Cheers,

Andrew