Hmm... I guess I clipped a bit too much for our XFS friends to know
the context.
bonnie is being run on an XFS filesystem on md/raid6. and it gets
this warning a lot and essentially hangs.
NeilBrown
Looks a lot like an XFS problem to me.
Or at least, XFS people would be able to interpret this stack the
best.
Hmm , Ok , I'll post there , I can provide a -complete- boot ->
renboot log of the actions , But it ain't small ~ 649K . So I'll post that on
the back of my website , ie:
http://www.baby-dragons.com/bonnie++1.03c-2.6.26-rc9.console.trace.log
This includes the Sysrq-w & sysrq-b outputs at the bottom .
If you're interested in that kind of thing ;-) .
Hmm... I guess I clipped a bit too much for our XFS friends to know
the context.
bonnie is being run on an XFS filesystem on md/raid6. and it gets
this warning a lot and essentially hangs.
NeilBrown
Nah , wasn't you who did the clipping t'was me . in order to get the
size of the original message below ~ 80K .
Tnx , JimL
--
+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network&System Engineer | 2133 McCullam Ave | Give me Linux |
| babydr@baby-dragons.com | Fairbanks, AK. 99701 | only on AXP |
+------------------------------------------------------------------+
From: Richard Scobie <hidden> Date: 2008-07-22 00:57:18
Mr. James W. Laferriere wrote:
quoted
quoted
INFO: task pdflush:393 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
I see similar things now during resyncs and a few times during md RAID
"checks", since kernel 2.6.25.6-55.fc9.x86_64 on md RAID6 and XFS.
See : http://marc.info/?l=linux-raid&m=121408071129399&w=2
Regards,
Richard
Given that it's a log hang on 2.6.29-rc9, I'd first say add this commit:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=49641f1acfdfd437ed9b0a70b86bf36626c02afe
to your build (went in after -rc9 but before 2.6.26 was released)
and see if that solves the problem.
In more detail, this stack trace implies log I/O has not completed
after the log force was triggered in xfs_buf_wait_unpin(). The above
patch fixes a bug in log I/o dispatch where an non-atomic compare
and decrement would result in log I/O not being dispatched.
So, you've got a hang waiting for log I/o to complete on a kernel
that has a known problem with log I/O dispatch, so it's likely
that's what you've hit.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
From: Mr. James W. Laferriere <hidden> Date: 2008-07-27 01:47:32
Hello All (& Dan) ,
Unfair I/O thruput , Using ...
# uname -a
Linux filesrv2 2.6.26 #1 SMP Wed Jul 23 04:06:23 UTC 2008 i686 pentium4 i386 GNU/Linux
While the bonnie runs (ie: 5 of them) completed , the thruput just
pukes @ ~ 17MB/s , a single run of bonnie on 2.6.26-rc9 was showing a thruput
of ~ 107MB/s & 115MB/s on this system .
I recorded the outputs from the bonnies & posted them (see below) ,
note the "semop: semop failed.\n: Resource temporarily unavailable" at the printout
of the time used , Maybe something I need to report this to Russel & Tim .
Also posted are dmesg output (no errors showing) , attached the .config
for the kernel used .
1. dmesg-200807240256-2.6.26
http://www.baby-dragons.com/dmesg-200807240256-2.6.26
2. 200807240256-2.6.26.config <attached>
3. 131072MB-bonnie++-run-md3-xfs-15runs.log-200807240256-2.6.26 <5 processes>
http://www.baby-dragons.com/131072MB-bonnie++-run-md3-xfs-15runs.log-200807240256-2.6.26
4. 131072MB-bonnie++-run-md3-xfs-15runs.log-200807250222-2.6.26 <1 process>
http://www.baby-dragons.com/131072MB-bonnie++-run-md3-xfs-15runs.log-200807250222-2.6.26
5. device-thruput-200807260411-2.6.26.log
http://www.baby-dragons.com/device-thruput-200807260411-2.6.26.log
This shows that sda is being driven unmercifully while other drives in
the array aren't getting anywhere near the expected I/O , Very uneven .
Tia , JimL
--
+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network&System Engineer | 2133 McCullam Ave | Give me Linux |
| babydr@baby-dragons.com | Fairbanks, AK. 99701 | only on AXP |
+------------------------------------------------------------------+
Looks a lot like an XFS problem to me.
Or at least, XFS people would be able to interpret this stack the
best.
I presume if it is waiting in xfs_buf_wait_unpin() for a long
time (>2min) then maybe a journal-log io completion hasn't come
back to say that the matching buffer item has made to the ondisk log.
i.e the buffer hasn't been unpinned yet (pincount>0) which is supposed
to happen when its data hits the ondisk log.
Hmm... I guess I clipped a bit too much for our XFS friends to know
the context.
bonnie is being run on an XFS filesystem on md/raid6. and it gets
this warning a lot and essentially hangs.
Just for the record,
in rc-9 we hadn't removed the QUEUE_ORDERED tag check yet and
so I presume for md/raid6, barriers will be disabled.
So barrier writes on the log won't be being issued.
I don't see that as anything to do with the problem here -
that is more of an issue on replay if we have the cache on
and no barrier support - I just thought I'd mention it.
--Tim