Thread (5 messages) 5 messages, 2 authors, 2012-12-02

Re: Hang in md-raid1 with 3.7-rcX

From: NeilBrown <hidden>
Date: 2012-11-27 01:05:46
Also in: lkml
Subsystem: software raid (multiple disks) support, the rest · Maintainers: Song Liu, Yu Kuai, Linus Torvalds

On Sat, 24 Nov 2012 10:18:44 +0100 Torsten Kaiser
[off-list ref] wrote:
After my system got stuck with 3.7.0-rc2 as reported in
http://marc.info/?l=linux-kernel&m=135142236520624 LOCKDEP seem to
blame XFS, because it found 2 possible deadlocks. But after these
locking issues where fixed, my system got stuck again with 3.7.0-rc6
as reported in http://marc.info/?l=linux-kernel&m=135344072325490
Dave Chinner thinks its an issue within md, that it gets stuck and
that will then prevent any further xfs activity, and that I should
report it to the raid mailing list.

The issue seems to be that multiple processes (kswapd0, xfsaild/md4
and flush-9:4) get stuck in md_super_wait() like this:
[<ffffffff816b1224>] schedule+0x24/0x60
[<ffffffff814f9dad>] md_super_wait+0x4d/0x80
[<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60
[<ffffffff81500753>] bitmap_unplug+0x173/0x180
[<ffffffff810b6acf>] ? write_cache_pages+0x12f/0x420
[<ffffffff810b6700>] ? set_page_dirty_lock+0x60/0x60
[<ffffffff814e8eb8>] raid1_unplug+0x98/0x110
[<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240
[<ffffffff81278c13>] blk_finish_plug+0x13/0x50

The full hung-tasks stack traces and the output from SysRq+W can be
found at http://marc.info/?l=linux-kernel&m=135344072325490 or in the
LKML thread 'Hang in XFS reclaim on 3.7.0-rc3'.
Yes, it does look like an md bug....
Can you test to see if this fixes it?
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 636bae0..a0f7309 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -963,7 +963,7 @@ static void raid1_unplug(struct blk_plug_cb *cb, bool from_schedule)
 	struct r1conf *conf = mddev->private;
 	struct bio *bio;
 
-	if (from_schedule) {
+	if (from_schedule || current->bio_list) {
 		spin_lock_irq(&conf->device_lock);
 		bio_list_merge(&conf->pending_bio_list, &plug->pending);
 		conf->pending_count += plug->pending_cnt;
I tried to understand how this could happen, but I don't see anything
wrong. Only that md_super_wait() looks like an open coded version of
__wait_event() and could be replaced by using it.
yeah.  md_super_wait was much more complex back when we had to support
barrier operations.  When they were removed it was simplified a lot and as
you say it could be simplifier further.  Patches welcome.

http://marc.info/?l=linux-raid&m=135283030027665 looks like the same
issue, but using ext4 instead of xfs.
yes, sure does.
My setup wrt. md is two normal sata disks on a normal ahci controller
(AMD SB850 southbridge).
Both disks are divided into 4 partitions and each one assembled into a
separate raid1.
One (md5) is used for swap, the others hold xfs filesystems for /boot/
(md4), / (md6) and /home/ (md7).

I will try to provide any information you ask, but I can't reproduce
the hang on demand so gathering more information about that state is
not so easy, but I will try.
I'm fairly confident the above patch will fixes it, and in any case it fixes
a real bug.  So if you could just run with it and confirm in a week or so
that the problem hasn't recurred, that might have to do.

Thanks,
NeilBrown

Attachments

Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help