debugging md2_resync hang at raise_barrier

Started by Ray Morris on Mar 1, 2012 · 4 messages from 2 authors · Last reply Mar 13, 2012

open the first message on its own page

Message details and related content
I am attempting to debug a hang in raid1 and possibly one raid5.
I have experienced the same problem with many kernel versions
over a couple of years, and with disparate hardware.

My current plan, if noone more experienced suggests I do otherwise, is
to compile a kernel with some printk() in strategic locations and try to 
narrow down the problem. I know very little about kernel work, so I am 
seeking suggestions from those who know better than I.

In the case logged below, it appears to hang at raise_barrier in md2_resync
at raise_barrier, then further access to the device hangs. I'm just a Perl 
programmer who dabbles in C, but my intuition said I check that if perhaps 
lower_barrier had been called with conf->barrier already at zero, so that's
the one printk I've added so far. It may take a week or more before it 
crashes again, so is there any more debugging I should add before waiting 
for it to hang?

Also below is some older logging from similar symptoms with raid5, 
hanging at raid5_quiesce. I got rid of the raid5 in hopes of getting 
rid of the problem, but if anyone has suggestions on how to further 
debug that I maybe be able to add a raid5 array.

The load when I've noticed it is rsync to LVM volumes with snapshots.
After some discussion, lvm-devel suggested linux-raid would be the next 
logical step. Tested kernels include 2.6.32-220.4.1.el6.x86_64 
2.6.32.26-175.fc12.x86_64, vmlinuz-2.6.32.9-70.fc12.x86_64, and others.
Since I already have updated the kernel several times in the last couple 
of years, I figured I'd try some debugging with the current EL 6 kernel.

Anyway, any thoughts on how to debug, where to stick some printk, other 
debugging functions? 


Feb 28 23:16:52 clonebox3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 28 23:16:52 clonebox3 kernel: md2_resync    D 0000000000000002     0  9681      2 0x00000080
Feb 28 23:16:52 clonebox3 kernel: ffff88006b961bb0 0000000000000046 0000000000000000 0000000300000001
Feb 28 23:16:52 clonebox3 kernel: ffff88006b961b30 ffff88003c9ea7d0 0000000000000286 0000000000000003
Feb 28 23:16:52 clonebox3 kernel: ffff880037c8d038 ffff88006b961fd8 000000000000f4e8 ffff880037c8d038
Feb 28 23:16:52 clonebox3 kernel: Call Trace:
Feb 28 23:16:52 clonebox3 kernel: [<ffffffffa021ae0d>] raise_barrier+0x9d/0x1a0 [raid1]

154 more lines

Message details and related content
On Wed, 29 Feb 2012 18:44:13 -0600 Ray Morris [off-list ref] wrote:
I am attempting to debug a hang in raid1 and possibly one raid5.
I have experienced the same problem with many kernel versions
over a couple of years, and with disparate hardware.

My current plan, if noone more experienced suggests I do otherwise, is
to compile a kernel with some printk() in strategic locations and try to 
narrow down the problem. I know very little about kernel work, so I am 
seeking suggestions from those who know better than I.

In the case logged below, it appears to hang at raise_barrier in md2_resync
at raise_barrier, then further access to the device hangs. I'm just a Perl 
programmer who dabbles in C, but my intuition said I check that if perhaps 
lower_barrier had been called with conf->barrier already at zero, so that's
the one printk I've added so far. It may take a week or more before it 
crashes again, so is there any more debugging I should add before waiting 
for it to hang?

Also below is some older logging from similar symptoms with raid5, 
hanging at raid5_quiesce. I got rid of the raid5 in hopes of getting 
rid of the problem, but if anyone has suggestions on how to further 
debug that I maybe be able to add a raid5 array.

The load when I've noticed it is rsync to LVM volumes with snapshots.
After some discussion, lvm-devel suggested linux-raid would be the next 
logical step. Tested kernels include 2.6.32-220.4.1.el6.x86_64 
2.6.32.26-175.fc12.x86_64, vmlinuz-2.6.32.9-70.fc12.x86_64, and others.
Since I already have updated the kernel several times in the last couple 
of years, I figured I'd try some debugging with the current EL 6 kernel.

Anyway, any thoughts on how to debug, where to stick some printk, other 
debugging functions? 
I might know what is happening.

It is kind-a complicated and involved the magic code in
block/blk-core.c:generic_make_request which turns recursive calls into tail
recursion.

The fs sends a request to dm.
dm split it in 2 for some reason and sends them both to md.
This involves them getting queued in generic_make_request.
The first gets actioned by md/raid1 and converted into a request to the
underlying device (it must be a read request for this to happen - so just one
device).  This gets added to the queue and is counted in nr_pending.

At this point sync_request is called by another thread and it tries to
raise_battier.  It gets past the first hurdle, increments ->barrier, and
waits for nr_pending to hit zero.

Now the second request from dm to md is passed to raid1.c:make_request where
it tries to wait_barrier.  This blocks because ->barrier is up, and we have a
deadlock - the request to the underlying device will not progress until this
md request progresses, and it is stuck.

This patch might fix it.  Maybe.  If it compiles.

NeilBrown

Index: linux-2.6.32-SLE11-SP1/drivers/md/raid1.c
===================================================================
--- linux-2.6.32-SLE11-SP1.orig/drivers/md/raid1.c	2012-03-01 12:28:05.000000000 +1100
+++ linux-2.6.32-SLE11-SP1/drivers/md/raid1.c	2012-03-01 12:28:22.427992913 +1100
@@ -695,7 +695,11 @@ static void wait_barrier(conf_t *conf)
 	spin_lock_irq(&conf->resync_lock);
 	if (conf->barrier) {
 		conf->nr_waiting++;
-		wait_event_lock_irq(conf->wait_barrier, !conf->barrier,
+		wait_event_lock_irq(conf->wait_barrier,
+				    !conf->barrier ||
+				    (current->bio_tail &&
+				     current->bio_list &&
+				     conf->nr_pending),
 				    conf->resync_lock,
 				    raid1_unplug(conf->mddev->queue));
 		conf->nr_waiting--;
Message details and related content
On Wed, 29 Feb 2012 18:44:13 -0600 Ray Morris [off-list ref]
wrote:
quoted
I am attempting to debug a hang in raid1 and possibly one raid5.

On Thu, 1 Mar 2012 12:34:18 +1100
NeilBrown [off-list ref] wrote:
I might know what is happening.

It is kind-a complicated and involved the magic code in
block/blk-core.c:generic_make_request which turns recursive calls
into tail recursion.
...
This patch might fix it.  Maybe.  If it compiles.
That appears to have solved it. Thanks so much. The delay in my
response is because I wanted to give it some time to recur, as it
didn't always manifest right away. Is there anything I should do
regarding this path in future versions? Is this patch already in or
planned to be in future versions, or should users possibly keep the
patch on hand in case it's still needed later? Is there any testing
or anything that I need to do to get included in future versions?


-- 
Ray Morris
support@bettercgi.com

Strongbox - The next generation in site security:
http://www.bettercgi.com/strongbox/

Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/

Strongbox / Throttlebox affiliate program:
http://www.bettercgi.com/affiliates/user/register.php




On Thu, 1 Mar 2012 12:34:18 +1100
NeilBrown [off-list ref] wrote:
On Wed, 29 Feb 2012 18:44:13 -0600 Ray Morris [off-list ref]
wrote:
quoted
I am attempting to debug a hang in raid1 and possibly one raid5.
I have experienced the same problem with many kernel versions
over a couple of years, and with disparate hardware.

My current plan, if noone more experienced suggests I do otherwise,
is to compile a kernel with some printk() in strategic locations
and try to narrow down the problem. I know very little about kernel
work, so I am seeking suggestions from those who know better than I.

In the case logged below, it appears to hang at raise_barrier in
md2_resync at raise_barrier, then further access to the device
hangs. I'm just a Perl programmer who dabbles in C, but my
intuition said I check that if perhaps lower_barrier had been
called with conf->barrier already at zero, so that's the one printk
I've added so far. It may take a week or more before it crashes
again, so is there any more debugging I should add before waiting
for it to hang?

Also below is some older logging from similar symptoms with raid5, 
hanging at raid5_quiesce. I got rid of the raid5 in hopes of
getting rid of the problem, but if anyone has suggestions on how to
further debug that I maybe be able to add a raid5 array.

The load when I've noticed it is rsync to LVM volumes with
snapshots. After some discussion, lvm-devel suggested linux-raid
would be the next logical step. Tested kernels include
2.6.32-220.4.1.el6.x86_64 2.6.32.26-175.fc12.x86_64,
vmlinuz-2.6.32.9-70.fc12.x86_64, and others. Since I already have
updated the kernel several times in the last couple of years, I
figured I'd try some debugging with the current EL 6 kernel.

Anyway, any thoughts on how to debug, where to stick some printk,
other debugging functions? 
I might know what is happening.

It is kind-a complicated and involved the magic code in

44 more quoted lines

Message details and related content
On Mon, 12 Mar 2012 13:46:03 -0500 Ray Morris [off-list ref] wrote:
quoted
On Wed, 29 Feb 2012 18:44:13 -0600 Ray Morris [off-list ref]
wrote:
quoted
I am attempting to debug a hang in raid1 and possibly one raid5.

On Thu, 1 Mar 2012 12:34:18 +1100
NeilBrown [off-list ref] wrote:
quoted
I might know what is happening.

It is kind-a complicated and involved the magic code in
block/blk-core.c:generic_make_request which turns recursive calls
into tail recursion.
...
quoted
This patch might fix it.  Maybe.  If it compiles.
That appears to have solved it. Thanks so much. The delay in my
response is because I wanted to give it some time to recur, as it
didn't always manifest right away. Is there anything I should do
regarding this path in future versions? Is this patch already in or
planned to be in future versions, or should users possibly keep the
patch on hand in case it's still needed later? Is there any testing
or anything that I need to do to get included in future versions?
Thanks a lot.

I will probably submit this fix to Linus shortly after 3.3 is out, with a
request for it to be included in other -stable releases.

You seem to be using kernels from Redhat.  If you want them to include the
patch you should probably raise it as an issue with them.

Thanks,
NeilBrown
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help