On Mon, Sep 20, 2010 at 10:26 PM, Neil Brown [off-list ref] wrote:

On Wed, 15 Sep 2010 17:49:44 -0400
Mike Hartman [off-list ref] wrote:

quoted

quoted

quoted

Hmmm..
 Can you try mounting with
   -o barrier=0

just to see if my theory is at all correct?

Thanks,
NeilBrown

Progress report:

I made the barrier change shortly after sending my last message (about
40 hours ago). With that in place, I was able to finish emptying one
of the non-assimilated drives onto the array, after which I added that
drive as a hot spare and started the process to grow the array onto it
- the same procedure I've been applying since I created the RAID the
other week. No problems so far, and the reshape is at 46%.

It's hard to be positive that the barrier deactivation is responsible
yet though - while the last few lockups have only been 1-16 hours
apart, I believe the first two had at least 2 or 3 days between them.
I'll keep the array busy to enhance the chances of a lockup though -
each one so far has been during a reshape or a large batch of writing
to the array's partition. If I make it another couple days (meaning
time for this reshape to complete, another drive to be emptied onto
the array, and another reshape at least started) I'll be pretty
confident the problem has been identified.

Thanks for the update.

quoted

Assuming the barrier is the culprit (and I'm pretty sure you're right)
what are the consequences of just leaving it off? I gather the idea of
the barrier is to prevent journal corruption in the event of a power
failure or other sudden shutdown, which seems pretty important, but it
also doesn't seem like it was enabled by default in ext3/4 until 2008,
which makes it seem less critical.

Correct.  Without the barriers the chance of corruption during powerfail is
higher.  I don't really know how much higher, it depends a lot on the
filesystem design and the particular implementation.  I think ext4 tends to
be fairly safe - after all some devices don't support barriers and it has to
do best-effort on those too.

quoted

Even if the ultimate solution for me is to just leave it disabled I'm
happy to keep trying patches if you want to get it properly fixed in
md. We may have to come up with an alternate way to work the array
hard enough to trigger the lockups though - my last 1.5TB drive is
what's being merged in now. After that completes I only have one more
pair of 750GBs (that will have to be shoehorned in using RAID0 again).
I do have a single 750GB left over, so I'll probably find a mate for
it and get it added to. After that we're maxed out on hardware for a
while.

Mike

I'll stare at the code a bit more and see if anything jumps out at me.

Thanks,
NeilBrown

I've just finished my last grow-and-copy with no problems. The only
drive that's not part of the array now is the leftover 750GB, which is
now empty. I haven't experienced any further lockups so your barrier
diagnosis seems to be spot on. I'm planning to just leave that option
turned off, but as I said, I'm happy to test any patches you come up
with. Thanks for all your help.

Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html