On Sunday November 25, nagilum@nagilum.org wrote:
----- Message from nagilum@nagilum.org ---------
Date: Sat, 24 Nov 2007 12:02:09 +0100
From: Nagilum [off-list ref]
Reply-To: Nagilum [off-list ref]
Subject: raid5 reshape/resync
To: linux-raid@vger.kernel.org
quoted
Hi,
I'm running 2.6.23.8 x86_64 using mdadm v2.6.4.
I was adding a disk (/dev/sdf) to an existing raid5 (/dev/sd[a-e] -> md0)
During that reshape (at around 4%) /dev/sdd reported read errors and
went offline.
Sad.
quoted
I replaced /dev/sdd with a new drive and tried to reassemble the array
(/dev/sdd was shown as removed and now as spare).
There must be a step missing here.
Just because one drive goes offline, that doesn't mean that you need
to reassemble the array. It should just continue with the reshape
until that is finished. Did you shut the machine down or did it crash
or what
quoted
Assembly worked but it would not run unless I use --force.
That suggests an unclean shutdown. Maybe it did crash?
quoted
Since I'm always reluctant to use force I put the bad disk back in,
this time as /dev/sdg . I re-added the drive and could run the array.
The array started to resync (since the disk can be read until 4%) and
then I marked the disk as failed. Now the array is "active, degraded,
recovering":
It should have restarted the reshape from whereever it was up to, so
it should have hit the read error almost immediately. Do you remember
where it started the reshape from? If it restarted from the beginning
that would be bad.
Did you just "--assemble" all the drives or did you do something else?
quoted
What I find somewhat confusing/disturbing is that does not appear to
utilize /dev/sdd. What I see here could be explained by md doing a
RAID5 resync from the 4 drives sd[a-c,e] to sd[a-c,e,f] but I would
have expected it to use the new spare sdd for that. Also the speed is
md cannot recover to a spare while a reshape is happening. It
completes the reshape, then does the recovery (as you discovered).
quoted
unusually low which seems to indicate a lot of seeking as if two
operations are happening at the same time.
Well reshape is always slow as it has to read from one part of the
drive and write to another part of the drive.
quoted
Also when I look at the data rates it looks more like the reshape is
continuing even though one drive is missing (possible but risky).
Yes, that is happening.
quoted
Can someone relief my doubts as to whether md does the right thing here?
Thanks,
I believe it is do "the right thing".
quoted
----- End message from nagilum@nagilum.org -----
Ok, so the reshape tried to continue without the failed drive and
after that resynced to the new spare.
As I would expect.
Unfortunately the result is a mess. On top of the Raid5 I have
Hmm. This I would not expect.
dm-crypt and LVM.
Although dmcrypt and LVM dont appear to have a problem the filesystems
on top are a mess now.
Can you be more specific about what sort of "mess" they are in?
NeilBrown
I still have the failed drive, I can read the superblock from that
drive and up to 4% from the beginning and probably backwards from the
end towards that point.
So in theory it could be possible to reorder the stripe blocks which
appears to have been messed up.(?)
Unfortunately I'm not sure what exactly went wrong or what I did
wrong. Can someone please give me hint?
Thanks,
Alex.
========================================================================
# _ __ _ __ http://www.nagilum.org/ \n icq://69646724 #
# / |/ /__ ____ _(_) /_ ____ _ nagilum@nagilum.org \n +491776461165 #
# / / _ `/ _ `/ / / // / ' \ Amiga (68k/PPC): AOS/NetBSD/Linux #
# /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/ Mac (PPC): MacOS-X / NetBSD /Linux #
# /___/ x86: FreeBSD/Linux/Solaris/Win2k ARM9: EPOC EV6 #
========================================================================
----------------------------------------------------------------
cakebox.homeunix.net - all the machine one needs..