On Thu, 7 Apr 2011 22:00:58 +0200 sanktnelson 1 [off-list ref]
wrote:

Hi all,
I need some advice on recovering from a catastrophic RAID5 failure
(and a possible f***-up on my part in trying to recover). I didn't
find this covered in any of the howtos out there, so here is my
problem:
I was trying to rebuild my RAID5 (5 drives) after I had replaced one
failed drive (sdc) with a fresh one. During the rebuild another drive
(sdd) failed. so I was left with the sdc as spare (S) and sdd failed
(F).
first question: what should I have done at this point? I'm fairly
certain that the error that failed sdd was a fluke, loose cable or
something, so I wanted mdadm to just assume it was clean and retry the
rebuild.

What I would have done is stop the array (mdadm -S /dev/md0) and re-assemble
it with --force.  This would get you the degraded array back.
Then I would backup and data that I really couldn't live without - just in
case.

Then I would stop the array, dd-rescue sdd to some other device, possible
sdc, and assemble the array with the known-good devices and the new device
(which might have been sdc) and NOT sdd. 
This would give me a degraded array of good devices.
Then I would add another good device - maybe sdc if I thought that it was
just a ready error and writes would work.
Then wait and hope.

what I actually did was reboot the machine in the hope it would just
restart the array in the previous degraded state, which of course it
did not. Instead all drives except the failed sdd were reported as
spare(S) in /proc/mdstat.

so I tried

root@enterprise:~# mdadm --run /dev/md0
mdadm: failed to run array /dev/md0: Input/output error

syslog showed at this point:

Apr  7 20:37:49 localhost kernel: [  893.981851] md: kicking non-fresh
sdd1 from array!
Apr  7 20:37:49 localhost kernel: [  893.981864] md: unbind<sdd1>
Apr  7 20:37:49 localhost kernel: [  893.992526] md: export_rdev(sdd1)
Apr  7 20:37:49 localhost kernel: [  893.995844] raid5: device sdb1
operational as raid disk 3
Apr  7 20:37:49 localhost kernel: [  893.995848] raid5: device sdf1
operational as raid disk 4
Apr  7 20:37:49 localhost kernel: [  893.995852] raid5: device sde1
operational as raid disk 2
Apr  7 20:37:49 localhost kernel: [  893.996353] raid5: allocated 5265kB for md0
Apr  7 20:37:49 localhost kernel: [  893.996478] 3: w=1 pa=0 pr=5 m=1
a=2 r=5 op1=0 op2=0
Apr  7 20:37:49 localhost kernel: [  893.996482] 4: w=2 pa=0 pr=5 m=1
a=2 r=5 op1=0 op2=0
Apr  7 20:37:49 localhost kernel: [  893.996485] 2: w=3 pa=0 pr=5 m=1
a=2 r=5 op1=0 op2=0
Apr  7 20:37:49 localhost kernel: [  893.996488] raid5: not enough
operational devices for md0 (2/5 failed)
Apr  7 20:37:49 localhost kernel: [  893.996514] RAID5 conf printout:
Apr  7 20:37:49 localhost kernel: [  893.996517]  --- rd:5 wd:3
Apr  7 20:37:49 localhost kernel: [  893.996520]  disk 2, o:1, dev:sde1
Apr  7 20:37:49 localhost kernel: [  893.996522]  disk 3, o:1, dev:sdb1
Apr  7 20:37:49 localhost kernel: [  893.996525]  disk 4, o:1, dev:sdf1
Apr  7 20:37:49 localhost kernel: [  893.996898] raid5: failed to run
raid set md0
Apr  7 20:37:49 localhost kernel: [  893.996901] md: pers->run() failed ...

so I figured I'd re-add sdd:

root@enterprise:~# mdadm --re-add /dev/md0 /dev/sdd1
mdadm: re-added /dev/sdd1
root@enterprise:~# mdadm --run /dev/md0
mdadm: started /dev/md0

This should be effectively equivalent to --assemble --force (I think).

Apr  7 20:44:16 localhost kernel: [ 1281.139654] md: bind<sdd1>
Apr  7 20:44:16 localhost mdadm[1523]: SpareActive event detected on
md device /dev/md0, component device /dev/sdd1
Apr  7 20:44:32 localhost kernel: [ 1297.147581] raid5: device sdd1
operational as raid disk 0
Apr  7 20:44:32 localhost kernel: [ 1297.147585] raid5: device sdb1
operational as raid disk 3
Apr  7 20:44:32 localhost kernel: [ 1297.147588] raid5: device sdf1
operational as raid disk 4
Apr  7 20:44:32 localhost kernel: [ 1297.147591] raid5: device sde1
operational as raid disk 2
Apr  7 20:44:32 localhost kernel: [ 1297.148102] raid5: allocated 5265kB for md0
Apr  7 20:44:32 localhost kernel: [ 1297.148704] 0: w=1 pa=0 pr=5 m=1
a=2 r=5 op1=0 op2=0
Apr  7 20:44:32 localhost kernel: [ 1297.148708] 3: w=2 pa=0 pr=5 m=1
a=2 r=5 op1=0 op2=0
Apr  7 20:44:32 localhost kernel: [ 1297.148712] 4: w=3 pa=0 pr=5 m=1
a=2 r=5 op1=0 op2=0
Apr  7 20:44:32 localhost kernel: [ 1297.148715] 2: w=4 pa=0 pr=5 m=1
a=2 r=5 op1=0 op2=0
Apr  7 20:44:32 localhost kernel: [ 1297.148718] raid5: raid level 5
set md0 active with 4 out of 5 devices, algorithm 2
Apr  7 20:44:32 localhost kernel: [ 1297.148722] RAID5 conf printout:
Apr  7 20:44:32 localhost kernel: [ 1297.148725]  --- rd:5 wd:4
Apr  7 20:44:32 localhost kernel: [ 1297.148728]  disk 0, o:1, dev:sdd1
Apr  7 20:44:32 localhost kernel: [ 1297.148731]  disk 2, o:1, dev:sde1
Apr  7 20:44:32 localhost kernel: [ 1297.148734]  disk 3, o:1, dev:sdb1
Apr  7 20:44:32 localhost kernel: [ 1297.148737]  disk 4, o:1, dev:sdf1
Apr  7 20:44:32 localhost kernel: [ 1297.148779] md0: detected
capacity change from 0 to 6001196531712
Apr  7 20:44:32 localhost kernel: [ 1297.149047]  md0:RAID5 conf printout:
Apr  7 20:44:32 localhost kernel: [ 1297.149559]  --- rd:5 wd:4
Apr  7 20:44:32 localhost kernel: [ 1297.149562]  disk 0, o:1, dev:sdd1
Apr  7 20:44:32 localhost kernel: [ 1297.149565]  disk 1, o:1, dev:sdc1
Apr  7 20:44:32 localhost kernel: [ 1297.149568]  disk 2, o:1, dev:sde1
Apr  7 20:44:32 localhost kernel: [ 1297.149570]  disk 3, o:1, dev:sdb1
Apr  7 20:44:32 localhost kernel: [ 1297.149573]  disk 4, o:1, dev:sdf1
Apr  7 20:44:32 localhost kernel: [ 1297.149846] md: recovery of RAID array md0
Apr  7 20:44:32 localhost kernel: [ 1297.149849] md: minimum
_guaranteed_  speed: 1000 KB/sec/disk.
Apr  7 20:44:32 localhost kernel: [ 1297.149852] md: using maximum
available idle IO bandwidth (but not more than 200000 KB/sec) for
recovery.
Apr  7 20:44:32 localhost kernel: [ 1297.149858] md: using 128k
window, over a total of 1465135872 blocks.
Apr  7 20:44:32 localhost kernel: [ 1297.188272]  unknown partition table
Apr  7 20:44:32 localhost mdadm[1523]: RebuildStarted event detected
on md device /dev/md0

I figured this was definitely wrong, since I still couldn't mount
/dev/md0, so I manually failed sdc and sdd to stop any further
destruction on my part and to go seek expert help, so here I am. Is my
data still there or have the first few hundred MB been zeroed to
initialize a fresh array? how do I force mdadm to assume sdd is fresh
and give me access to the array without any writes happening to it?
Sorry to come running to the highest authority on linuxraid here, but
all the howtos out there are pretty thin when it comes to anything
more complicated than creating and array and recovering from one
failed drive.

Well I wouldn't have failed the devices.  I would simply have stopped the
array
  mdamd -S /dev/md0

But I'm very surprised that this didn't work.
md and mdadm never write zeros to initialise anything (Except a bitmap).

Maybe the best thing to do at this point is post the output of
  mdadm -E /dev/sd[bcdef]1
and I'll see if I can make sense of it.

NeilBrown

Any advice (even if it is just for doing the right thing the next
time), is greatly appreciated.

-Felix
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html