Re: RAID5 disk failure during rebuild of spare, any chance of recovery when one of the failed devices is suspected to be intact?
From: Nicolas Jungers <hidden>
Date: 2010-08-16 16:37:56
On 08/16/2010 06:27 PM, Tor Arne Vestbø wrote:
On Mon, Aug 16, 2010 at 10:43 AM, Tim Small[off-list ref] wrote:quoted
On 16/08/10 07:12, Nicolas Jungers wrote:quoted
On 08/16/2010 07:54 AM, Tor Arne Vestbø wrote:quoted
You mean you sdc and sde plus either sdb or sdd, depending on which one I think is more sane a this point?I'd try both. Do a ddrescue of the failing one and try that (with copy of the others) and check what's coming out.As an alternative to using ddrescue, you could quickly prototype various arrangements (without writing anything to the drives) using a device-mapper copy-on-write mapping - I posted some details to the list a while back when I was trying to use this to reconstruct a hw raid array... Check the list archives for details.Cool, here's what I tried: Created spares files for each of the devices dd if=/dev/zero of=sdb_cow bs=1 count=0 seek=2GB Mapped that to a loop device losetup /dev/loop1 sdb_cow Then ran the following for each device: cow_size=`blockdev --getsize /dev/sdb1` chunk_size=64 echo "0 $cow_size snapshot /dev/sdb1 /dev/loop1 p $chunk_size" | dmsetup create sdb1_cow After these were created I tried the following: # mdadm -v -C /dev/md0 -l5 -n4 /dev/mapper/sdb1_cow /dev/mapper/sdc1_cow missing /dev/mapper/sde1_cow mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 64K mdadm: /dev/mapper/sdb1_cow appears to be part of a raid array: level=raid5 devices=4 ctime=Sun Mar 2 22:52:53 2008 mdadm: /dev/mapper/sdc1_cow appears to be part of a raid array: level=raid5 devices=4 ctime=Sun Mar 2 22:52:53 2008 mdadm: /dev/mapper/sde1_cow appears to be part of a raid array: level=raid5 devices=4 ctime=Sun Mar 2 22:52:53 2008 mdadm: size set to 732571904K Continue creating array? Y mdadm: array /dev/md0 started. # mdadm --detail /dev/md0 /dev/md0: Version : 00.90 Creation Time : Mon Aug 16 18:20:06 2010 Raid Level : raid5 Array Size : 2197715712 (2095.91 GiB 2250.46 GB) Used Dev Size : 732571904 (698.64 GiB 750.15 GB) Raid Devices : 4 Total Devices : 3 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Mon Aug 16 18:20:06 2010 State : clean, degraded Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K UUID : 916ceaa2:b877a3cc:3973abef:31f2d600 (local to host monstre) Events : 0.1 Number Major Minor RaidDevice State 0 251 9 0 active sync /dev/block/251:9 1 251 10 1 active sync /dev/block/251:10 2 0 0 2 removed 3 251 12 3 active sync /dev/block/251:12 And I can now mount /dev/mapper/raid-home ! The question now is, what next? Should I start copying things off to a backup, or run fsck first or something else to try to repair errors? Or perhaps are the 2GB sparse files to small for anything like that?
For me: first, copy everything. You have an unreliable disk in the middle of your data. N. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html