RAID5 problem
From: Alfons Andorfer <hidden>
Date: 2005-12-04 14:21:48
Hi,
I have a RAID5 array consisting of 4 disks:
/dev/hda3
/dev/hdc3
/dev/hde3
/dev/hdg3
and the Linux machine that this system was running on crashed yesterday due to a faulty Kernel driver (i.e. the machine just halted).
So I resetted it, but it didn't come up again.
I started the machine with a Knoppix CD and found out that the array had been running in degraded mode for about two months (/dev/hda3 went off then).
When I do a
mdadm --assemble /dev/md0 --force /dev/hd[ceg]3
I get
mdadm: forcing event count in /dev/hdc3(1) from 515 upto 516
mdadm: /dev/md0 has been started with 3 drives (out of 4).
I can mount the array with
mount /dev/md0 /mount/
and the data seems to be OK.
But after a
umount /dev/md0
and a
fsck -n /dev/md0
it stops with an error
"pass 1: checking Inodes, Blocks, and sizes
read error - Block 131460 (Attempt to read block from filesystem resulted in short read) during Inode-Scan Ignore error?"
and if I do the fsck with
e2fsck -y /dev/md0
I get tons of read errors of the type "(Attempt to read block from filesystem resulted in short read)" and the event counter of the /dev/hdc3 is then just one _behind_ of the event counters of /dev/hde3 and /dev/hdg3 which is really strange to me!?!
Then I tried
mdadm -S /dev/md0
mdadm --create /dev/md0 -c32 -l5 -n4 missing /dev/hdc3 /dev/hde3 /dev/hdg3
which resultet in
mdadm: /dev/hdc3 appears to be part of a raid array:
level=5 devices=4 ctime=Fri May 30 14:25:47 2003
mdadm: /dev/hde3 appears to be part of a raid array:
level=5 devices=4 ctime=Fri May 30 14:25:47 2003
mdadm: /dev/hdg3 appears to contain an ext2fs file system
size=493736704K mtime=Tue Jan 3 04:48:21 2006
mdadm: /dev/hdg3 appears to be part of a raid array:
level=5 devices=4 ctime=Fri May 30 14:25:47 2003
Continue creating array? no
mdadm: create aborted.
I aborted the above because it look strange to me that /dev/hdg3 appears two times and /dev/hda3 doesn't at all!?!
So this is where I got stuck, any help appreciated!
Here are the outputs of
cat /mount/etc/raidtab
and
mdadm --examine /dev/hd[aceg]3
----------------------------------------------------------------------
cat /mount/etc/raidtab:
-----------------------
raiddev /dev/md0
raid-level 5
nr-raid-disks 4
nr-spare-disks 0
persistent-superblock 1
parity-algorithm left-symmetric
chunk-size 32
device /dev/hda3
raid-disk 0
device /dev/hdc3
raid-disk 1
device /dev/hde3
raid-disk 2
device /dev/hdg3
raid-disk 3
----------------------------------------------------------------------
mdadm --examine /dev/hd[aceg]3:
-------------------------------
/dev/hda3:
Magic : a92b4efc
Version : 00.90.00
UUID : 02d9c6f2:53c8584d:8815ae94:e4af8e1c
Creation Time : Fri May 30 14:25:47 2003
Raid Level : raid5
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sat Dec 3 18:56:59 2005
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : f620ca21 - correct
Events : 0.390
Layout : left-symmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 0 3 3 0 active sync
0 0 3 3 0 active sync
1 1 0 0 1 faulty removed
2 2 33 3 2 active sync
3 3 34 3 3 active sync
/dev/hdc3:
Magic : a92b4efc
Version : 00.90.00
UUID : 02d9c6f2:53c8584d:8815ae94:e4af8e1c
Creation Time : Fri May 30 14:25:47 2003
Raid Level : raid5
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sun Dec 4 15:03:42 2005
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : f621e626 - correct
Events : 0.524
Layout : left-symmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 1 22 3 1 active sync
0 0 0 0 0 removed
1 1 22 3 1 active sync
2 2 33 3 2 active sync
3 3 34 3 3 active sync
/dev/hde3:
Magic : a92b4efc
Version : 00.90.00
UUID : 02d9c6f2:53c8584d:8815ae94:e4af8e1c
Creation Time : Fri May 30 14:25:47 2003
Raid Level : raid5
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sun Dec 4 15:03:42 2005
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : f621e633 - correct
Events : 0.524
Layout : left-symmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 2 33 3 2 active sync
0 0 0 0 0 removed
1 1 22 3 1 active sync
2 2 33 3 2 active sync
3 3 34 3 3 active sync
/dev/hdg3:
Magic : a92b4efc
Version : 00.90.00
UUID : 02d9c6f2:53c8584d:8815ae94:e4af8e1c
Creation Time : Fri May 30 14:25:47 2003
Raid Level : raid5
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sun Dec 4 15:03:42 2005
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : f621e636 - correct
Events : 0.524
Layout : left-symmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 3 34 3 3 active sync
0 0 0 0 0 removed
1 1 22 3 1 active sync
2 2 33 3 2 active sync
3 3 34 3 3 active sync