Re: Please Help! RAID5 -> 6 reshapre gone bad

On Tue, 7 Feb 2012 14:50:57 +1100 Richard Herd [off-list ref] wrote:

Hi Neil,

OK, git head is: mdadm-3.2.3-21-gda8fe5a

I have 8 disks.  They get muddled about each boot (an issue I have
never addressed).   Ignore sde (esata HD) and sdh (usb boot).

It seems even with --force, dmesg always reports 'kicking non-fresh
sdc/g1 from array!'.  Leaving sdg out as suggested by Phil doesn't
help unfortunately.

root@raven:/neil/mdadm# ./mdadm -Avvv --force
--backup-file=/usb/md0.backup /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1
/dev/sdd1 /dev/sdf1 /dev/sdg1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 0.
mdadm:/dev/md0 has an active reshape - checking if critical section
needs to be restored
mdadm: accepting backup with timestamp 1328559119 for array with
timestamp 1328567549
mdadm: restoring critical section
mdadm: added /dev/sdg1 to /dev/md0 as 0
mdadm: added /dev/sda1 to /dev/md0 as 2
mdadm: added /dev/sdc1 to /dev/md0 as 3
mdadm: added /dev/sdf1 to /dev/md0 as 4
mdadm: added /dev/sdd1 to /dev/md0 as 5
mdadm: added /dev/sdb1 to /dev/md0 as 1
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error

Hmmm.... maybe your kernel isn't quite doing the right thing.
 commit 674806d62fb02a22eea948c9f1b5e58e0947b728 is important.
It is in 2.6.35.  What kernel are you running?  
Definitely something older given the "1: w=1 pa=18...." messages.  They
disappear in 2.6.34.

So I'm afraid you're going to need a new kernel.

NeilBrown

and dmesg:
[13964.591801] md: bind<sdg1>
[13964.595371] md: bind<sda1>
[13964.595668] md: bind<sdc1>
[13964.595900] md: bind<sdf1>
[13964.599084] md: bind<sdd1>
[13964.599652] md: bind<sdb1>
[13964.600478] md: kicking non-fresh sdc1 from array!
[13964.600493] md: unbind<sdc1>
[13964.612138] md: export_rdev(sdc1)
[13964.612163] md: kicking non-fresh sdg1 from array!
[13964.612183] md: unbind<sdg1>
[13964.624077] md: export_rdev(sdg1)
[13964.628203] raid5: reshape will continue
[13964.628243] raid5: device sdb1 operational as raid disk 1
[13964.628252] raid5: device sdf1 operational as raid disk 4
[13964.628260] raid5: device sda1 operational as raid disk 2
[13964.629614] raid5: allocated 6308kB for md0
[13964.629731] 1: w=1 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
[13964.629742] 5: w=1 pa=18 pr=6 m=2 a=2 r=6 op1=1 op2=0
[13964.629751] 4: w=2 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
[13964.629760] 2: w=3 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
[13964.629767] raid5: not enough operational devices for md0 (3/6 failed)
[13964.640403] RAID5 conf printout:
[13964.640409]  --- rd:6 wd:3
[13964.640416]  disk 1, o:1, dev:sdb1
[13964.640423]  disk 2, o:1, dev:sda1
[13964.640429]  disk 4, o:1, dev:sdf1
[13964.640436]  disk 5, o:1, dev:sdd1
[13964.641621] raid5: failed to run raid set md0
[13964.649886] md: pers->run() failed ...