Re: RAID5 faild while in degraded mode, need help

On Sun, 8 Jul 2012 21:05:02 +0200 Dietrich Heise [off-list ref] wrote:

Hi,

the following Problem,
One of four drives has S.M.A.R.T. errors, so I removed it and
replaced, with a new one.

In the time the drive was rebuilding, one of the three left devices
has an I/O error (sdd1) (sdc1 was the replaced drive an was syncing).

Now the following happends (two drives are spare drives :( )

It looks like you tried to --add /dev/sdd1 back in after it failed, and mdadm
let new.  Newer versions of mdadm will refuse as that is not a good thing to
do but it shouldn't stop you getting your data back.

First thing to realise is that you could have data corruption.  There is at
least one block in the array which cannot be recovered, possibly more.  i.e.
any block on sdd1 which is bad, and any block at the same offset in sdc1.
These blocks may not be in files which would be lucky, or they may contain
important metadata which might mean you've lost lots of files.

If you hadn't tried to --add /dev/sdd1 you could just force-assemble the
array back to degraded mode (without sdc1) and back up any critical data.
As sdd1 now thinks it is a spare you need to re-create the array instead:

 mdadm -S /dev/md1
 mdadm -C /dev/md1 -l5 -n4 -e 1.2 -c 512 /dev/sdf1 /dev/sde1 /dev/sdd1 missing
or
 mdadm -C /dev/md1 -l5 -n4 -e 1.2 -c 512 /dev/sdf1 /dev/sde1 missing /dev/sdd1

depending on whether sdd1 as the 3rd or 4th device in the array - I cannot
tell from the output here.

You should then be able to mount the array and backup stuff.

You then want to use 'ddrescue' to copy sdd1 onto a device with no bad
blocks, and assemble  the array using the device instead of sdd1.

Finally, you can add the new spare (sdc1) to the array and it should rebuild
successfully - providing there are no bad blocks on sdf1 or sde1.

I hope that makes sense.  Do ask if anything is unclear.

NeilBrown

p3 disks # mdadm -D /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Mon Feb 28 19:57:56 2011
     Raid Level : raid5
  Used Dev Size : 1465126400 (1397.25 GiB 1500.29 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Jul  8 20:37:12 2012
          State : active, FAILED, Not Started
 Active Devices : 2
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 512K

           Name : p3:0  (local to host p3)
           UUID : 6d4ebfd4:491bcb50:d98d5e67:f226f362
         Events : 121205

    Number   Major   Minor   RaidDevice State
       0       8       81        0      active sync   /dev/sdf1
       1       8       65        1      active sync   /dev/sde1
       2       0        0        2      removed
       3       0        0        3      removed

       4       8       49        -      spare   /dev/sdd1
       5       8       33        -      spare   /dev/sdc1

here is more information:

p3 disks # mdadm -E /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6d4ebfd4:491bcb50:d98d5e67:f226f362
           Name : p3:0  (local to host p3)
  Creation Time : Mon Feb 28 19:57:56 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930275057 (1397.26 GiB 1500.30 GB)
     Array Size : 8790758400 (4191.76 GiB 4500.87 GB)
  Used Dev Size : 2930252800 (1397.25 GiB 1500.29 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : caefb029:526187ef:2051f578:db2b82b7

    Update Time : Sun Jul  8 20:37:12 2012
       Checksum : 18e2bfe1 - correct
         Events : 121205

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AA.. ('A' == active, '.' == missing)
p3 disks # mdadm -E /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6d4ebfd4:491bcb50:d98d5e67:f226f362
           Name : p3:0  (local to host p3)
  Creation Time : Mon Feb 28 19:57:56 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
     Array Size : 8790758400 (4191.76 GiB 4500.87 GB)
  Used Dev Size : 2930252800 (1397.25 GiB 1500.29 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 4231e244:60e27ed4:eff405d0:2e615493

    Update Time : Sun Jul  8 20:37:12 2012
       Checksum : 4bec6e25 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AA.. ('A' == active, '.' == missing)
p3 disks # mdadm -E /dev/sde1
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6d4ebfd4:491bcb50:d98d5e67:f226f362
           Name : p3:0  (local to host p3)
  Creation Time : Mon Feb 28 19:57:56 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930253889 (1397.25 GiB 1500.29 GB)
     Array Size : 8790758400 (4191.76 GiB 4500.87 GB)
  Used Dev Size : 2930252800 (1397.25 GiB 1500.29 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 28b08f44:4cc24663:84d39337:94c35d67

    Update Time : Sun Jul  8 20:37:12 2012
       Checksum : 15faa8a1 - correct
         Events : 121205

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.. ('A' == active, '.' == missing)
p3 disks # mdadm -E /dev/sdf1
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6d4ebfd4:491bcb50:d98d5e67:f226f362
           Name : p3:0  (local to host p3)
  Creation Time : Mon Feb 28 19:57:56 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
     Array Size : 8790758400 (4191.76 GiB 4500.87 GB)
  Used Dev Size : 2930252800 (1397.25 GiB 1500.29 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 78d5600a:91927758:f78a1cea:3bfa3f5b

    Update Time : Sun Jul  8 20:37:12 2012
       Checksum : 7767cb10 - correct
         Events : 121205

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.. ('A' == active, '.' == missing)

Is there a way to repair the raid?

thanks!
Dietrich
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html