raid 5, drives marked as failed. Can I recover?

5 messages, 3 authors, 2009-01-30 · open the first message on its own page

raid 5, drives marked as failed. Can I recover?

From: Tom <hidden>
Date: 2009-01-29 23:17:14

Hello,

2 drives have failed on my raid5 setup and I need to recover the data
on the raid.
I am sure that the drives still works or at least one of them still works.

How do I recover my drives?

I can't mount the raid no more and I am missing a hard drive when i
run ls /dev/sd?
I have 7 drives on my raid.

Here is output of /var/log/messages in following link

http://matx.pastebin.com/m35423452

also some more information

tom@desu ~ $ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4] [multipath]
md2 : inactive sdc1[1] sdd1[4] sdf1[3] sde1[2]
      1953214208 blocks


tom@desu ~ $ sudo mdadm --detail /dev/md2
Password:
/dev/md2:
        Version : 00.90.03
  Creation Time : Thu Sep  4 20:14:31 2008
     Raid Level : raid5
  Used Dev Size : 488303552 (465.68 GiB 500.02 GB)
   Raid Devices : 7
  Total Devices : 4
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Thu Jan 29 21:16:34 2009
          State : active, degraded, Not Started
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : cf3d1948:1d0e65b6:c028c7c8:
56f0c54c
         Events : 0.1411738

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       33        1      active sync   /dev/sdc1
       2       8       65        2      active sync   /dev/sde1
       3       8       81        3      active sync   /dev/sdf1
       4       8       49        4      active sync   /dev/sdd1
       5       0        0        5      removed
       6       0        0        6      removed




Thank you for your time in advance.

Re: raid 5, drives marked as failed. Can I recover?

From: Justin Piszcz <hidden>
Date: 2009-01-30 14:18:03

Try to assmeble the array with --force.

On Thu, 29 Jan 2009, Tom wrote:
Hello,

2 drives have failed on my raid5 setup and I need to recover the data
on the raid.
I am sure that the drives still works or at least one of them still works.

How do I recover my drives?

I can't mount the raid no more and I am missing a hard drive when i
run ls /dev/sd?
I have 7 drives on my raid.

Here is output of /var/log/messages in following link

http://matx.pastebin.com/m35423452

also some more information

tom@desu ~ $ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4] [multipath]
md2 : inactive sdc1[1] sdd1[4] sdf1[3] sde1[2]
     1953214208 blocks


tom@desu ~ $ sudo mdadm --detail /dev/md2
Password:
/dev/md2:
       Version : 00.90.03
 Creation Time : Thu Sep  4 20:14:31 2008
    Raid Level : raid5
 Used Dev Size : 488303552 (465.68 GiB 500.02 GB)
  Raid Devices : 7
 Total Devices : 4
Preferred Minor : 2
   Persistence : Superblock is persistent

   Update Time : Thu Jan 29 21:16:34 2009
         State : active, degraded, Not Started
Active Devices : 4
Working Devices : 4
Failed Devices : 0
 Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 64K

          UUID : cf3d1948:1d0e65b6:c028c7c8:
56f0c54c
        Events : 0.1411738

   Number   Major   Minor   RaidDevice State
      0       0        0        0      removed
      1       8       33        1      active sync   /dev/sdc1
      2       8       65        2      active sync   /dev/sde1
      3       8       81        3      active sync   /dev/sdf1
      4       8       49        4      active sync   /dev/sdd1
      5       0        0        5      removed
      6       0        0        6      removed




Thank you for your time in advance.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid 5, drives marked as failed. Can I recover?

From: David Greaves <hidden>
Date: 2009-01-30 14:56:26

Justin Piszcz wrote:
Try to assmeble the array with --force.
hmmmm? not yet...

On Thu, 29 Jan 2009, Tom wrote:
quoted
Hello,

2 drives have failed on my raid5 setup and I need to recover the data
on the raid.
I am sure that the drives still works or at least one of them still
works.

How do I recover my drives?
How important is it?
The more important the data the more you should reduce the risk of a subsequent
failure.
If you "don't care" then we just force it back together and cross fingers.
Otherwise we run tests on all the drives before trying a restore.
I'd say to run these tests on each drive; as a minimum do the first test on the
failed drive, more paranoia, more tests and include the non-failed drives (to
ensure they don't fail during recovery):
* smartctl -t short
* smartctl -t long
* badblocks

What happened? Smoke?
Are the drives faulty (what does smartctl -a tell you)
Did the cables just wiggle? Is the controller broken?
You probably don't know :)

I would obtain replacements for the failed drives and use ddrescue to copy from
the failed drive to a replacement.
Then install the good drives and begin recovery.
quoted
I can't mount the raid no more and I am missing a hard drive when i
run ls /dev/sd?
I have 7 drives on my raid.
You say you have 7 drives and 2 are failed.
And yet I see 4 drives, not 5.

Where is sdg?
quoted
Here is output of /var/log/messages in following link

http://matx.pastebin.com/m35423452
Jan 29 21:14:11  sda died
Jan 29 21:14:12  sdb died
quoted
also some more information
Also need:
 Distro
 Kernel version
 Mdadm version
 mdadm --examine for each available component.

David

-- 
"Don't worry, you'll be fine; I saw it work in a cartoon once..."

Re: raid 5, drives marked as failed. Can I recover?

From: Tom <hidden>
Date: 2009-01-30 15:06:33

Hello,

I spent a night trying out mdadm --assemble on a virtual machine to
see how it attempts to fix a raid where 2 or more drives have been
marked faulty.
I was quite sure that the drives were fine and that they were wrongly
marked as bad.
I think I just have a bad ata controller.

I used --assemble on real machine and it seemed to have detected the raid again.
1 drive was found to be bad and it is recreating it now.
But my data is there and I can open it.
I am going to get some dvd's and back all this up before it dies again!


Regards and thanks for your help!


2009/1/30 David Greaves [off-list ref]:
Justin Piszcz wrote:
quoted
Try to assmeble the array with --force.
hmmmm? not yet...

quoted
On Thu, 29 Jan 2009, Tom wrote:
quoted
Hello,

2 drives have failed on my raid5 setup and I need to recover the data
on the raid.
I am sure that the drives still works or at least one of them still
works.

How do I recover my drives?
How important is it?
The more important the data the more you should reduce the risk of a subsequent
failure.
If you "don't care" then we just force it back together and cross fingers.
Otherwise we run tests on all the drives before trying a restore.
I'd say to run these tests on each drive; as a minimum do the first test on the
failed drive, more paranoia, more tests and include the non-failed drives (to
ensure they don't fail during recovery):
* smartctl -t short
* smartctl -t long
* badblocks

What happened? Smoke?
Are the drives faulty (what does smartctl -a tell you)
Did the cables just wiggle? Is the controller broken?
You probably don't know :)

I would obtain replacements for the failed drives and use ddrescue to copy from
the failed drive to a replacement.
Then install the good drives and begin recovery.
quoted
quoted
I can't mount the raid no more and I am missing a hard drive when i
run ls /dev/sd?
I have 7 drives on my raid.
You say you have 7 drives and 2 are failed.
And yet I see 4 drives, not 5.

Where is sdg?
quoted
quoted
Here is output of /var/log/messages in following link

http://matx.pastebin.com/m35423452
Jan 29 21:14:11  sda died
Jan 29 21:14:12  sdb died
quoted
quoted
also some more information
Also need:
 Distro
 Kernel version
 Mdadm version
 mdadm --examine for each available component.

David

--
"Don't worry, you'll be fine; I saw it work in a cartoon once..."

Re: raid 5, drives marked as failed. Can I recover?

From: David Greaves <hidden>
Date: 2009-01-30 15:24:20

Tom wrote:
Hello,

I spent a night trying out mdadm --assemble on a virtual machine to
see how it attempts to fix a raid where 2 or more drives have been
marked faulty.
I was quite sure that the drives were fine and that they were wrongly
marked as bad.
I think I just have a bad ata controller.
Given 2 drives died in 1 second then I'd agree.
I used --assemble on real machine and it seemed to have detected the raid again.
1 drive was found to be bad and it is recreating it now.
But my data is there and I can open it.
I am going to get some dvd's and back all this up before it dies again!
OK, that's good :)

A forced assemble will make md assume all the disks are good and that all writes
succeeded. ie all is well.

They probably didn't and it probably isn't. OTOH you probably lost a few hundred
bytes in many many Gb so nothing to panic over.

You should fsck and, ideally, checksum compare your filesystem against a backup.
I would run a read-only fsck before doing anything. Then if you just have light
damage, repair.

David
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help