Hi,
I have some trouble with my md raid-5 array. It has four 500GB drives
(sdb1-sde1). sde started giving some SMART error (Pending Bad
Sectors). So just to be safe, I decided to replace it. I declared it
to be faulty and removed it from the array and added a new drive and
the rebuilding was automatic. But before the rebuilding can finish, I
got an I/O error from sdb1 and it was declared faulty by md.
Now I have two faulty drives and things don't look good. However, I
was able to add the sdb back to the array and md seemed not mind and
still reported "active sync". At this point I shut down computer and
decided to clone sdb with clonezilla so that I can have a good sdb to
finish rebuilding sde. Not sure if it will complete without I/O
errors. It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
The only reason I'm not in a total panic mode is that I did a back up
before doing all this. Now I'm keeping my finger crossed that my
backup drive won't die. In retrospect, I should have just shut down
the computer and cloned sde instead of letting md to rebuild the
array.
Any suggestions on the best way to proceed at this point are highly
appreciated, especially on the scenario that I won't be able to clone
sdb. Is there any way to avoid building a new array?
Regards,
DS
PS, if in the end I have to build a new array, I'll probably go with a
raid 6 instead.
PS, if in the end I have to build a new array, I'll probably go with a
raid 6 instead.
Agreed, someone recently posted that for a raid-5 composed of 1TB
drives the odds of a rebuild failure are 1 in 67 even if the remaining
drives are within spec. (ie. the unrecoverable bit error rate is
slowing succumbing to the ever increasing size of drives.)
You have 500GB drives, but you have 3 left to rebuild from, so that's
1.5 TB your trying to read. I'm not sure how the original calculation
was done, so your odds of failed rebuild were either 1 in 134 or about
1 in 42. Either not very good for something that is supposed to
protect your data.
Greg
--
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
Preservation and Forensic processing of Exchange Repositories White Paper -
<http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html>
The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
On Mon, Feb 8, 2010 at 4:04 PM, Greg Freemyer [off-list ref] wrote:
quoted
PS, if in the end I have to build a new array, I'll probably go with a
raid 6 instead.
Agreed, someone recently posted that for a raid-5 composed of 1TB
drives the odds of a rebuild failure are 1 in 67 even if the remaining
drives are within spec. (ie. the unrecoverable bit error rate is
slowing succumbing to the ever increasing size of drives.)
Has anyone done similar calculation for raid-6?
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky [off-list ref] wrote:
It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
BTW, is it normal for dd to be this slow, or is there something wrong
with my setup? These are quite modern drives and should be able to
handle sequential transfer of 10's of MB/s. I have never used dd
before so I don't have any experience.
DS
On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky [off-list ref] wrote:
Hi,
Now I have two faulty drives and things don't look good. However, I
was able to add the sdb back to the array and md seemed not mind and
still reported "active sync". At this point I shut down computer and
decided to clone sdb with clonezilla so that I can have a good sdb to
finish rebuilding sde. Not sure if it will complete without I/O
errors. It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
As expected, dd encountered the same UNC error. Now I'm trying to
ddrescue the drive to see what happens. My question is whether this
is worth doing. Assuming ddrescue cannot read the bad sector either
and writes 0's to the new drive, will I be able to rebuild the raid-5,
from 2 good disks and this disk with a bad sector? I can assume there
will be a bad file but will the array still function?
Or I'm better off just build a new array from scratch.
Any suggestions are appreciated.
Regards,
DS
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Feb 8, 2010 at 7:23 PM, Dawning Sky [off-list ref] wrote:
On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky [off-list ref] wrote:
quoted
It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
BTW, is it normal for dd to be this slow, or is there something wrong
with my setup? These are quite modern drives and should be able to
handle sequential transfer of 10's of MB/s. I have never used dd
before so I don't have any experience.
dd should be pretty close to the max possible throughput of yer drive.
For example, I dd'd a 1T WD black not too long ago & it was around
120MB/s on the outside & 70MB/s in the middle.
Wil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Steven Haigh <hidden> Date: 2010-02-09 04:26:32
On Mon, 8 Feb 2010 22:24:09 -0600, Wil Reichert [off-list ref]
wrote:
On Mon, Feb 8, 2010 at 7:23 PM, Dawning Sky [off-list ref]
wrote:
quoted
On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky [off-list ref]
wrote:
quoted
It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
BTW, is it normal for dd to be this slow, or is there something wrong
with my setup? These are quite modern drives and should be able to
handle sequential transfer of 10's of MB/s. I have never used dd
before so I don't have any experience.
dd should be pretty close to the max possible throughput of yer drive.
For example, I dd'd a 1T WD black not too long ago & it was around
120MB/s on the outside & 70MB/s in the middle.
I think this depends more on the block size that is used. 1MB block sizes
are much faster than 512 byte blocks...
--
Steven Haigh
Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Feb 8, 2010 at 10:26 PM, Steven Haigh [off-list ref] wrote:
On Mon, 8 Feb 2010 22:24:09 -0600, Wil Reichert [off-list ref]
wrote:
quoted
On Mon, Feb 8, 2010 at 7:23 PM, Dawning Sky [off-list ref]
wrote:
quoted
On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky [off-list ref]
wrote:
quoted
It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
BTW, is it normal for dd to be this slow, or is there something wrong
with my setup? These are quite modern drives and should be able to
handle sequential transfer of 10's of MB/s. I have never used dd
before so I don't have any experience.
dd should be pretty close to the max possible throughput of yer drive.
For example, I dd'd a 1T WD black not too long ago & it was around
120MB/s on the outside & 70MB/s in the middle.
I think this depends more on the block size that is used. 1MB block sizes
are much faster than 512 byte blocks...
Good point, I was using bs=1M.
Wil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Stefan Hübner <hidden> Date: 2010-02-09 06:57:09
Hi!
I do RAID-recoveries at least once a month and get paid for it. Rule of
thumb: if your have one drive dropped and another one with pending
sectors, your rebuild will fail - no need for calculations there.
ddrescue on a clean disk is about half as fast as dd with a blocksize
beyond 1M. ddrescue on a disk with pending sectors is just no pita as
dd or sg_dd would be, because it adds the neccesary intelligence.
Do you have the original sde still around? If yes, ddrescue both: sdb
and sde. My experience says: there will only be a few KB lost. Then
re-create your raid (it will only write the superblock new) with
"--assume-clean". After that worked, you might make another (big)
backup first, then run fsck and see what happens. If the lost bytes
have screwed the filesystem, you might want to re-create the raid with
another (personally I prefer xfs) fs and replay your backup into it.
A few commands to make the intentions cleaner:
ddrescue -dv -r5 /dev/oldsdb1 /dev/newsdb1 /root/sdblog
ddrescue -dv -r5 /dev/oldsde1 /dev/newsde1 /root/sdelog
... find out which drive is which raid-device -> mdadm -E /dev/sdX1
mdadm --create /dev/md0 --raid-devices=4 --level=5
--chunk=${your_chunk_size_in_kb} --assume-clean
${ordered_list_of_raid_devices}
Hope this helps,
Stefan Hübner
Am 09.02.2010 05:20, schrieb Dawning Sky:
On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky [off-list ref] wrote:
quoted
Hi,
Now I have two faulty drives and things don't look good. However, I
was able to add the sdb back to the array and md seemed not mind and
still reported "active sync". At this point I shut down computer and
decided to clone sdb with clonezilla so that I can have a good sdb to
finish rebuilding sde. Not sure if it will complete without I/O
errors. It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
As expected, dd encountered the same UNC error. Now I'm trying to
ddrescue the drive to see what happens. My question is whether this
is worth doing. Assuming ddrescue cannot read the bad sector either
and writes 0's to the new drive, will I be able to rebuild the raid-5,
from 2 good disks and this disk with a bad sector? I can assume there
will be a bad file but will the array still function?
Or I'm better off just build a new array from scratch.
Any suggestions are appreciated.
Regards,
DS
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Thanks for the good advice. ddrescue on sdb returned an error of 4kB.
I do still have my old sde. But one thing I did, which was stupid,
was trying to rebuild the raid-5 when it was mounted. So I don't know
the old sde is still consistent with the rest 3 disks, since some
files would have been modified between the times when I took the old
sde offline and when the rebuild failed.
So at this point, I guess I'll get 4 new drives and set up a brand new
raid-6 and try to restore my data from my backup in an external drive
and hope for the best. I'll keep the 4 drives from my old raid-5 just
in case if I need to recover something from them.
I guess I learned my lesson. I should have ddrescued all the disks I
want to replace, instead of using md's rebuild mechanism.
DS
On Mon, Feb 8, 2010 at 10:57 PM, Stefan Hübner
[off-list ref] wrote:
Hi!
I do RAID-recoveries at least once a month and get paid for it. Rule of
thumb: if your have one drive dropped and another one with pending
sectors, your rebuild will fail - no need for calculations there.
ddrescue on a clean disk is about half as fast as dd with a blocksize
beyond 1M. ddrescue on a disk with pending sectors is just no pita as
dd or sg_dd would be, because it adds the neccesary intelligence.
Do you have the original sde still around? If yes, ddrescue both: sdb
and sde. My experience says: there will only be a few KB lost. Then
re-create your raid (it will only write the superblock new) with
"--assume-clean". After that worked, you might make another (big)
backup first, then run fsck and see what happens. If the lost bytes
have screwed the filesystem, you might want to re-create the raid with
another (personally I prefer xfs) fs and replay your backup into it.
A few commands to make the intentions cleaner:
ddrescue -dv -r5 /dev/oldsdb1 /dev/newsdb1 /root/sdblog
ddrescue -dv -r5 /dev/oldsde1 /dev/newsde1 /root/sdelog
... find out which drive is which raid-device -> mdadm -E /dev/sdX1
mdadm --create /dev/md0 --raid-devices=4 --level=5
--chunk=${your_chunk_size_in_kb} --assume-clean
${ordered_list_of_raid_devices}
Hope this helps,
Stefan Hübner
Am 09.02.2010 05:20, schrieb Dawning Sky:
quoted
On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky [off-list ref] wrote:
quoted
Hi,
Now I have two faulty drives and things don't look good. However, I
was able to add the sdb back to the array and md seemed not mind and
still reported "active sync". At this point I shut down computer and
decided to clone sdb with clonezilla so that I can have a good sdb to
finish rebuilding sde. Not sure if it will complete without I/O
errors. It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
As expected, dd encountered the same UNC error. Now I'm trying to
ddrescue the drive to see what happens. My question is whether this
is worth doing. Assuming ddrescue cannot read the bad sector either
and writes 0's to the new drive, will I be able to rebuild the raid-5,
from 2 good disks and this disk with a bad sector? I can assume there
will be a bad file but will the array still function?
Or I'm better off just build a new array from scratch.
Any suggestions are appreciated.
Regards,
DS
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Mikael Abrahamsson <hidden> Date: 2010-02-09 08:05:31
On Mon, 8 Feb 2010, Dawning Sky wrote:
Thanks for the good advice. ddrescue on sdb returned an error of 4kB.
I do still have my old sde. But one thing I did, which was stupid,
was trying to rebuild the raid-5 when it was mounted. So I don't know
the old sde is still consistent with the rest 3 disks, since some
files would have been modified between the times when I took the old
sde offline and when the rebuild failed.
Please see the threads I have initiated the past 2-3 weeks, I had the
exact same problem.
What you should do is just put in your new cloned drive, make sure it runs
fine with 3 drives, and then start the rebuild with the drive that you
were resyncing to. I've done this numerous times, and my experience says
you will only have lost those 4 kilobytes of data that ddrescue couldn't
read.
I'm now happily at RAID6 after doing a re-shape from 6 drive raid5-> 8
drive raid6, change of SATA cables and a new marvell 8channel
controller... No more read timeouts anyway, and I should be better
protected from single read errors like you are having.
--
Mikael Abrahamsson email: swmike@swm.pp.se
From: Jon Hardcastle <hidden> Date: 2010-02-09 10:22:37
--- On Tue, 9/2/10, Dawning Sky <the.dawning.sky@gmail.com> wrote:
From: Dawning Sky <redacted>
Subject: Re: Disk I/O error while rebuilding an md raid-5 array
To: "Greg Freemyer" <redacted>
Cc: linux-raid@vger.kernel.org
Date: Tuesday, 9 February, 2010, 0:45
On Mon, Feb 8, 2010 at 4:04 PM, Greg
Freemyer [off-list ref]
wrote:
quoted
quoted
PS, if in the end I have to build a new array,
I'll probably go with a
quoted
quoted
raid 6 instead.
Agreed, someone recently posted that for a raid-5
composed of 1TB
quoted
drives the odds of a rebuild failure are 1 in 67 even
if the remaining
quoted
drives are within spec. (ie. the unrecoverable bit
error rate is
quoted
slowing succumbing to the ever increasing size of
drives.)
Has anyone done similar calculation for raid-6?
quoted
Greg
I am not sure of the math, but i was under the impression the odds were calculated PER DRIVE, not combining the array size. I HAD a 6 drive array comprised of 4x500GB and 2x750 (with some wasted space) as I wanted to push the new drives into 1~+ TB drives to make use of this lost space I decided to go RAID6 so i now have a 7 drive array.. 4x500GB 2x750 1x1TB. Now i am happy it is ok (see lots of earlier posts about mismatches) I am gearing up to replace 2 of the 500GB's with 1TB drives, and in 6 months I will repeat.
Bottom line, my 'gut' says if your drives are going to stray over 1TB go to RAID6 also, I have staggered DAILY smart reads (i think each drive gets 1 day a week 'off' and the rest are either long or short but never all doing 1 type) i also do a weekly 'scrub' of my array (this is so amazingly important).
My only downside is due to FSB speed issues i dont get more than 30m/s when accessing all 7 drives simultaneously but that is enough for RAID.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: John Robinson <hidden> Date: 2010-02-09 11:57:06
On 09/02/2010 00:04, Greg Freemyer wrote:
quoted
PS, if in the end I have to build a new array, I'll probably go with a
raid 6 instead.
Agreed, someone recently posted that for a raid-5 composed of 1TB
drives the odds of a rebuild failure are 1 in 67 even if the remaining
drives are within spec. (ie. the unrecoverable bit error rate is
slowing succumbing to the ever increasing size of drives.)
Actually the odds were 1 in 67 of an unrecoverable read error while
reading 2TB of data, if the odds were 1 in 10^15 per bit read[1], which
was the worst-case spec offered by Western Digital. Others disagreed
with my analysis, and I may be wrong.
This was nothing to do with RAID, but my suggestion followed on that
RAID-5 was now only useful for defending against unrecoverable errors,
and not dead drives, and if you wanted to defend against dead drives as
well you need RAID-6.
You have 500GB drives, but you have 3 left to rebuild from, so that's
1.5 TB your trying to read. I'm not sure how the original calculation
was done, so your odds of failed rebuild were either 1 in 134 or about
1 in 42. Either not very good for something that is supposed to
protect your data.
Actually there are 4 to read from - the original sde is still available.
This would be a situation where I think having the hot-rebuild facility
recently discussed on this list would be ideal, as if you can't read the
data from the drive you're hot-replacing, you then get a second chance
to read it from the rest of the drives using the parity information, and
the odds of an unrecoverable read error at the same LBA on two drives is
smaller - but I can't remember that bit of the probability course I did
years ago to work out exactly what it is.
Cheers,
John.
[1] If the probability of an error while reading 1 bit is p, then the
probability of an error while reading n bits is 1-(1-p)^n. In this case
p=1E-15, n=1.6E13 and you need a scientific calculator to do the sum.
Thanks to everyone for the good advices. Currently I plan to build a
5x500GB raid-6 from new drives and restore the data from the backup.
I'm keeping the original set of 4 offline just in case I need to
recover anything from it.
I still have a couple of questions I hope someone can help me with.
1. Can I still add the original sde back to the raid-5? While I was
doing rebuilding, I believe some files were modified by the OS and the
old sde would be out of sync.
2. Quite a few people talked about "scrubbing" the raid, can someone
point me to somewhere I can read more about it? I'm new to the list.
I tried googling and reading the list wiki and had a hard time finding
a tutorial for setting this up.
Regards,
DS
From: Mikael Abrahamsson <hidden> Date: 2010-02-09 19:28:45
On Tue, 9 Feb 2010, Dawning Sky wrote:
2. Quite a few people talked about "scrubbing" the raid, can someone
point me to somewhere I can read more about it? I'm new to the list. I
tried googling and reading the list wiki and had a hard time finding a
tutorial for setting this up.