From: Goswin von Brederlow <hidden> Date: 2009-08-26 11:18:41
Jon Hardcastle [off-list ref] writes:
Guys,
I have been having some problems with my arrays that I think i have nailed down to a pci controller (well I say that - it is always the drives connected to *a* controller but I have tried 2!) anyway the latest saga is i was trying some new kernel options last night - which didn't work.
But when i booted up again this morning it said one of the drives was in an inconsistent state (not sure of the *exact* error message). I then kicked off an add of the drive and it started syncing. It got about 5% in and then the second drive in on that controller complained and the array failed.
Is there any hope for my data? If i get a good controller in there will the resync continue? can I try and tell it to assume the drives are good (which they ought to be)?
Please help!
The inconsistency is probably just a block here or there and I'm
assuming none of your drives actualy failed. So 99.9999% of your data
should be there. Just rebooting might actualy just get your raid back
(to syncing). If not then you have to force reassembly from the drives
with the newest serials. That will give you some data corruption,
whatever was writing when the controler gave errors. Worst case you
have to recreate the raid with --assume-clean.
I recommend adding a bitmap to the raid. That way a wrongfully failed
drive can be resynced in a matter of minutes instead of hours or
days. Makes it way less likely another error occurs during resync.
MfG
Goswin
From: Jon Hardcastle <hidden> Date: 2009-08-26 11:29:39
--- On Wed, 26/8/09, Goswin von Brederlow <goswin-v-b@web.de> wrote:
From: Goswin von Brederlow <redacted>
Subject: Re: Raid 5 - not clean and then a failure.
To: Jon@eHardcastle.com
Cc: linux-raid@vger.kernel.org
Date: Wednesday, 26 August, 2009, 12:18 PM
Jon Hardcastle [off-list ref]
writes:
quoted
Guys,
I have been having some problems with my arrays that I
think i have nailed down to a pci controller (well I say
that - it is always the drives connected to *a* controller
but I have tried 2!) anyway the latest saga is i was trying
some new kernel options last night - which didn't work.
quoted
But when i booted up again this morning it said one of
the drives was in an inconsistent state (not sure of the
*exact* error message). I then kicked off an add of the
drive and it started syncing. It got about 5% in and then
the second drive in on that controller complained and the
array failed.
quoted
Is there any hope for my data? If i get a good
controller in there will the resync continue? can I try and
tell it to assume the drives are good (which they ought to
be)?
quoted
Please help!
The inconsistency is probably just a block here or there
and I'm
assuming none of your drives actualy failed. So 99.9999% of
your data
should be there. Just rebooting might actualy just get your
raid back
(to syncing). If not then you have to force reassembly from
the drives
with the newest serials. That will give you some data
corruption,
whatever was writing when the controler gave errors. Worst
case you
have to recreate the raid with --assume-clean.
I recommend adding a bitmap to the raid. That way a
wrongfully failed
drive can be resynced in a matter of minutes instead of
hours or
days. Makes it way less likely another error occurs during
resync.
MfG
Goswin
--
To unsubscribe from this list: send the line "unsubscribe
linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I did look into bitmaps *abit* i could easily have the imagine for my 6 drive raid 5 stored on the raid1 I have in the same system.. The googling I did tho did not paint a pretty picture it talked about huge performance hits?
-----------------------
N: Jon Hardcastle
E: Jon@eHardcastle.com
'Do not worry about tomorrow, for tomorrow will bring worries of its own.'
-----------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: John Robinson <hidden> Date: 2009-08-26 12:47:13
On 26/08/2009 12:29, Jon Hardcastle wrote:
[...]
I did look into bitmaps *abit* i could easily have the imagine for my 6 drive raid 5 stored on the raid1 I have in the same system.. The googling I did tho did not paint a pretty picture it talked about huge performance hits?
There is a performance hit but it can be minimised by picking a bitmap
chunk size to suit; I ended up getting about 80% of bitmap-less write
performance using a 16MB bitmap chunk size instead of about 40% with the
default size, on my 3-drive RAID-5 array.
Cheers,
John.
From: Ryan Wagoner <hidden> Date: 2009-08-26 14:14:31
Wouldn't weekly RAID consistency checks reveal a bad block before you
had a failure that required the need to do a full resync? It only
takes 3 hours to resync my 3 x 1TB drives and having a bitmap would
reduce the performance. I've never had to have a resync in the year
I've had the array up. I just wonder if the performance drawback is
worth having the bitmap to save a possible resync once every couple
years. Or are the RAID consistency checks not reliable enough to
prevent more errors during a resync?
Ryan
On Wed, Aug 26, 2009 at 7:18 AM, Goswin von Brederlow[off-list ref] wrote:
Jon Hardcastle [off-list ref] writes:
quoted
Guys,
I have been having some problems with my arrays that I think i have nailed down to a pci controller (well I say that - it is always the drives connected to *a* controller but I have tried 2!) anyway the latest saga is i was trying some new kernel options last night - which didn't work.
But when i booted up again this morning it said one of the drives was in an inconsistent state (not sure of the *exact* error message). I then kicked off an add of the drive and it started syncing. It got about 5% in and then the second drive in on that controller complained and the array failed.
Is there any hope for my data? If i get a good controller in there will the resync continue? can I try and tell it to assume the drives are good (which they ought to be)?
Please help!
The inconsistency is probably just a block here or there and I'm
assuming none of your drives actualy failed. So 99.9999% of your data
should be there. Just rebooting might actualy just get your raid back
(to syncing). If not then you have to force reassembly from the drives
with the newest serials. That will give you some data corruption,
whatever was writing when the controler gave errors. Worst case you
have to recreate the raid with --assume-clean.
I recommend adding a bitmap to the raid. That way a wrongfully failed
drive can be resynced in a matter of minutes instead of hours or
days. Makes it way less likely another error occurs during resync.
MfG
Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Jon Hardcastle <hidden> Date: 2009-08-26 14:19:51
Can a bitmap be easily removed? I might give it ago if it can.
I am never sure how thorough these checks are. Are they read/write, or just read? for example. I make of point of doing read/write badblocks checks with e2fck -cc when I do run them (not the automatic ones tho - dunno how) but that only checks that partition, which is on LVM, which is on RAID so WHO KNOWS what underlying drives are being checked.
I have before now, dismantled the array and run read/write badblocks directly on the constituent drives so at least smart is aware of them and although i aim to do this once every six months, I think I have actually done it only 1nce in the 2 year life of the array.
-----------------------
N: Jon Hardcastle
E: Jon@eHardcastle.com
'Do not worry about tomorrow, for tomorrow will bring worries of its own.'
-----------------------
--- On Wed, 26/8/09, Ryan Wagoner <rswagoner@gmail.com> wrote:
From: Ryan Wagoner <redacted>
Subject: Re: Raid 5 - not clean and then a failure.
To: "Goswin von Brederlow" <redacted>
Cc: Jon@ehardcastle.com, linux-raid@vger.kernel.org
Date: Wednesday, 26 August, 2009, 3:14 PM
Wouldn't weekly RAID consistency
checks reveal a bad block before you
had a failure that required the need to do a full resync?
It only
takes 3 hours to resync my 3 x 1TB drives and having a
bitmap would
reduce the performance. I've never had to have a resync in
the year
I've had the array up. I just wonder if the performance
drawback is
worth having the bitmap to save a possible resync once
every couple
years. Or are the RAID consistency checks not reliable
enough to
prevent more errors during a resync?
Ryan
On Wed, Aug 26, 2009 at 7:18 AM, Goswin von Brederlow[off-list ref]
wrote:
quoted
Jon Hardcastle [off-list ref]
writes:
quoted
quoted
Guys,
I have been having some problems with my arrays
that I think i have nailed down to a pci controller (well I
say that - it is always the drives connected to *a*
controller but I have tried 2!) anyway the latest saga is i
was trying some new kernel options last night - which didn't
work.
quoted
quoted
But when i booted up again this morning it said
one of the drives was in an inconsistent state (not sure of
the *exact* error message). I then kicked off an add of the
drive and it started syncing. It got about 5% in and then
the second drive in on that controller complained and the
array failed.
quoted
quoted
Is there any hope for my data? If i get a good
controller in there will the resync continue? can I try and
tell it to assume the drives are good (which they ought to
be)?
quoted
quoted
Please help!
The inconsistency is probably just a block here or
there and I'm
quoted
assuming none of your drives actualy failed. So
99.9999% of your data
quoted
should be there. Just rebooting might actualy just get
your raid back
quoted
(to syncing). If not then you have to force reassembly
from the drives
quoted
with the newest serials. That will give you some data
corruption,
quoted
whatever was writing when the controler gave errors.
Worst case you
quoted
have to recreate the raid with --assume-clean.
I recommend adding a bitmap to the raid. That way a
wrongfully failed
quoted
drive can be resynced in a matter of minutes instead
of hours or
quoted
days. Makes it way less likely another error occurs
during resync.
quoted
MfG
Goswin
--
To unsubscribe from this list: send the line
--
To unsubscribe from this list: send the line "unsubscribe
linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Robin Hill <hidden> Date: 2009-08-26 14:33:16
On Wed Aug 26, 2009 at 10:14:31AM -0400, Ryan Wagoner wrote:
Wouldn't weekly RAID consistency checks reveal a bad block before you
had a failure that required the need to do a full resync? It only
takes 3 hours to resync my 3 x 1TB drives and having a bitmap would
reduce the performance. I've never had to have a resync in the year
I've had the array up. I just wonder if the performance drawback is
worth having the bitmap to save a possible resync once every couple
years. Or are the RAID consistency checks not reliable enough to
prevent more errors during a resync?
If your system is that stable, then bitmaps will be a waste of time for
you. A lot of people have hardware/software issues which cause drives
to be kicked out of arrays occasionally, or arrays to fail to shut down
cleanly. A bitmap will save time when adding the drive back into the
array in these cases.
Cheers,
Robin
--
___
( ' } | Robin Hill [off-list ref] |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
From: Robin Hill <hidden> Date: 2009-08-26 14:50:22
On Wed Aug 26, 2009 at 07:19:51AM -0700, Jon Hardcastle wrote:
Can a bitmap be easily removed? I might give it ago if it can.
Yes - you can add/remove a bitmap at any time (on a non-degraded array).
I am never sure how thorough these checks are. Are they read/write, or
just read? for example. I make of point of doing read/write badblocks
checks with e2fck -cc when I do run them (not the automatic ones tho -
dunno how) but that only checks that partition, which is on LVM, which
is on RAID so WHO KNOWS what underlying drives are being checked.
My understanding is that the md "check" action does a read-only check,
verifying the checksum is valid for the data. The "repair" action
will rewrite the checksum if it's not valid. Neither of these will
write to the data blocks, or any valid checksum blocks.
Running e2fsck -cc should do a read/write check. This will only check
the filesystem data blocks though (and only on ext2/ext3 filesystems of
course), so will miss the LVM metadata and RAID checksums and metadata.
I have before now, dismantled the array and run read/write badblocks
directly on the constituent drives so at least smart is aware of them
and although i aim to do this once every six months, I think I have
actually done it only 1nce in the 2 year life of the array.
If you mean running badblocks in read/write mode, that'll be a
destructive test then. In this case, you're trading the risk of a
failure on one disk for the risk of a failure on one of the others
during rebuild.
You could also run background SMART tests (though this has caused drives
to be kicked out of the array on some occasions for me) - these look to
be mostly read-only tests again (though I'm not 100% sure on that).
Cheers,
Robin
--
___
( ' } | Robin Hill [off-list ref] |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |