Re: Fw: Why does one get mismatches?

From: Roger Heflin <hidden>
Date: 2010-01-24 21:52:36

Jon Hardcastle wrote:

quoted hunk ↗ jump to hunk

--- On Fri, 22/1/10, Goswin von Brederlow <goswin-v-b@web.de> wrote:

quoted

From: Goswin von Brederlow <redacted>
Subject: Re: Fw: Why does one get mismatches?
To: Jon@eHardcastle.com
Cc: linux-raid@vger.kernel.org
Date: Friday, 22 January, 2010, 18:13
Jon Hardcastle [off-list ref]
writes:

quoted

--- On Tue, 19/1/10, Jon Hardcastle <jd_hardcastle@yahoo.com>

wrote:

quoted

From: Jon Hardcastle <redacted>
Subject: Why does one get mismatches?
To: linux-raid@vger.kernel.org
Date: Tuesday, 19 January, 2010, 10:04
Hi,

I kicked off a check/repair cycle on my machine

after i

quoted

moved the phyiscal ordering of my drives around

and I am now

quoted

on my second check/repair cycle and it has kept

finding

quoted

mismatches.

Is it correct that the mismatch value after a

repair was

quoted

needed should equal the value present after a

check? What if

quoted

it doesn't? What does it mean if another check

STILL reveals

quoted

mismatches?

I had something similar after i reshaped from raid

5 to 6 i

quoted

had to run check/repair/check/repair several times

before i

quoted

got my 0.

Guys,

Anyone got any suggestions here? I am now on my ~5

check/repair and after a reboot the first check is still
returning 8.

quoted

All i have done is move the drives around. It is the

same controllers/cables/etc

quoted

I really dont like the seeming random nature of what

can/does/has caused the mismatches?

There is some unknown corruption going on with raid1 that
causes
mismatches but it is believed that it will never occur on
any used
block. Swapping is a likely cause.

Any swap device on the raid? Try turning that off.
If that doesn't help try umounting filesystems or
remounting RO.

MfG
        Goswin

Hello, my usual savior Goswin!

The deal is it is a 7 drive raid 6 array. it has LVM on it and is not used for swapping. I have umounted all LV's and still got mismatches, i run smartctl --test=long on all drives - nothing. I have now dismantled the array and am 3/4 the way through 'badblocks -svn' on each of the component drive. I have a hunch that it may be a dodgy SATA cable but have no evidence. No errors in log, nothing on dmesg.

Is there any way to get more information? I am starting to think this is more happened since i changed from raid 5 to 6..... which i did < 1 month ago.

The only lead i have is that whilst doing the bad blocks 1 drive ran at ~10~15MB/s whereas the rest are going at ~30 i have another identical model drive coming up so i will see if that one is slow too. But the lack of logging info is not helpful and worrying! and the prospect of silent corruption a big worry!

It is possible that the reads are somehow corrupting sometimes.

I have seen a couple of different controllers fail and result in read 
corruptions, basically you have 50 largish files or so on the disk 
with the same checksum (50xsize needs to be 2x greater than ram), and 
you cksum all of the files and see if the cksum changes, if it does 
the "bad" file will move around, so in this case the data on disk 
should be ok.   I have seen a couple of different companies controller 
fail this way, usually it is from a bad PCI interface chip or a bad 
config (too fast) causing PCI parity errors.  I had one controller 
fail (broken) and cause errors (replaced with spare corrected), and in 
the second case I found that the MB was running the PCI bus too faster 
for the number of cards (two different companies FC card fails--both 
in slightly different ways-one silently corrupted, the other crashed 
the machine about the time an error would have been expected), and had 
to slow the bus down one step (PCIX-133 -> PCIX-100, or PCIX-100 to 
PCIX-66) and the issue went away.

In both cases I did not find any write corruptions, but found read 
corruptions often, if you have this happening with a raid5 device it 
would be bad if you had to use parity (corrupt read would mean 
regenerated parity would be wrong, and restore from parity would lead 
to corrupted data).

I don't know how strong the internal SATA communication is, if it uses 
CRC's errors are almost impossible on the cable, if it uses parity 
errors are easy, the PCI bus uses parity, so it is pretty easy for 
errors to get through, but I have only seen them very rarely, maybe 5 
times in 10,000 years of machine operations (2000+ machines for 
several years).

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help