From: Neil Brown <hidden> Date: 2007-06-21 02:57:40
On Monday June 18, dgc@sgi.com wrote:
On Sat, Jun 16, 2007 at 07:59:29AM +1000, Neil Brown wrote:
quoted
Combining these thoughts, it would make a lot of sense for the
filesystem to be able to say to the block device "That blocks looks
wrong - can you find me another copy to try?". That is an example of
the sort of closer integration between filesystem and RAID that would
make sense.
I think that this would only be useful on devices that store
discrete copies of the blocks on different devices i.e. mirrors. If
it's an XOR based RAID, you don't have another copy you can
retreive....
You could reconstruct the block in question from all the other blocks
(including parity) and see if that differs from the data block read
from disk... For RAID6, there would be a number of different ways to
calculate alternate blocks. Not convinced that it is actually
something we want to do, but it is a possibility.
I have that - apparently naive - idea that drives use strong checksum,
and will never return bad data, only good data or an error. If this
isn't right, then it would really help to understand what the cause of
other failures are before working out how to handle them....
NeilBrown
From: David Chinner <hidden> Date: 2007-06-21 06:40:16
On Thu, Jun 21, 2007 at 12:56:44PM +1000, Neil Brown wrote:
On Monday June 18, dgc@sgi.com wrote:
quoted
On Sat, Jun 16, 2007 at 07:59:29AM +1000, Neil Brown wrote:
quoted
Combining these thoughts, it would make a lot of sense for the
filesystem to be able to say to the block device "That blocks looks
wrong - can you find me another copy to try?". That is an example of
the sort of closer integration between filesystem and RAID that would
make sense.
I think that this would only be useful on devices that store
discrete copies of the blocks on different devices i.e. mirrors. If
it's an XOR based RAID, you don't have another copy you can
retreive....
You could reconstruct the block in question from all the other blocks
(including parity) and see if that differs from the data block read
from disk... For RAID6, there would be a number of different ways to
calculate alternate blocks. Not convinced that it is actually
something we want to do, but it is a possibility.
Agreed - it's not as straight forward as a mirror, and it kind of assumes
that you have software RAID.
/me had his head stuck in hw raid land ;)
I have that - apparently naive - idea that drives use strong checksum,
and will never return bad data, only good data or an error. If this
isn't right, then it would really help to understand what the cause of
other failures are before working out how to handle them....
The drive is not the only source of errors, though. You could
have a path problem that is corrupting random bits between the drive
and the filesystem. So the data on the disk might be fine, and
reading it via a redundant path might be all that is needed.
Yeah, so I can see how having a different retry semantic would be a
good idea. i.e. if we do a READ_VERIFY I/O, the underlying device
attempts to verify the data is good in as many ways as possible
before returning the verified data or an error.
I guess a filesystem read would become something like this:
verified = 0
error = read(block)
if (error) {
read_verify:
error = read_verify(block)
if (error) {
OMG THE SKY IS FALLING
return error
}
verified = 1
}
/* check contents */
if (contents are bad) {
if (!verified)
goto read_verify
OMG THE SKY HAS FALLEN
return -EIO
}
Is this the sort of erro handling and re-issuing of
I/O that you had in mind?
FWIW, I don't think this really removes the need for a filesystem to
be able to keep multiple copies of stuff about. If the copy(s) on a
device are gone, you've still got to have another copy somewhere
else to get it back...
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
On Thu, Jun 21, 2007 at 12:56:44PM +1000, Neil Brown wrote:
quoted
I have that - apparently naive - idea that drives use strong checksum,
and will never return bad data, only good data or an error. If this
isn't right, then it would really help to understand what the cause of
other failures are before working out how to handle them....
The drive is not the only source of errors, though. You could
have a path problem that is corrupting random bits between the drive
and the filesystem. So the data on the disk might be fine, and
reading it via a redundant path might be all that is needed.
one of the 'killer features' of zfs is that it does checksums of every
file on disk. so many people don't consider the disk infallable.
several other filesystems also do checksums
both bitkeeper and git do checksums of files to detect disk corruption
as david C points out there are many points in the path where the data
could get corrupted besides on the platter.
David Lang
From: David Greaves <hidden> Date: 2007-06-21 09:00:05
david@lang.hm wrote:
On Thu, 21 Jun 2007, David Chinner wrote:
one of the 'killer features' of zfs is that it does checksums of every
file on disk. so many people don't consider the disk infallable.
several other filesystems also do checksums
both bitkeeper and git do checksums of files to detect disk corruption
How different is that to raid1/5/6 being set to a 'paranoid' "read-verify" mode
(as per Dan's recent email) where a read reads from _all_ spindles and verifies
(and with R6 maybe corrects) the stripe before returning it?
Doesn't solve DaveC's issue about the fs doing redundancy but isn't that
essentially just fs level mirroring?
David
From: David Chinner <hidden> Date: 2007-06-21 11:00:47
On Thu, Jun 21, 2007 at 04:39:36PM +1000, David Chinner wrote:
FWIW, I don't think this really removes the need for a filesystem to
be able to keep multiple copies of stuff about. If the copy(s) on a
device are gone, you've still got to have another copy somewhere
else to get it back...
Speaking of knowing where you can safely put multiple copies, I'm in
the process of telling XFS about linear alignment of the underlying
array so that we can:
- spread out the load across it faster.
- provide workload isolation
- know where *not* to put duplicate or EDAC data
I'm aiming at identical subvolumes so it's simple to implement. All
I need to know is the size of each subvolume. I can supply that at
mkfs time or in a mount option, but I want something that can works
automatically so I need to query dm to find out the size of each
underlying device during mount. We should also pass stripe
unit/width with the same interface while we are at it...
What's the correct and safe way to get this information from dm
both into the kernel and out to userspace (mkfs)?
FWIW, my end goal is to be able to map the underlying block device
address spaces directly into the filesystem so that the filesystem
is able to use the underlying devices intelligently and I can
logically separate caches and writeback for the separate subdevices.
A struct address_space per subdevice would be ideal - anyone got
ideas on how to get that?
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
I have that - apparently naive - idea that drives use strong checksum,
and will never return bad data, only good data or an error. If this
isn't right, then it would really help to understand what the cause of
other failures are before working out how to handle them....
In theory, that's how storage should work. In practice, silent data
corruption does happen. If not from the disks themselves, somewhere along
the path of cables, controllers, drivers, buses, etc. If you add in fcal,
you'll get even more sources of failure, but usually you can avoid SANs
(if you care about your data).
Well, here is a couple of the issues that I've seen myself:
A hw-raid controller returning every 64th bit as 0, no matter what's on
disk. With no error condition at all. (I've also heard from a collegue
about this on every 64k, but not seen that myself.)
An fcal switch occasionally resetting, garbling the blocks in transit with
random data. Lost a few TB of user data that way.
Add to this the random driver breakage that happens now and then. I've
also had a few broken filesystems due to in-memory corruption due to bad
ram, not sure there is much hope of fixing that though.
Also, this presentation is pretty worrying on the frequency of silent data
corruption:
https://indico.desy.de/contributionDisplay.py?contribId=65&sessionId=42&confId=257
/Mattias Wadenstein
I have that - apparently naive - idea that drives use strong checksum,
and will never return bad data, only good data or an error. If this
isn't right, then it would really help to understand what the cause of
other failures are before working out how to handle them....
In theory, that's how storage should work. In practice, silent data
corruption does happen. If not from the disks themselves, somewhere along the
path of cables, controllers, drivers, buses, etc. If you add in fcal, you'll
get even more sources of failure, but usually you can avoid SANs (if you care
about your data).
Well, here is a couple of the issues that I've seen myself:
A hw-raid controller returning every 64th bit as 0, no matter what's on disk.
With no error condition at all. (I've also heard from a collegue about this
on every 64k, but not seen that myself.)
An fcal switch occasionally resetting, garbling the blocks in transit with
random data. Lost a few TB of user data that way.
Add to this the random driver breakage that happens now and then. I've also
had a few broken filesystems due to in-memory corruption due to bad ram, not
sure there is much hope of fixing that though.
Also, this presentation is pretty worrying on the frequency of silent data
corruption:
https://indico.desy.de/contributionDisplay.py?contribId=65&sessionId=42&confId=257
/Mattias Wadenstein
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Very interesting slides/presentation, going to watch it shortly.
I have that - apparently naive - idea that drives use strong checksum,
and will never return bad data, only good data or an error. If this
isn't right, then it would really help to understand what the cause of
other failures are before working out how to handle them....
In theory, that's how storage should work. In practice, silent data
corruption does happen. If not from the disks themselves, somewhere along the
path of cables, controllers, drivers, buses, etc. If you add in fcal, you'll
get even more sources of failure, but usually you can avoid SANs (if you care
about your data).
heh, the pitch I get from the self proclaimed experts is that if you care
about your data you put it on the san (so you can take advantage of the
more expensive disk arrays, various backup advantages, and replication
features that tend to be focused on the san becouse it's a big target)
David Lang
Well, here is a couple of the issues that I've seen myself:
A hw-raid controller returning every 64th bit as 0, no matter what's on disk.
With no error condition at all. (I've also heard from a collegue about this
on every 64k, but not seen that myself.)
An fcal switch occasionally resetting, garbling the blocks in transit with
random data. Lost a few TB of user data that way.
Add to this the random driver breakage that happens now and then. I've also
had a few broken filesystems due to in-memory corruption due to bad ram, not
sure there is much hope of fixing that though.
Also, this presentation is pretty worrying on the frequency of silent data
corruption:
https://indico.desy.de/contributionDisplay.py?contribId=65&sessionId=42&confId=257
/Mattias Wadenstein
From: Mark Lord <hidden> Date: 2007-06-21 17:00:48
david@lang.hm wrote:
On Thu, 21 Jun 2007, David Chinner wrote:
quoted
On Thu, Jun 21, 2007 at 12:56:44PM +1000, Neil Brown wrote:
quoted
I have that - apparently naive - idea that drives use strong checksum,
and will never return bad data, only good data or an error. If this
isn't right, then it would really help to understand what the cause of
other failures are before working out how to handle them....
The drive is not the only source of errors, though. You could
have a path problem that is corrupting random bits between the drive
and the filesystem. So the data on the disk might be fine, and
reading it via a redundant path might be all that is needed.
one of the 'killer features' of zfs is that it does checksums of every
file on disk. so many people don't consider the disk infallable.
several other filesystems also do checksums
both bitkeeper and git do checksums of files to detect disk corruption
No, all of those checksums are to detect *filesystem* corruption,
not device corruption (a mere side-effect).
as david C points out there are many points in the path where the data
could get corrupted besides on the platter.
Yup, that too.
But drives either return good data, or an error.
Cheers