Has anybody here tried using a >16TB RAID0? When I recently got my
hands on some 2TB drives, I decided to check out the current status of
large EXT4 filesystems on 32-bit systems. I created a ~17TB RAID0,
and immediately had problems. Andreas Dilger advised me to try out
his llverdev utility (kindly hosted by Val Aurora at
http://valhenson.livejournal.com/38933.html) to verify that the
underlying device is functioning properly. Running "llverdev -p -v
/dev/md0" on a >16TB array resulted in a runaway process, with
llverdev and pdflush each eating up 100% of the CPU time on my two
cores, but not advancing the write offset. The process did not appear
to be interruptable after more than 30 minutes, and I had to do a hard
shutdown. A RAID0 just under 16TB ran through llverdev without a
hitch.
I'm in the process of running a RAID5 array through the same test now,
but it's already past the point where the RAID0 failed; so if it does
fail, it'll likely be due to a different cause. Does anyone have any
ideas? Has this never been done before? I'm running debian on a
2.6.30.1 kernel.
-Justin
On Fri, July 17, 2009 1:36 pm, Justin Maggard wrote:
Has anybody here tried using a >16TB RAID0? When I recently got my
hands on some 2TB drives, I decided to check out the current status of
large EXT4 filesystems on 32-bit systems. I created a ~17TB RAID0,
and immediately had problems. Andreas Dilger advised me to try out
his llverdev utility (kindly hosted by Val Aurora at
http://valhenson.livejournal.com/38933.html) to verify that the
underlying device is functioning properly. Running "llverdev -p -v
/dev/md0" on a >16TB array resulted in a runaway process, with
llverdev and pdflush each eating up 100% of the CPU time on my two
cores, but not advancing the write offset. The process did not appear
to be interruptable after more than 30 minutes, and I had to do a hard
shutdown. A RAID0 just under 16TB ran through llverdev without a
hitch.
I'm in the process of running a RAID5 array through the same test now,
but it's already past the point where the RAID0 failed; so if it does
fail, it'll likely be due to a different cause. Does anyone have any
ideas? Has this never been done before? I'm running debian on a
2.6.30.1 kernel.
I'm not 100% sure, but a quick look at the code suggests that
16TB is the upper limit for normal read/write operations on
a block device like /dev/md0 on a 32bit host. This is because it uses
the page cache, and that uses 4K pages with a 32bit index, hence
16TB.
A filesystem using a 16TB device would not be limited in the same way
as it caches individual files, not the whole device (individual files
would still be limited to 16TB).
I think if you access the block device with O_DIRECT you might be
able to bypass the 16TB barrier. Try hacking the program
to add O_DIRECT to the open mode.
However this doesn't explain why it seems to work for RAID5. If I
am right, RAID5 should fail in the same way as RAID0.
But I would certainly expect RAID0 to work if anything does.
Confused.
NeilBrown
On Thu, Jul 16, 2009 at 8:59 PM, NeilBrown[off-list ref] wrote:
I'm not 100% sure, but a quick look at the code suggests that
16TB is the upper limit for normal read/write operations on
a block device like /dev/md0 on a 32bit host. This is because it uses
the page cache, and that uses 4K pages with a 32bit index, hence
16TB.
Yep, you're right. Sorry, I should have looked more into that first.
I was assuming LBD took care of that somehow, because the kernel docs
just say that LBD allows it to go over 2TB, and doesn't mention the
new 16TB upper limit.
However this doesn't explain why it seems to work for RAID5. If I
am right, RAID5 should fail in the same way as RAID0.
But I would certainly expect RAID0 to work if anything does.
And again you're correct. It eventually failed the comparison on
RAID5. But it failed in a much different way on RAID5 than RAID0 for
some reason. The same disk set on an x86_64 machine appears to be
working fine. Sorry for the false alarm. So I guess now my question
is, should the kernel and/or mdadm refuse to create or run a >16TB
array on a 32-bit kernel? :)
Thanks much for your help!
-Justin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I'm not 100% sure, but a quick look at the code suggests that
16TB is the upper limit for normal read/write operations on
a block device like /dev/md0 on a 32bit host. This is because it uses
the page cache, and that uses 4K pages with a 32bit index, hence
16TB.
Uh-oh! I hope this doesn't apply to a 64 bit host running
2.6.26-2-amd64, or is it a limitation of md? In fairly short order (a year,
maybe), I'm going to need to go beyond 16T employing RAID6 on some number
more than eight 3T drives. I know a lot can happen in a year, but when it
comes to upgrading code, sometimes a lot doesn't happen, either.
A filesystem using a 16TB device would not be limited in the same way
as it caches individual files, not the whole device (individual files
would still be limited to 16TB).
I think if you access the block device with O_DIRECT you might be
able to bypass the 16TB barrier. Try hacking the program
to add O_DIRECT to the open mode.
I'm running XFS, but perhaps I could get the developers to put in a
switch for this, if need be.
On Sat, July 18, 2009 8:39 am, Leslie Rhorer wrote:
quoted
I'm not 100% sure, but a quick look at the code suggests that
16TB is the upper limit for normal read/write operations on
a block device like /dev/md0 on a 32bit host. This is because it uses
the page cache, and that uses 4K pages with a 32bit index, hence
16TB.
Uh-oh! I hope this doesn't apply to a 64 bit host running
2.6.26-2-amd64, or is it a limitation of md? In fairly short order (a
year,
maybe), I'm going to need to go beyond 16T employing RAID6 on some number
more than eight 3T drives. I know a lot can happen in a year, but when it
comes to upgrading code, sometimes a lot doesn't happen, either.
No, this does not apply to 64bit hosts. Only 32bit.
quoted
A filesystem using a 16TB device would not be limited in the same way
as it caches individual files, not the whole device (individual files
would still be limited to 16TB).
I think if you access the block device with O_DIRECT you might be
able to bypass the 16TB barrier. Try hacking the program
to add O_DIRECT to the open mode.
I'm running XFS, but perhaps I could get the developers to put in a
switch for this, if need be.
If the filesystem uses the block device to cache metadata - as several do,
though I don't know about XFS - then no O_DIRECT change is useful.
If the filesystem doesn't depend on the block device's cache, then
there might be value in using O_DIRECT for e.g. mkfs/fsck on
16TB devices on 32 bit systems.
I have just sent an Email to fs devel asking about this.
But just to repeat: on a 64bit host, this is not an issue. You can have
devices so large that I don't know the correct greek prefix and they
should still work. It is only on 32bit hosts where 16TB can be
a problem.
NeilBrown
On Sat, July 18, 2009 5:47 am, Justin Maggard wrote:
On Thu, Jul 16, 2009 at 8:59 PM, NeilBrown[off-list ref] wrote:
quoted
I'm not 100% sure, but a quick look at the code suggests that
16TB is the upper limit for normal read/write operations on
a block device like /dev/md0 on a 32bit host. This is because it uses
the page cache, and that uses 4K pages with a 32bit index, hence
16TB.
Yep, you're right. Sorry, I should have looked more into that first.
I was assuming LBD took care of that somehow, because the kernel docs
just say that LBD allows it to go over 2TB, and doesn't mention the
new 16TB upper limit.
quoted
However this doesn't explain why it seems to work for RAID5. If I
am right, RAID5 should fail in the same way as RAID0.
But I would certainly expect RAID0 to work if anything does.
And again you're correct. It eventually failed the comparison on
RAID5. But it failed in a much different way on RAID5 than RAID0 for
some reason. The same disk set on an x86_64 machine appears to be
working fine. Sorry for the false alarm. So I guess now my question
is, should the kernel and/or mdadm refuse to create or run a >16TB
array on a 32-bit kernel? :)
I'm not sure that refusal is appropriate as there might be some use
cases that work perfectly.
Emitting a warning my be a good idea though, and possibly disabling
some of the use cases that are known to be unreliable.
Thanks,
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
But just to repeat: on a 64bit host, this is not an issue. You can have
devices so large that I don't know the correct greek prefix and they
should still work. It is only on 32bit hosts where 16TB can be
a problem.
Assuming 512 byte sectors and an unsigned 64 bit integer sector table, 18
exabytes.