From: Neil Brown <hidden> Date: 2007-06-16 03:47:27
On Friday June 15, wakko@animx.eu.org wrote:
As I understand the way
raid works, when you write a block to the array, it will have to read all
the other blocks in the stripe and recalculate the parity and write it out.
Your understanding is incomplete.
For raid5 on an array with more than 3 drive, if you attempt to write
a single block, it will:
- read the current value of the block, and the parity block.
- "subtract" the old value of the block from the parity, and "add"
the new value.
- write out the new data and the new parity.
If the parity was wrong before, it will still be wrong. If you then
lose a drive, you lose your data.
With the current implementation in md, this only affect RAID5. RAID6
will always behave as you describe. But I don't promise that won't
change with time.
It would be possible to have a 'this is not initialised' flag on the
array, and if that is not set, always do a reconstruct-write rather
than a read-modify-write. But the first time you have an unclean
shutdown you are going to resync all the parity anyway (unless you
have a bitmap....) so you may as well resync at the start.
And why is it such a big deal anyway? The initial resync doesn't stop
you from using the array. I guess if you wanted to put an array into
production instantly and couldn't afford any slowdown due to resync,
then you might want to skip the initial resync.... but is that really
likely?
NeilBrown
From: Dan Merillat <hidden> Date: 2007-06-16 04:40:29
For raid5 on an array with more than 3 drive, if you attempt to write
a single block, it will:
- read the current value of the block, and the parity block.
- "subtract" the old value of the block from the parity, and "add"
the new value.
- write out the new data and the new parity.
If the parity was wrong before, it will still be wrong. If you then
lose a drive, you lose your data.
Wow, that really needs to be put somewhere in 120 point red blinking
text. A lot of us are used to uninitialized disks calculating the
parity-on-first-write, but if linux MD is forgoeing that
'dangerous-no-resync' sounds really REALLY bad. How about at least a
'Warning: unlike other systems this WILL cause corruption if you
forego reconstruction' on mkraid?
It would be possible to have a 'this is not initialised' flag on the
array, and if that is not set, always do a reconstruct-write rather
than a read-modify-write. But the first time you have an unclean
shutdown you are going to resync all the parity anyway (unless you
have a bitmap....) so you may as well resync at the start.
And why is it such a big deal anyway? The initial resync doesn't stop
you from using the array. I guess if you wanted to put an array into
production instantly and couldn't afford any slowdown due to resync,
then you might want to skip the initial resync.... but is that really
likely?
in my case it takes 2+ days to resync the array before I can do any
performance testing with it. for some reason it's only doing the rebuild
at ~5M/sec (even though I've increased the min and max rebuild speeds and
a dd to the array seems to be ~44M/sec, even during the rebuild)
I want to test several configurations, from a 45 disk raid6 to a 45 disk
raid0. at 2-3 days per test (or longer, depending on the tests) this
becomes a very slow process.
also, when a rebuild is slow enough (and has enough of a performance
impact) it's not uncommon to want to operate in degraded mode just long
enought oget to a maintinance window and then recreate the array and
reload from backup.
David Lang
From: David Greaves <hidden> Date: 2007-06-16 13:33:37
Neil Brown wrote:
On Friday June 15, wakko@animx.eu.org wrote:
quoted
As I understand the way
raid works, when you write a block to the array, it will have to read all
the other blocks in the stripe and recalculate the parity and write it out.
Your understanding is incomplete.
Does this help?
[for future reference so you can paste a url and save the typing for code :) ]
http://linux-raid.osdl.org/index.php/Initial_Array_Creation
David
Initial Creation
When mdadm asks the kernel to create a raid array the most noticeable activity
is what's called the "initial resync".
The kernel takes one (or two for raid6) disks and marks them as 'spare'; it then
creates the array in degraded mode. It then marks spare disks as 'rebuilding'
and starts to read from the 'good' disks, calculate the parity and determines
what should be on any spare disks and then writes it. Once all this is done the
array is clean and all disks are active.
This can take quite a time and the array is not fully resilient whilst this is
happening (it is however fully useable).
--assume-clean
Some people have noticed the --assume-clean option in mdadm and speculated that
this can be used to skip the initial resync. Which it does. But this is a bad
idea in some cases - and a *very* bad idea in others.
raid5
For raid5 especially it is NOT safe to skip the initial sync. The raid5
implementation optimises use of the component disks and it is possible for all
updates to be "read-modify-write" updates which assume the parity is correct. If
it is wrong, it stays wrong. Then when you lose a drive, the parity blocks are
wrong so the data you recover using them is wrong. In other words - you will get
data corruption.
For raid5 on an array with more than 3 drive, if you attempt to write a single
block, it will:
* read the current value of the block, and the parity block.
* "subtract" the old value of the block from the parity, and "add" the new
value.
* write out the new data and the new parity.
If the parity was wrong before, it will still be wrong. If you then lose a
drive, you lose your data.
linear, raid0,1,10
These raid levels do not need an initial sync.
linear and raid0 have no redundancy.
raid1 always writes all data to all disks.
raid10 always writes all data to all relevant disks.
Other raid levels
Probably the most noticeable effect for the other raid levels is that if you
don't sync first, then every check will find lots of errors. (Of course you
could 'repair' instead of 'check'. Or do that once. Or something.)
For raid6 it is also safe to not sync first, though with the same caveat. Raid6
always updates parity by reading all blocks in the stripe that aren't known and
calculating P and Q. So the first write to a stripe will make P and Q correct
for that stripe. This is current behaviour. There is no guarantee it will never
changed (so theoretically one day you may upgrade your kernel and suffer data
corruption on an old raid6 array).
Summary
In summary, it is safe to use --assume-clean on a raid1 or raid1o, though a
"repair" is recommended before too long. For other raid levels it is best avoided.
Potential 'Solutions'
There have been 'solutions' suggested including the use of bitmaps to
efficiently store 'not yet synced' information about the array. It would be
possible to have a 'this is not initialised' flag on the array, and if that is
not set, always do a reconstruct-write rather than a read-modify-write. But the
first time you have an unclean shutdown you are going to resync all the parity
anyway (unless you have a bitmap....) so you may as well resync at the start. So
essentially, at the moment, there is no interest in implementing this since the
added complexity is not justified.
What's the problem anyway?
First of all RAID is all about being safe with your data.
And why is it such a big deal anyway? The initial resync doesn't stop you from
using the array. If you wanted to put an array into production instantly and
couldn't afford any slowdown due to resync, then you might want to skip the
initial resync.... but is that really likely?
So what is --assume-clean for then?
Disaster recovery. If you want to build an array from components that used to be
in a raid then this stops the kernel from scribbling on them. As the man page says :
"Use this ony if you really know what you are doing."
From: David Greaves <hidden> Date: 2007-06-16 13:38:51
david@lang.hm wrote:
On Sat, 16 Jun 2007, Neil Brown wrote:
I want to test several configurations, from a 45 disk raid6 to a 45 disk
raid0. at 2-3 days per test (or longer, depending on the tests) this
becomes a very slow process.
Are you suggesting the code that is written to enhance data integrity is
optimised (or even touched) to support this kind of test scenario?
Seriously? :)
also, when a rebuild is slow enough (and has enough of a performance
impact) it's not uncommon to want to operate in degraded mode just long
enought oget to a maintinance window and then recreate the array and
reload from backup.
so would mdadm --remove the rebuilding disk help?
David
From: Wakko Warner <hidden> Date: 2007-06-16 14:13:00
Neil Brown wrote:
On Friday June 15, wakko@animx.eu.org wrote:
quoted
As I understand the way
raid works, when you write a block to the array, it will have to read all
the other blocks in the stripe and recalculate the parity and write it out.
Your understanding is incomplete.
For raid5 on an array with more than 3 drive, if you attempt to write
a single block, it will:
- read the current value of the block, and the parity block.
- "subtract" the old value of the block from the parity, and "add"
the new value.
- write out the new data and the new parity.
If the parity was wrong before, it will still be wrong. If you then
lose a drive, you lose your data.
I see, I didn't know that the MD's raid5 did this.
And why is it such a big deal anyway? The initial resync doesn't stop
you from using the array. I guess if you wanted to put an array into
production instantly and couldn't afford any slowdown due to resync,
then you might want to skip the initial resync.... but is that really
likely?
When I've had an unclean shutdown on one of my systems (10x 50gb raid5) it's
always slowed the system down when booting up. Quite significantly I must
say. I wait until I can login and change the rebuild max speed to slow it
down while I'm using it. But that is another thing.
Thanks for the clarification on raid5.
--
Lab tests show that use of micro$oft causes cancer in lab animals
Got Gas???
On Sat, 16 Jun 2007, Neil Brown wrote:
I want to test several configurations, from a 45 disk raid6 to a 45 disk
raid0. at 2-3 days per test (or longer, depending on the tests) this
becomes a very slow process.
Are you suggesting the code that is written to enhance data integrity is
optimised (or even touched) to support this kind of test scenario?
Seriously? :)
actually, if it can be done without a huge impact to the maintainability
of the code I think it would be a good idea for the simple reason that I
think the increased experimentation would result in people finding out
what raid level is really appropriate for their needs.
there is a _lot_ of confusion around about what the performance
implications of different raid levels are (especially when you consider
things like raid 10/50/60 where you have two layers combined) and anything
that encourages experimentation would be a good thing.
quoted
also, when a rebuild is slow enough (and has enough of a performance
impact) it's not uncommon to want to operate in degraded mode just long
enought oget to a maintinance window and then recreate the array and
reload from backup.
so would mdadm --remove the rebuilding disk help?
no. let me try again
drive fails monday morning
scenerio 1
replace the failed drive, start the rebuild. system will be slow (degraded
mode + rebuild) for the next three days.
scenerio 2
leave it in degraded mode until monday night (accepting the speed penalty
for degraded mode, but not the rebuild penalty)
monday night shutdown the system, put in the new drive, reinitialize the
array, reload the system from backup.
system is back to full speed tuesday morning.
scenerio 2 isn't supported with md today, although it sounds as if the
skip rebuild could do this except for raid 5
on my test system, the rebuild says it's running at 5M/s a DD to a file on
the array says it's doing 45M/s (even while the rebuild is running), so it
seems to me that there may be value in this approach.
David Lang
From: dean gaudet <hidden> Date: 2007-06-17 01:44:23
On Sat, 16 Jun 2007, David Greaves wrote:
Neil Brown wrote:
quoted
On Friday June 15, wakko@animx.eu.org wrote:
quoted
As I understand the way
raid works, when you write a block to the array, it will have to read all
the other blocks in the stripe and recalculate the parity and write it
out.
i fixed a typo and added one more note which i think is quite fair:
It is also safe to use --assume-clean if you are performing
performance measurements of different raid configurations. Just
be sure to rebuild your array without --assume-clean when you
decide on your final configuration.
-dean
From: dean gaudet <hidden> Date: 2007-06-17 01:47:43
On Sat, 16 Jun 2007, Wakko Warner wrote:
When I've had an unclean shutdown on one of my systems (10x 50gb raid5) it's
always slowed the system down when booting up. Quite significantly I must
say. I wait until I can login and change the rebuild max speed to slow it
down while I'm using it. But that is another thing.
i use an external write-intent bitmap on a raid1 to avoid this... you
could use internal bitmap but that slows down i/o too much for my tastes.
i also use an external xfs journal for the same reason. 2 disk raid1 for
root/journal/bitmap, N disk raid5 for bulk storage. no spindles in
common.
-dean
From: Wakko Warner <hidden> Date: 2007-06-17 13:32:33
dean gaudet wrote:
On Sat, 16 Jun 2007, Wakko Warner wrote:
quoted
When I've had an unclean shutdown on one of my systems (10x 50gb raid5) it's
always slowed the system down when booting up. Quite significantly I must
say. I wait until I can login and change the rebuild max speed to slow it
down while I'm using it. But that is another thing.
i use an external write-intent bitmap on a raid1 to avoid this... you
could use internal bitmap but that slows down i/o too much for my tastes.
i also use an external xfs journal for the same reason. 2 disk raid1 for
root/journal/bitmap, N disk raid5 for bulk storage. no spindles in
common.
I must remember this if I have to rebuild the array. Although I'm
considering moving to a hardware raid solution when I upgrade my storage.
--
Lab tests show that use of micro$oft causes cancer in lab animals
Got Gas???
From: Bill Davidsen <hidden> Date: 2007-06-17 17:16:36
david@lang.hm wrote:
On Sat, 16 Jun 2007, Neil Brown wrote:
quoted
It would be possible to have a 'this is not initialised' flag on the
array, and if that is not set, always do a reconstruct-write rather
than a read-modify-write. But the first time you have an unclean
shutdown you are going to resync all the parity anyway (unless you
have a bitmap....) so you may as well resync at the start.
And why is it such a big deal anyway? The initial resync doesn't stop
you from using the array. I guess if you wanted to put an array into
production instantly and couldn't afford any slowdown due to resync,
then you might want to skip the initial resync.... but is that really
likely?
in my case it takes 2+ days to resync the array before I can do any
performance testing with it. for some reason it's only doing the
rebuild at ~5M/sec (even though I've increased the min and max rebuild
speeds and a dd to the array seems to be ~44M/sec, even during the
rebuild)
I want to test several configurations, from a 45 disk raid6 to a 45
disk raid0. at 2-3 days per test (or longer, depending on the tests)
this becomes a very slow process.
I've been doing stuff like this, but I just build the array on a
partition per drive so the init is livable. For the stuff I'm doing a
total of 500-100GB is ample to do performance testing.
also, when a rebuild is slow enough (and has enough of a performance
impact) it's not uncommon to want to operate in degraded mode just
long enought oget to a maintinance window and then recreate the array
and reload from backup.
--
bill davidsen [off-list ref]
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979
From: dean gaudet <hidden> Date: 2007-06-17 17:28:19
On Sun, 17 Jun 2007, Wakko Warner wrote:
dean gaudet wrote:
quoted
On Sat, 16 Jun 2007, Wakko Warner wrote:
quoted
When I've had an unclean shutdown on one of my systems (10x 50gb raid5) it's
always slowed the system down when booting up. Quite significantly I must
say. I wait until I can login and change the rebuild max speed to slow it
down while I'm using it. But that is another thing.
i use an external write-intent bitmap on a raid1 to avoid this... you
could use internal bitmap but that slows down i/o too much for my tastes.
i also use an external xfs journal for the same reason. 2 disk raid1 for
root/journal/bitmap, N disk raid5 for bulk storage. no spindles in
common.
I must remember this if I have to rebuild the array. Although I'm
considering moving to a hardware raid solution when I upgrade my storage.
you can do it without a rebuild -- that's in fact how i did it the first
time.
to add an external bitmap:
mdadm --grow --bitmap /bitmapfile /dev/mdX
plus add "bitmap=/bitmapfile" to mdadm.conf... as in:
ARRAY /dev/md4 bitmap=/bitmap.md4 UUID=dbc3be0b:b5853930:a02e038c:13ba8cdc
you can also easily move an ext3 journal to an external journal with
tune2fs (see man page).
if you use XFS it's a bit more of a challenge to convert from internal to
external, but see this thread:
http://marc.theaimsgroup.com/?l=linux-xfs&m=106929781232520&w=2
i found that i had to do "sb 1", "sb 2", ..., "sb N" for all sb rather
than just the "sb 0" that email instructed me to do.
-dean
From: Wakko Warner <hidden> Date: 2007-06-17 19:35:31
dean gaudet wrote:
On Sun, 17 Jun 2007, Wakko Warner wrote:
quoted
quoted
i use an external write-intent bitmap on a raid1 to avoid this... you
could use internal bitmap but that slows down i/o too much for my tastes.
i also use an external xfs journal for the same reason. 2 disk raid1 for
root/journal/bitmap, N disk raid5 for bulk storage. no spindles in
common.
I must remember this if I have to rebuild the array. Although I'm
considering moving to a hardware raid solution when I upgrade my storage.
you can do it without a rebuild -- that's in fact how i did it the first
time.
to add an external bitmap:
mdadm --grow --bitmap /bitmapfile /dev/mdX
plus add "bitmap=/bitmapfile" to mdadm.conf... as in:
ARRAY /dev/md4 bitmap=/bitmap.md4 UUID=dbc3be0b:b5853930:a02e038c:13ba8cdc
I used evms to setup mine. I have used mdadm in the past. I use lvm ontop
of it which evms makes it a little easier to maintain. I have 3 arrays
total (only the raid5 was configured by evms, the other 2 raid1s were done
by hand)
you can also easily move an ext3 journal to an external journal with
tune2fs (see man page).
I only have 2 ext3 file systems (One of which is mounted R/O since it's
full), all my others are reiserfs (v3).
What benefit would I gain by using an external journel and how big would it
need to be?
if you use XFS it's a bit more of a challenge to convert from internal to
external, but see this thread:
I specifically didn't use XFS (or JFS) since neither one at the time could
be shrinked.
--
Lab tests show that use of micro$oft causes cancer in lab animals
Got Gas???
From: dean gaudet <hidden> Date: 2007-06-17 19:54:23
On Sun, 17 Jun 2007, Wakko Warner wrote:
What benefit would I gain by using an external journel and how big would it
need to be?
i don't know how big the journal needs to be... i'm limited by xfs'
maximum journal size of 128MiB.
i don't have much benchmark data -- but here are some rough notes i took
when i was evaluating a umem NVRAM card. since the pata disks in the
raid1 have write caching enabled it's somewhat of an unfair comparison,
but the important info is the 88 seconds for internal journal vs. 81
seconds for external journal.
-dean
time sh -c 'tar xf /var/tmp/linux-2.6.20.tar; sync'
xfs journal raid5 bitmap times
internal none 0.18s user 2.14s system 2% cpu 1:27.95 total
internal internal 0.16s user 2.16s system 1% cpu 2:01.12 total
raid1 none 0.07s user 2.02s system 2% cpu 1:20.62 total
raid1 internal 0.14s user 2.01s system 1% cpu 1:55.18 total
raid1 raid1 0.14s user 2.03s system 2% cpu 1:20.61 total
umem none 0.13s user 2.07s system 2% cpu 1:20.77 total
umem internal 0.15s user 2.16s system 2% cpu 1:51.28 total
umem umem 0.12s user 2.13s system 2% cpu 1:20.50 total
raid5:
- 4x seagate 7200.10 400GB on marvell MV88SX6081
- mdadm --create --level=5 --raid-devices=4 /dev/md4 /dev/sd[abcd]1
raid1:
- 2x maxtor 6Y200P0 on 3ware 7504
- two 128MiB partitions starting at cyl 1
- mdadm --create --level=1 --raid-disks=2 --auto=yes --assume-clean /dev/md1 /dev/sd[fg]1
- mdadm --create --level=1 --raid-disks=2 --auto=yes --assume-clean /dev/md2 /dev/sd[fg]2
- md1 is used for external xfs journal
- md2 has an ext3 filesystem for the external md4 bitmap
xfs:
- mkfs.xfs issued before each run using the defaults (aside from -l logdev=/dev/md1)
- mount -o noatime,nodiratime[,logdev=/dev/md1]
umem:
- 512MiB Micro Memory MM-5415CN
- 2 partitions similar to the raid1 setup
you can also easily move an ext3 journal to an external journal with
tune2fs (see man page).
I only have 2 ext3 file systems (One of which is mounted R/O since it's
full), all my others are reiserfs (v3).
What benefit would I gain by using an external journel and how big would it
need to be?
if you have the journal on a drive by itself you end up doing (almost)
sequential reads and writes to the journal and the disk head doesn't need
to move much.
this can greatly increase your write speeds since
1. the journal gets written faster (completeing the write as far as your
software is concerned)
2. the heads don't need to seek back and forth from the journal to the
final location that the data gets written.
as for how large it should be, it all depends on the volume of your
writes, once the journal fills up all writes stall until space is freed in
the journal, IIRC Ext3 is limited to 128M, with todays drive sizes I don't
see any reason to make it any smaller.
David Lang
What benefit would I gain by using an external journel and how big would it
need to be?
i don't know how big the journal needs to be... i'm limited by xfs'
maximum journal size of 128MiB.
i don't have much benchmark data -- but here are some rough notes i took
when i was evaluating a umem NVRAM card. since the pata disks in the
raid1 have write caching enabled it's somewhat of an unfair comparison,
but the important info is the 88 seconds for internal journal vs. 81
seconds for external journal.
if you turn on disk write caching the difference will be much larger.
-dean
time sh -c 'tar xf /var/tmp/linux-2.6.20.tar; sync'
I know that sync will force everything to get as far as the journal, will
it force the journal to be flushed?
David Lang
xfs journal raid5 bitmap times
internal none 0.18s user 2.14s system 2% cpu 1:27.95 total
internal internal 0.16s user 2.16s system 1% cpu 2:01.12 total
raid1 none 0.07s user 2.02s system 2% cpu 1:20.62 total
raid1 internal 0.14s user 2.01s system 1% cpu 1:55.18 total
raid1 raid1 0.14s user 2.03s system 2% cpu 1:20.61 total
umem none 0.13s user 2.07s system 2% cpu 1:20.77 total
umem internal 0.15s user 2.16s system 2% cpu 1:51.28 total
umem umem 0.12s user 2.13s system 2% cpu 1:20.50 total
raid5:
- 4x seagate 7200.10 400GB on marvell MV88SX6081
- mdadm --create --level=5 --raid-devices=4 /dev/md4 /dev/sd[abcd]1
raid1:
- 2x maxtor 6Y200P0 on 3ware 7504
- two 128MiB partitions starting at cyl 1
- mdadm --create --level=1 --raid-disks=2 --auto=yes --assume-clean /dev/md1 /dev/sd[fg]1
- mdadm --create --level=1 --raid-disks=2 --auto=yes --assume-clean /dev/md2 /dev/sd[fg]2
- md1 is used for external xfs journal
- md2 has an ext3 filesystem for the external md4 bitmap
xfs:
- mkfs.xfs issued before each run using the defaults (aside from -l logdev=/dev/md1)
- mount -o noatime,nodiratime[,logdev=/dev/md1]
umem:
- 512MiB Micro Memory MM-5415CN
- 2 partitions similar to the raid1 setup
in my case it takes 2+ days to resync the array before I can do any
performance testing with it. for some reason it's only doing the rebuild
at ~5M/sec (even though I've increased the min and max rebuild speeds
and a dd to the array seems to be ~44M/sec, even during the rebuild)
With performance like that, it sounds like you're saturating a bus
somewhere along the line. If you're using scsi, for instance, it's very
easy for a long chain of drives to overwhelm a channel. You might also
want to consider some other RAID layouts like 1+0 or 5+0 depending upon
your space vs. reliability needs.
--
Brendan Conoboy / Red Hat, Inc. / blc@redhat.com
in my case it takes 2+ days to resync the array before I can do any
performance testing with it. for some reason it's only doing the rebuild
at ~5M/sec (even though I've increased the min and max rebuild speeds and
a dd to the array seems to be ~44M/sec, even during the rebuild)
With performance like that, it sounds like you're saturating a bus somewhere
along the line. If you're using scsi, for instance, it's very easy for a
long chain of drives to overwhelm a channel. You might also want to consider
some other RAID layouts like 1+0 or 5+0 depending upon your space vs.
reliability needs.
I plan to test the different configurations.
however, if I was saturating the bus with the reconstruct how can I fire
off a dd if=/dev/zero of=/mnt/test and get ~45M/sec whild only slowing the
reconstruct to ~4M/sec?
I'm putting 10x as much data through the bus at that point, it would seem
to proove that it's not the bus that's saturated.
David Lang
From: Lennart Sorensen <hidden> Date: 2007-06-18 18:03:39
On Mon, Jun 18, 2007 at 10:28:38AM -0700, david@lang.hm wrote:
I plan to test the different configurations.
however, if I was saturating the bus with the reconstruct how can I fire
off a dd if=/dev/zero of=/mnt/test and get ~45M/sec whild only slowing the
reconstruct to ~4M/sec?
I'm putting 10x as much data through the bus at that point, it would seem
to proove that it's not the bus that's saturated.
dd 45MB/s from the raid sounds reasonable.
If you have 45 drives, doing a resync of raid5 or radi6 should probably
involve reading all the disks, and writing new parity data to one drive.
So if you are writing 5MB/s, then you are reading 44*5MB/s from the
other drives, which is 220MB/s. If your resync drops to 4MB/s when
doing dd, then you have 44*4MB/s which is 176MB/s or 44MB/s less read
capacity, which surprisingly seems to match the dd speed you are
getting. Seems like you are indeed very much saturating a bus
somewhere. The numbers certainly agree with that theory.
What kind of setup is the drives connected to?
--
Len Sorensen
I plan to test the different configurations.
however, if I was saturating the bus with the reconstruct how can I fire
off a dd if=/dev/zero of=/mnt/test and get ~45M/sec whild only slowing
the reconstruct to ~4M/sec?
I'm putting 10x as much data through the bus at that point, it would
seem to proove that it's not the bus that's saturated.
I am unconvinced. If you take ~1MB/s for each active drive, add in SCSI
overhead, 45M/sec seems reasonable. Have you look at a running iostat
while all this is going on? Try it out- add up the kb/s from each drive
and see how close you are to your maximum theoretical IO.
Also, how's your CPU utilization?
--
Brendan Conoboy / Red Hat, Inc. / blc@redhat.com
On Mon, Jun 18, 2007 at 10:28:38AM -0700, david@lang.hm wrote:
quoted
I plan to test the different configurations.
however, if I was saturating the bus with the reconstruct how can I fire
off a dd if=/dev/zero of=/mnt/test and get ~45M/sec whild only slowing the
reconstruct to ~4M/sec?
I'm putting 10x as much data through the bus at that point, it would seem
to proove that it's not the bus that's saturated.
dd 45MB/s from the raid sounds reasonable.
If you have 45 drives, doing a resync of raid5 or radi6 should probably
involve reading all the disks, and writing new parity data to one drive.
So if you are writing 5MB/s, then you are reading 44*5MB/s from the
other drives, which is 220MB/s. If your resync drops to 4MB/s when
doing dd, then you have 44*4MB/s which is 176MB/s or 44MB/s less read
capacity, which surprisingly seems to match the dd speed you are
getting. Seems like you are indeed very much saturating a bus
somewhere. The numbers certainly agree with that theory.
What kind of setup is the drives connected to?
simple ultra-wide SCSI to a single controller.
I didn't realize that the rate reported by /proc/mdstat was the write
speed that was takeing place, I thought it was the total data rate (reads
+ writes). the next time this message gets changed it would be a good
thing to clarify this.
David Lang
I plan to test the different configurations.
however, if I was saturating the bus with the reconstruct how can I fire
off a dd if=/dev/zero of=/mnt/test and get ~45M/sec whild only slowing the
reconstruct to ~4M/sec?
I'm putting 10x as much data through the bus at that point, it would seem
to proove that it's not the bus that's saturated.
I am unconvinced. If you take ~1MB/s for each active drive, add in SCSI
overhead, 45M/sec seems reasonable. Have you look at a running iostat while
all this is going on? Try it out- add up the kb/s from each drive and see
how close you are to your maximum theoretical IO.
I didn't try iostat, I did look at vmstat, and there the numbers look even
worse, the bo column is ~500 for the resync by itself, but with the DD
it's ~50,000. when I get access to the box again I'll try iostat to get
more details
Also, how's your CPU utilization?
~30% of one cpu for the raid 6 thread, ~5% of one cpu for the resync
thread
David Lang
From: Lennart Sorensen <hidden> Date: 2007-06-18 18:33:36
On Mon, Jun 18, 2007 at 11:12:45AM -0700, david@lang.hm wrote:
simple ultra-wide SCSI to a single controller.
Hmm, isn't ultra-wide limited to 40MB/s? Is it Ultra320 wide? That
could do a lot more, and 220MB/s sounds plausable for 320 scsi.
I didn't realize that the rate reported by /proc/mdstat was the write
speed that was takeing place, I thought it was the total data rate (reads
+ writes). the next time this message gets changed it would be a good
thing to clarify this.
Well I suppose itcould make sense to show rate of rebuild which you can
then compare against the total size of tha raid, or you can have rate of
write, which you then compare against the size of the drive being
synced. Certainly I would expect much higer speeds if it was the
overall raid size, while the numbers seem pretty reasonable as a write
speed. 4MB/s would take for ever if it was the overall raid resync
speed. I usually see SATA raid1 resync at 50 to 60MB/s or so, which
matches the read and write speeds of the drives in the raid.
--
Len Sorensen
On Mon, Jun 18, 2007 at 11:12:45AM -0700, david@lang.hm wrote:
quoted
simple ultra-wide SCSI to a single controller.
Hmm, isn't ultra-wide limited to 40MB/s? Is it Ultra320 wide? That
could do a lot more, and 220MB/s sounds plausable for 320 scsi.
yes, sorry, ultra 320 wide.
quoted
I didn't realize that the rate reported by /proc/mdstat was the write
speed that was takeing place, I thought it was the total data rate (reads
+ writes). the next time this message gets changed it would be a good
thing to clarify this.
Well I suppose itcould make sense to show rate of rebuild which you can
then compare against the total size of tha raid, or you can have rate of
write, which you then compare against the size of the drive being
synced. Certainly I would expect much higer speeds if it was the
overall raid size, while the numbers seem pretty reasonable as a write
speed. 4MB/s would take for ever if it was the overall raid resync
speed. I usually see SATA raid1 resync at 50 to 60MB/s or so, which
matches the read and write speeds of the drives in the raid.
as I read it right now what happens is the worst of the options, you show
the total size of the array for the amount of work that needs to be done,
but then show only the write speed for the rate pf progress being made
through the job.
total rebuild time was estimated at ~3200 min
David Lang
one channel, 2 OS drives plus the 45 drives in the array.
yes I realize that there will be bottlenecks with this, the large capacity
is to handle longer history (it's going to be a 30TB circular buffer being
fed by a pair of OC-12 links)
it appears that my big mistake was not understanding what /proc/mdstat is
telling me.
David Lang
From: Wakko Warner <hidden> Date: 2007-06-18 21:51:26
david@lang.hm wrote:
On Mon, 18 Jun 2007, Brendan Conoboy wrote:
quoted
david@lang.hm wrote:
quoted
yes, sorry, ultra 320 wide.
Exactly how many channels and drives?
one channel, 2 OS drives plus the 45 drives in the array.
Given that the drives only have 4 ID bits, how can you have 47 drives on 1
cable? You'd need a minimum of 3 channels for 47 drives. Do you have some
sort of external box that holds X number of drives and only uses a single
ID?
--
Lab tests show that use of micro$oft causes cancer in lab animals
Got Gas???
one channel, 2 OS drives plus the 45 drives in the array.
Given that the drives only have 4 ID bits, how can you have 47 drives on 1
cable? You'd need a minimum of 3 channels for 47 drives. Do you have some
sort of external box that holds X number of drives and only uses a single
ID?
yes, I'm useing promise drive shelves, I have them configured to export
the 15 drives as 15 LUNs on a single ID.
I'm going to be useing this as a huge circular buffer that will just be
overwritten eventually 99% of the time, but once in a while I will need to
go back into the buffer and extract and process the data.
David Lang
yes, I'm useing promise drive shelves, I have them configured to export
the 15 drives as 15 LUNs on a single ID.
Well, that would account for it. Your bus is very, very saturated. If
all your drives are active, you can't get more than ~7MB/s per disk
under perfect conditions.
--
Brendan Conoboy / Red Hat, Inc. / blc@redhat.com
one channel, 2 OS drives plus the 45 drives in the array.
Huh? You can only have 16 devices on a scsi bus, counting the host
adapter. And I don't think you can even manage that much reliably with
the newer higher speed versions, at least not without some very special
cables.
yes I realize that there will be bottlenecks with this, the large
capacity is to handle longer history (it's going to be a 30TB circular
buffer being fed by a pair of OC-12 links)
Building one of those nice packet sniffers for the NSA to install on
AT&Ts network eh? ;)
one channel, 2 OS drives plus the 45 drives in the array.
Huh? You can only have 16 devices on a scsi bus, counting the host adapter.
And I don't think you can even manage that much reliably with the newer
higher speed versions, at least not without some very special cables.
6 devices on the bus (2 OS drives, 3 promise drive shelves, controller
card)
quoted
yes I realize that there will be bottlenecks with this, the large capacity
is to handle longer history (it's going to be a 30TB circular buffer being
fed by a pair of OC-12 links)
Building one of those nice packet sniffers for the NSA to install on AT&Ts
network eh? ;)
just for going back in time to track hacker actions at a bank.
I'm hopeing that once I figure out the drives the rest of the software
will basicly boil down to tcpdump with the right options to write to a
circular buffer of files.
David Lang
From: Lennart Sorensen <hidden> Date: 2007-06-19 20:11:39
On Mon, Jun 18, 2007 at 02:56:10PM -0700, david@lang.hm wrote:
yes, I'm useing promise drive shelves, I have them configured to export
the 15 drives as 15 LUNs on a single ID.
I'm going to be useing this as a huge circular buffer that will just be
overwritten eventually 99% of the time, but once in a while I will need to
go back into the buffer and extract and process the data.
I would guess that if you ran 15 drives per channel on 3 different
channels, you would resync in 1/3 the time. Well unless you end up
saturating the PCI bus instead.
hardware raid of course has an advantage there in that it doesn't have
to go across the bus to do the work (although if you put 45 drives on
one scsi channel on hardware raid, it will still be limited).
--
Len Sorensen
On Mon, Jun 18, 2007 at 02:56:10PM -0700, david@lang.hm wrote:
quoted
yes, I'm useing promise drive shelves, I have them configured to export
the 15 drives as 15 LUNs on a single ID.
I'm going to be useing this as a huge circular buffer that will just be
overwritten eventually 99% of the time, but once in a while I will need to
go back into the buffer and extract and process the data.
I would guess that if you ran 15 drives per channel on 3 different
channels, you would resync in 1/3 the time. Well unless you end up
saturating the PCI bus instead.
hardware raid of course has an advantage there in that it doesn't have
to go across the bus to do the work (although if you put 45 drives on
one scsi channel on hardware raid, it will still be limited).
I fully realize that the channel will be the bottleneck, I just didn't
understand what /proc/mdstat was telling me. I thought that it was telling
me that the resync was processing 5M/sec, not that it was writing 5M/sec
on each of the two parity locations.
David Lang