Performance of a software raid 5

Hi,

[first of all i'm not really sure if i'm right here. If this is the
wrong place then please just
tell me what to rtfm or where to post]

i'm currently trying to create a raid 5 out of three 1 TB hdd. For now
there is one hdd missing so i get 3 TB of usable space.

One hdd is connected to
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)

the other two to
04:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA
Raid II Controller (rev 01)

The CPU is a AMD X2 4200+ and the system has 2 GB RAM.

The performance of the array is underwhelming.
time dd if=/dev/zero of=big_file bs=4096 count=2560000
10485760000 bytes (10 GB) copied, 187.691 s, 55.9 MB/s
dd if=/dev/zero of=big_file bs=4096 count=2560000 0.70s user 26.05s
system 14% cpu 3:08.12 total

time dd if=big_file of=/dev/null bs=4096 count=2560000
10485760000 bytes (10 GB) copied, 297.345 s, 35.3 MB/s
dd if=big_file of=/dev/null bs=4096 count=2560000 0.50s user 10.60s
system 3% cpu 4:57.35 total

So i get a write performance of 55 MB/s and a read speed of 35 MB/s. The hdd

 Model=SAMSUNG HD103UJ                         , FwRev=1AA01113,
SerialNo=S13PJDWS250990
 Config={ Fixed }
 RawCHS=16383/16/63, TrkSize=34902, SectSize=554, ECCbytes=4
 BuffType=DualPortCache, BuffSize=32767kB, MaxMultSect=16, MultSect=?16?
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=1953525168
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2

56 more lines

On 20/04/2009 18:12, Johannes Segitz wrote:

i'm currently trying to create a raid 5 out of three 1 TB hdd. For now
there is one hdd missing so i get 3 TB of usable space.

[...]

Stride and stripe-width will be correct when i add another two hdd of
which one will carry data. Can someone please give me a hint why i
could get such bad performance especially while reading?

I would have thought it's because you're running in degraded mode and one in 3 sectors is having to be regenerated from the parity. It still seems a bit slow, though.

Here I have a 3-disc RAID-5 of similar drives:

# hdparm -i /dev/sda
/dev/sda:

 Model=SAMSUNG HD103UJ                         , FwRev=1AA01112, SerialNo=S1PVJ1CQ602164
 Config={ Fixed }
 RawCHS=16383/16/63, TrkSize=34902, SectSize=554, ECCbytes=4
 BuffType=DualPortCache, BuffSize=32767kB, MaxMultSect=16, MultSect=?0?
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2
 AdvancedPM=yes: disabled (255) WriteCache=enabled
 Drive conforms to: unknown:  ATA/ATAPI-3 ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6 ATA/ATAPI-7

# mdadm --detail /dev/md1
/dev/md1:
        Version : 00.90.03
  Creation Time : Mon Jul 28 15:49:09 2008
     Raid Level : raid5
     Array Size : 1953310720 (1862.82 GiB 2000.19 GB)
  Used Dev Size : 976655360 (931.41 GiB 1000.10 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 1
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Tue Apr 21 00:35:26 2009
          State : active
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

38 more lines

On Tue, Apr 21, 2009 at 1:46 AM, John Robinson
[off-list ref] wrote:

I would have thought it's because you're running in degraded mode and one in
3 sectors is having to be regenerated from the parity. It still seems a bit
slow, though.

i don't think that that is a problem. The data is there without
redundancy so i can't see
how there would be the need to calculate anything

I'm not quite sure whether I should be disappointed at my writes being so
slow. Certainly there's a lot of rattling during writing, which probably
indicates lots of seeks to write ext3's journal. But reads are roughly what
I expected, at about three times the single-disc throughput.

200 MB/s reads would be nice but i expected quite a bit more speed
writing too. I know
that you don't get an O(n) speedup but falling behind the normal performance of
a single drive seems not okay.

Johannes
Btw.:The controller is on PCI-E so no bottleneck there

On 21/04/2009 00:46, I wrote:
[...]

# time dd if=/dev/zero of=big_file bs=4096 count=2560000
2560000+0 records in
2560000+0 records out
10485760000 bytes (10 GB) copied, 264.448 seconds, 39.7 MB/s

[...]

I'm not quite sure whether I should be disappointed at my writes being so slow. Certainly there's a lot of rattling during writing, which probably indicates lots of seeks to write ext3's journal.

No, that's not it. Using a scratch logical volume over the md RAID-5 isn't much better:

# time dd if=/dev/zero of=/dev/mapper/vg0-scratch bs=4096 count=2560000
2560000+0 records in
2560000+0 records out
10485760000 bytes (10 GB) copied, 230.036 seconds, 45.6 MB/s

real    3m50.077s
user    0m1.608s
sys     0m11.097s

It still rattles a lot, suggesting a lot of seeking. Now if I turn off the bitmap and try again:
# mdadm --grow /dev/md1 --bitmap=none
# time dd if=/dev/zero of=/dev/mapper/vg0-scratch bs=4096 count=2560000
2560000+0 records in
2560000+0 records out
10485760000 bytes (10 GB) copied, 110.17 seconds, 95.2 MB/s

real    1m50.346s
user    0m1.900s
sys     0m13.537s

That's more like it, and no more rattling. Can I tune settings for the internal bitmap, or is this something which will have improved anyway since my kernel (2.6.18-128.1.6.el5.centos.plusxen so essentially a prominent North American Enterprise Linux vendor's EL5 codebase for md/raid5)? I mean, I do want the bitmap, but I hadn't realised it was quite so expensive (not that it matters much in this particular application).

Cheers,

John.

On 21/04/2009 01:10, Johannes Segitz wrote:

On Tue, Apr 21, 2009 at 1:46 AM, John Robinson
[off-list ref] wrote:

quoted

I would have thought it's because you're running in degraded mode and one in
3 sectors is having to be regenerated from the parity. It still seems a bit
slow, though.

i don't think that that is a problem. The data is there without
redundancy so i can't see
how there would be the need to calculate anything

There's no redundancy but it's still the RAID-5 4-disc layout with 3 data and 1 parity, the parity on a different disc in each stripe. In your case with a missing disc, for 3 stripes in 4 you have 2 data and 1 parity. Of course the parity is having to be calculated when you're writing, and whatever would be written to your missing disc is being discarded.

On the other hand if you were using RAID-0 over 3 discs there would be no need to calculate anything.

Cheers,

John.

On Tue, Apr 21, 2009 at 2:52 AM, John Robinson
[off-list ref] wrote:

There's no redundancy but it's still the RAID-5 4-disc layout with 3 data
and 1 parity, the parity on a different disc in each stripe. In your case
with a missing disc, for 3 stripes in 4 you have 2 data and 1 parity. Of
course the parity is having to be calculated when you're writing, and
whatever would be written to your missing disc is being discarded.

you're right, i didn't think of that. But calculating an xor isn't really
a big deal (especially with the aes on top of it) so i still can't see why
it's so slow

Johannes

On 21/04/2009 02:05, Johannes Segitz wrote:

On Tue, Apr 21, 2009 at 2:52 AM, John Robinson
[off-list ref] wrote:

quoted

There's no redundancy but it's still the RAID-5 4-disc layout with 3 data
and 1 parity, the parity on a different disc in each stripe. In your case
with a missing disc, for 3 stripes in 4 you have 2 data and 1 parity. Of
course the parity is having to be calculated when you're writing, and
whatever would be written to your missing disc is being discarded.

you're right, i didn't think of that. But calculating an xor isn't really
a big deal (especially with the aes on top of it) so i still can't see why
it's so slow

No nor can I, especially since your `time` output shows a very modest amount of system time; it may be worth trying fewer layers (i.e. no encryption and/or no filesystem) to eliminate them, or monitoring with other tools like iostat to see if you can get to the bottom of it.

Cheers,

John.

On Tue, April 21, 2009 11:05 am, Johannes Segitz wrote:

On Tue, Apr 21, 2009 at 2:52 AM, John Robinson
[off-list ref] wrote:

quoted

There's no redundancy but it's still the RAID-5 4-disc layout with 3
data
and 1 parity, the parity on a different disc in each stripe. In your
case
with a missing disc, for 3 stripes in 4 you have 2 data and 1 parity. Of
course the parity is having to be calculated when you're writing, and
whatever would be written to your missing disc is being discarded.

you're right, i didn't think of that. But calculating an xor isn't really
a big deal (especially with the aes on top of it) so i still can't see why
it's so slow

Large sequential writes to a degraded RAID5 will be the same speed as to a
non-degraded RAID5.  Smaller random write can still be slower as the
amount of pre-reading can increase.

Reads from a degraded raid5 will be slower not because of the XOR, but
because of needing to read all that extra data to feed in to the XOR.

Have you done any testing without the crypto layer to see what effect
that has?

Can I suggest:

  for d in /dev/sd[gjk]1 /dev/md6 /dev/mapper/data bigfile
  do
    dd if=$d of=/dev/null bs=1M count=100
  done

and report the times.

NeilBrown

On Tue, April 21, 2009 10:44 am, John Robinson wrote:

That's more like it, and no more rattling. Can I tune settings for the
internal bitmap, or is this something which will have improved anyway
since my kernel (2.6.18-128.1.6.el5.centos.plusxen so essentially a
prominent North American Enterprise Linux vendor's EL5 codebase for
md/raid5)? I mean, I do want the bitmap, but I hadn't realised it was
quite so expensive (not that it matters much in this particular
application).

I don't think newer kernels make any different to bitmap related
performance, though there might be some general raid5 improvements since
then.

There are two tunables for bitmaps.  Chuck size and delay (though the
delay doesn't seem to be in the man page).

Choosing a larger --bitmap-chunk size will require fewer updates to the
bitmap before writes are allowed to proceed.   However a larger bitmap-chunk
size will also increase the amount of work needed after a crash or
re-added device.
Check your current (default) chunk size with "mdadm -X /dev/sdxx" and
create a new bitmap with (say) 16 or 64 times the chunk size.
See if that makes a difference.

The delay tunable sets how quickly bits are removed from the bitmap.
This is done fairly lazily and opportunistically so it isn't likely
to affect throughput directly.  However if you are updates the same
area on disk periodically, and this period is a little bit more than
the timeout, you will get excessive bitmap updates that service little
purpose.
You could try increasing this (to say 30 or 60 seconds), but I doubt
it will have much effect.

Your other option is to put the bitmap in a file on some other device.
If you have a device this is rarely used (maybe your root filesystem)
you can create an external bitmap on there.  You would want to be
sure that the device storing the bitmap is always going to be
available when you start the array that the bitmap belongs to.

Maybe you could create an external bitmap in a tmpfs.... but then
it wouldn't survive a crash and so has little value.  It would be
fast though :-)

NeilBrown

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, Apr 21, 2009 at 3:19 AM, NeilBrown [off-list ref] wrote:

Have you done any testing without the crypto layer to see what effect
that has?

Can I suggest:

 for d in /dev/sd[gjk]1 /dev/md6 /dev/mapper/data bigfile
 do
   dd if=$d of=/dev/null bs=1M count=100
 done

and report the times.

tested it with 1gb instead of 100 mb

sdg
1048576000 bytes (1.0 GB) copied, 9.89311 s, 106 MB/s
sdj
1048576000 bytes (1.0 GB) copied, 10.094 s, 104 MB/s
sdk
1048576000 bytes (1.0 GB) copied, 8.53513 s, 123 MB/s
/dev/md6
1048576000 bytes (1.0 GB) copied, 11.4741 s, 91.4 MB/s
/dev/mapper/data
1048576000 bytes (1.0 GB) copied, 34.4544 s, 30.4 MB/s
bigfile
1048576000 bytes (1.0 GB) copied, 26.6532 s, 39.3 MB/s

so the crypto indeed slows it down (and i'm surprised that it's that
bad because i've read
it's not a big hit on current CPUs and the X2 isn't new but not that
old) but still read speed
from md6 is worse than from one drive alone

Johannes
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 21/04/2009 02:33, NeilBrown wrote:
[...]

Choosing a larger --bitmap-chunk size will require fewer updates to the
bitmap before writes are allowed to proceed.   However a larger bitmap-chunk
size will also increase the amount of work needed after a crash or
re-added device.

Ah, from my reading of the mdadm man page I thought you could only specify the chunk size when using an external bitmap:
--bitmap-chunk=
       Set  the  chunksize  of the bitmap. Each bit corresponds to that
       many Kilobytes of storage.  When using a file based bitmap,  the
       default  is  to  use  the  smallest  size that is at-least 4 and
       requires no more than  2^21  chunks.   When  using  an  internal
       bitmap,  the  chunksize is automatically determined to make best
       use of available space.

Check your current (default) chunk size with "mdadm -X /dev/sdxx" and
create a new bitmap with (say) 16 or 64 times the chunk size.
See if that makes a difference.

It certainly does. Upping it from 2M to 32M gets me from 45MB/s to 81MB/s on the scratch LV, and there's noticeably less seek noise.

Eeek! Trying to `mdadm --grow /dev/md1 --bitmap=none` from my large chunk size caused a reboot! There's nothing in the log, and I didn't see the console. I still have my 32M chunksize but I don't want to try that again in a hurry :-)

[...]

Your other option is to put the bitmap in a file on some other device.
If you have a device this is rarely used (maybe your root filesystem)

Can't do that, my root filesystem is on the RAID-5, and part of the reason for wanting the bitmap is because the md can't be stopped while shutting down, so it was always wanting to resync at startup, which is rather tedious.

Maybe you could create an external bitmap in a tmpfs.... but then
it wouldn't survive a crash and so has little value.  It would be
fast though :-)

"Ooh, virtual memory! Now I can have a really big RAM disc!"

Now, it's time I checked my discs to see if I've lost data in that crash. `mdadm -X` is stuck saying there are 10 dirty chunks.

Cheers,

John.

NeilBrown wrote:

On Tue, April 21, 2009 10:44 am, John Robinson wrote:
  

quoted

That's more like it, and no more rattling. Can I tune settings for the
internal bitmap, or is this something which will have improved anyway
since my kernel (2.6.18-128.1.6.el5.centos.plusxen so essentially a
prominent North American Enterprise Linux vendor's EL5 codebase for
md/raid5)? I mean, I do want the bitmap, but I hadn't realised it was
quite so expensive (not that it matters much in this particular
application).

    

I don't think newer kernels make any different to bitmap related
performance, though there might be some general raid5 improvements since
then.

There are two tunables for bitmaps.  Chuck size and delay (though the
delay doesn't seem to be in the man page).
  

It isn't in the man page, and
 mdadm --help | egrep 'bitmap|delay'
comes up empty as well. So then I looked at the strings:
 strings $(type -p mdadm) | less
and I not only found it, but found that you have a vast bunch of duplicate strings, including some which are saying the same thing but expressed in several ways. If I might quote my offspring, "That's ugly, dude!"

Anyway, setting "--delay N" (sec) does exactly what you predicted, not much.

-- 
bill davidsen [off-list ref]
 CTO TMR Associates, Inc

"You are disgraced professional losers. And by the way, give us our money back."
   - Representative Earl Pomeroy,  Democrat of North Dakota
on the A.I.G. executives who were paid bonuses  after a federal bailout.

Johannes Segitz wrote:

On Tue, Apr 21, 2009 at 3:19 AM, NeilBrown [off-list ref] wrote:

quoted

Have you done any testing without the crypto layer to see what effect
that has?

Can I suggest:

 for d in /dev/sd[gjk]1 /dev/md6 /dev/mapper/data bigfile
 do
   dd if=$d of=/dev/null bs=1M count=100
 done

and report the times.

tested it with 1gb instead of 100 mb

sdg
1048576000 bytes (1.0 GB) copied, 9.89311 s, 106 MB/s
sdj
1048576000 bytes (1.0 GB) copied, 10.094 s, 104 MB/s
sdk
1048576000 bytes (1.0 GB) copied, 8.53513 s, 123 MB/s
/dev/md6
1048576000 bytes (1.0 GB) copied, 11.4741 s, 91.4 MB/s
/dev/mapper/data
1048576000 bytes (1.0 GB) copied, 34.4544 s, 30.4 MB/s
bigfile
1048576000 bytes (1.0 GB) copied, 26.6532 s, 39.3 MB/s

so the crypto indeed slows it down (and i'm surprised that it's that
bad because i've read
it's not a big hit on current CPUs and the X2 isn't new but not that
old) but still read speed
from md6 is worse than from one drive alone

If it helps, some recent dd benchmarks I did indicate that twofish is
about 25% faster than aes on my Athlon64.

Athlon64 3400+ 2.4 GHz, 64-bit Linux 2.6.28.2

Both aes and twofish are using the asm implementations according to
/proc/crypto.

All numbers are in MB/s; average of three tests for a 512MB dd
read/write to the encrypted device.

                                 read       write
aes                              69.4        61.0
twofish                          86.8        76.6
aes-cbc-essiv:sha256             65.1        56.3
twofish-cbc-essiv:sha256         82.6        73.5


-Corey

Corey Hickey wrote:

Johannes Segitz wrote:
  

quoted

On Tue, Apr 21, 2009 at 3:19 AM, NeilBrown [off-list ref] wrote:
    

quoted

Have you done any testing without the crypto layer to see what effect
that has?

Can I suggest:

 for d in /dev/sd[gjk]1 /dev/md6 /dev/mapper/data bigfile
 do
   dd if=$d of=/dev/null bs=1M count=100
 done

and report the times.
      

tested it with 1gb instead of 100 mb

sdg
1048576000 bytes (1.0 GB) copied, 9.89311 s, 106 MB/s
sdj
1048576000 bytes (1.0 GB) copied, 10.094 s, 104 MB/s
sdk
1048576000 bytes (1.0 GB) copied, 8.53513 s, 123 MB/s
/dev/md6
1048576000 bytes (1.0 GB) copied, 11.4741 s, 91.4 MB/s
/dev/mapper/data
1048576000 bytes (1.0 GB) copied, 34.4544 s, 30.4 MB/s
bigfile
1048576000 bytes (1.0 GB) copied, 26.6532 s, 39.3 MB/s

so the crypto indeed slows it down (and i'm surprised that it's that
bad because i've read
it's not a big hit on current CPUs and the X2 isn't new but not that
old) but still read speed
from md6 is worse than from one drive alone
    

If it helps, some recent dd benchmarks I did indicate that twofish is
about 25% faster than aes on my Athlon64.

15 more quoted lines

Good info, but was the CPU maxed or was something else the limiting factor?

-- 
bill davidsen [off-list ref]
 CTO TMR Associates, Inc

"You are disgraced professional losers. And by the way, give us our money back."
   - Representative Earl Pomeroy,  Democrat of North Dakota
on the A.I.G. executives who were paid bonuses  after a federal bailout.

Bill Davidsen wrote:

Corey Hickey wrote:

quoted

Johannes Segitz wrote:
  

quoted

On Tue, Apr 21, 2009 at 3:19 AM, NeilBrown [off-list ref] wrote:
    

quoted

Have you done any testing without the crypto layer to see what effect
that has?

Can I suggest:

 for d in /dev/sd[gjk]1 /dev/md6 /dev/mapper/data bigfile
 do
   dd if=$d of=/dev/null bs=1M count=100
 done

and report the times.
      

tested it with 1gb instead of 100 mb

sdg
1048576000 bytes (1.0 GB) copied, 9.89311 s, 106 MB/s
sdj
1048576000 bytes (1.0 GB) copied, 10.094 s, 104 MB/s
sdk
1048576000 bytes (1.0 GB) copied, 8.53513 s, 123 MB/s
/dev/md6
1048576000 bytes (1.0 GB) copied, 11.4741 s, 91.4 MB/s
/dev/mapper/data
1048576000 bytes (1.0 GB) copied, 34.4544 s, 30.4 MB/s
bigfile
1048576000 bytes (1.0 GB) copied, 26.6532 s, 39.3 MB/s

so the crypto indeed slows it down (and i'm surprised that it's that
bad because i've read
it's not a big hit on current CPUs and the X2 isn't new but not that
old) but still read speed
from md6 is worse than from one drive alone
    

If it helps, some recent dd benchmarks I did indicate that twofish is
about 25% faster than aes on my Athlon64.

14 more quoted lines

   no encryption                    237        131

quoted

Good info, but was the CPU maxed or was something else the limiting factor?

To be honest, I didn't check when I benchmarked, but the underlying
device is much faster. I added the numbers to the table above. This is
for an md RAID-0 of two 1TB Samsung drives. I don't know why the write
speed for the RAID-0 is so much slower, except that it's not md's fault;
writing to the individual drives is slower, too. I would have
investigated more, but, at the time, I really wanted to get my computer
operational again. :)

That might be lowering my encrypted write speeds a bit relative to the
read speeds, but, even if so, I think it would affect the faster of the
two ciphers more than the slower--and twofish still leads by a
significant margin.


Also, to the original poster:
Check which crypto drivers in your kernel have ASM implementations loaded:
$ grep asm /proc/crypto

AES, twofish, and salsa20 are available.


-Corey