On Fri, Dec 06, 2013 at 08:57:12AM +1100, NeilBrown wrote:

On Thu, 5 Dec 2013 20:24:54 +0100 Piergiorgio Sartor
[off-list ref] wrote:

quoted

Hi all,

I've a system, with an LSI 2308 SAS controller
and 5 2.5" HDD attached.
Each HDD can do around 100MB/sec read/write.
This was tested will all HDDs in parallel, to
make sure the controller can sustain them.
Single disk has same performance.

I was testing RAID 0/5/6 perfomances and I found
something I could not clearly understand.

The test was done with "dd", I wanted to know the
maximum possible performance.
Specifically, for reading:

dd if=/dev/md127 of=/dev/null bs=4k

For writing:

dd if=/dev/zero of=/dev/md127 bs=4k conv=fdatasync

Note than large block size did not change the
results. I guess the page size is quite optimal.

I tested each RAID with 4 and 5 HDDs, with chunk
size of 512k, 64k and 16k.
The "stripe_cache_size" was set to the max 32768.

The results were observed with "iostat -k 5",
taking care to consider variations and ramp up.

The table, with MB/sec, the number are the HDDs
the "r" is read, "w" is write:

chunk RAID 4r  4w  5r  5w
512k   0   400 400 500 500
512k   5   260 300 360 400
512k   6    55 180 100 290

 64k   0   400 400 440 500
 64k   5   150 300 160 400
 64k   6   100 180 140 290

 16k   0   380 400 350 500
 16k   5   100 300 130 390
 16k   6    80 180 100 290

Now, RAID-0/5 seem to perform as expected,
depending on the number of HDDs. Expecially
with large chunk size.
Write performances are not a problem, even
if those are CPU intensive, with parity RAID.
RAID-0/5 do not react well with small chunk.
RAID-6, on the other hand, seems to have an
idea of its own.
First of all, it does not seem to respect
proportionality. I would think a 4 HDDs
RAID-6 should more or less read as fast as
2 HDDs. I can understand some loss, due to
the parity skip, but not so much. In fact it
improves with smaller chunk.
With 5 HDDs, I would expect something better
than 100MB/sec.

Any idea on this? Am I doing something wrong?
Some suggestion on tuning something in order
to try to improve RAID-6?

Thanks,

bye,

Does look strange.
First thing I would check is the read-ahead size.
md sets it for you but might be messing up some how.
Have a look at 
   /sys/block/mdX/bdi/read_ahead_kb
for each configuration and see if making it some uniform large number has any
effect.

Hi Neil,

thanks for the hint, I knew I needed _the_ expert :-)

Using a chunk of 64k (best of the table above), with
5 HDDs RAID-6, the default read_ahead_kb is 384.
I tried to increase it, with following improvement:

read_ahead_kb 5r
1024    -->   200MB/sec
4096    -->   300MB/sec
8192    -->   310MB/sec
32768   -->   310MB/sec

So, it seems that between 4k and 8k the max is reached,
which is somehow what I would expect for a 5 HDDs RAID-6.

I'll try (tomorrow) with different chunk to see what changes.

In any case, 384 seems a bit too little. Maybe 5 HDDs are
not a real RAID-6 use case, I do not know.

Thanks again,

bye,

pg

-- 

piergiorgio