On 2012-07-16 11:29 Shaohua Li [off-list ref] Wrote:

2012/7/15 majianpeng [off-list ref]:

quoted

Create a raid5 using four disk and the chunksize is 512K.
Test command is: dd if=/dev/zero of=/dev/md0 bs=1536K count=90000 oflag=direct

In RHEL6(kernel 2.6.32):speed about 240MB/s
In 3.5.0-rc5:speed about 77MB/S
Add two patch in 3.5.0-rc5, speed about 200MB/S.

So the performance of odirect-wrirte for block-deivce was obvious reduced.
PATCH 1/2: Add blk_plug function for odirect-write block-device
PATCH 2/2: Remove REQ_SYNC for odirect-write in raid456.

PATCH 2/2 maybe not correct because it alse for odirect-write for regular file.
Jianpeng Ma (2):
  fs/block-dev.c:fix performance regression in O_DIRECT writes to
    md block devices.

In raid5, all requests are submitted by raid5d thread, which already has
plug. Why doesn't it work?

No. the purpose of two patch is to reduce the read operation when write which was not full-write.
I tested in RHEL6.The read operation is zero.But in 3.5.0-rc5, the read operaiton may equal to write-operation.
And i used the bs was 1536k(3*512k(chunk-size)).

quoted

  raid5: For write performance, remove REQ_SYNC when write was odirect.

REQ_SYNC only impacts CFQ, this sounds not reasonable. So the disks
are using CFQ ioscheduler. Can you check if you can see the same issue
with deadline?

I tested and the result is the same like cfq.
But in RHEL6, the ioscheduler is also cfq.

Let me guess, without REQ_SYNC, read will get higher priority against write
in CFQ, so in this case, write gets delayed, and maybe get better write
request merge. And now with REQ_SYNC, read and write has the same
priority, there is less request merge.

Thanks,
Shaohua

For harddisk,the read for not full-write will remarkly reduce the performance.
So the first it to make write full-write as posible.