2012/7/15 majianpeng [off-list ref]:

Create a raid5 using four disk and the chunksize is 512K.
Test command is: dd if=/dev/zero of=/dev/md0 bs=1536K count=90000 oflag=direct

In RHEL6(kernel 2.6.32):speed about 240MB/s
In 3.5.0-rc5:speed about 77MB/S
Add two patch in 3.5.0-rc5, speed about 200MB/S.

So the performance of odirect-wrirte for block-deivce was obvious reduced.
PATCH 1/2: Add blk_plug function for odirect-write block-device
PATCH 2/2: Remove REQ_SYNC for odirect-write in raid456.

PATCH 2/2 maybe not correct because it alse for odirect-write for regular file.
Jianpeng Ma (2):
  fs/block-dev.c:fix performance regression in O_DIRECT writes to
    md block devices.

In raid5, all requests are submitted by raid5d thread, which already has
plug. Why doesn't it work?

  raid5: For write performance, remove REQ_SYNC when write was odirect.

REQ_SYNC only impacts CFQ, this sounds not reasonable. So the disks
are using CFQ ioscheduler. Can you check if you can see the same issue
with deadline?

Let me guess, without REQ_SYNC, read will get higher priority against write
in CFQ, so in this case, write gets delayed, and maybe get better write
request merge. And now with REQ_SYNC, read and write has the same
priority, there is less request merge.

Thanks,
Shaohua