On Thu, Oct 22, 2020 at 2:56 AM Finlayson, James M CIV (USA)
[off-list ref] wrote:

All,
I'm working on creating raid5 or raid6 arrays of 800K IOP nvme drives.   Each of the drives performs well with a queue depth of 128 and I set to 1023 if allowed.   In order for me to try to max out the queue depth on each RAID member, so I'd like to set the sysfs nr_requests on the md device to something greater than 128, like #raid members * 128.   Even though /sys/block/md127/queue/nr_requests is mode 644, when I try to change nr_requests in any way as root, I get write error: invalid argument.  When I'm hitting the md device with random reads, my nvme drives are 100% utilized, but only doing 160K IOPS because they have no queue depth.

Am I doing something silly?

It only works for blk-mq block devices. MD is not blk-mq.

You can exchange simplicity for performance: instead of creating one
RAID-5/6 array you can partition drives in N equal sized partitions,
create N RAID-5/6 arrays using one partition from every disk, then
stripe them into top-level RAID-0. So that would be RAID-5+0 (or 6+0).

It is awful, but simulates multiqueue and performs better in parallel
loads. Especially for writes (on RAID-5/6).


-- 
wbr, Vitaly