md: Speed shrinks with drives number

TL;DR; The more drives are in the array, the slower it is.

I found this when I was debugging NVME array issues, but I was able to 
reduce it to the case with with ram-disks, so no special hardware involved.

It can be reproduced on any machine with reasonable amount of memory 
(16-32Gb). I checked it on Ubuntu Focal (5.4.0-65-generic), Groovy 
(5.8.0-41-generic), Centos 8

For a very fast underlying device (NVME or brd: block ram disk) the more 
drives are added to the array, the slower it become, and it's irrelevant 
of the array type: raid1, 0, 10.

The speed lost is very high: a single ram-disk is yielding 200-250 
kIOPS, the raid0 of 4 ram disks yields less than 150kIOPS, and 100 RAM 
disks in raid0 yields mere 30 kIOPS.

The script to reproduce the issue:


export NUM=100  # or any other number between 1 and 1920.

modprobe brd rd_nr=${NUM} rd_size=10000
mdadm --create /dev/md42 -n ${NUM} -l raid0 --assume-clean /dev/ram* -e 
1.2 --force
fio --name test --ioengine=libaio   --blocksize=4k --iodepth=${NUM} 
--rw=randwrite --time_based   --runtime=5s --ramp_time=1s   --fsync=1 
--direct=1 --disk_util=0 --filename=/dev/md42
mdadm --stop /dev/md42
rmmod brd

My laptop (baseline for ram0 without raid - 197k IOPS)

disks in array - IOPS

1 - 108k
2 - 103k
4 - 99.0k
8 - 89.8k
16 - 75.5k
32 - 52.6k
64 - 34.7k
128 - 20.5k
256 - 11.3k
512 - 5.9k
1024 - 3.4k
1920 - 1.5k

(without --assume-clean it's the same, using --iodepth=1 intead of 
${NUM} gives the same).

I feel that having 1.5k IOPS on RAID0 consisting of 1920 ram-disks is a 
bit slow.

P.S. The real issue is with NVME devices in 10 drives per array, they 
shows even faster decline, but ram-disk make it more visual.