On Thu, 15 Dec 2011 09:00:10 +0800 Wu Fengguang [off-list ref]
wrote:

quoted

I found sometimes one disk hasn't any request inflight, but we can't
send request to the disk, because the scsi host's resource (the queue
depth) is used out, looks we send too many requests from other disks and
leave some disks starved. The resource imbalance in scsi isn't a new
problem, even 3.1 has such issue, so I'd think writeback introduces new
imbalance between the 12 disks. In fact, if I limit disk's queue depth
to 10, in this way the 12 disks will not impact each other in scsi
layer, the performance regression fully disappears for both writeback
and order mode.

I observe similar issue in MD. The default

        q->nr_requests = BLKDEV_MAX_RQ;

is too small for large arrays, and I end up doing

        echo 1280 > /sys/block/md0/queue/nr_requests

in my tests.

And you find this makes a difference?

That is very surprising because md devices don't use requests (and really use
the 'queue' at all) and definitely don't make use of nr_requests.

NeilBrown