Re: [PATCH] blk-mq: plug request for shared sbitmap

On 5/18/21 2:37 PM, John Garry wrote:

On 18/05/2021 12:54, Hannes Reinecke wrote:

quoted

quoted

In yanhui's virt workload(qemu, libaio, dio, high queue depth, single
job), the patch can improve throughput much(>50%) when running
sequential write(dio, libaio, 16 jobs) to XFS. And it is observed that
IO merge is recovered to level of disabling host tags.

I've found a good testcase for this by using null_blk:

modprobe null_blk submit_queues=32 queue_mode=2 irqmode=0 bs=4096
hw_queue_depth=2048 completion_nsec=1 nr_devices=4 shared_tags=1
shared_tag_bitmap=1

and using a simple fio job with libaio and rw=read and numjobs=32 will
do the trick:

[nullb]
rw=read
size=16g
ioengine=libaio
direct=1
buffered=0
group_reporting=1
bs=4096
numjobs=32

(of course, the 'numjobs' and 'submit_queues' parameter would need to be
adjusted to your machine).
Omitting the 'shared_tag_bitmap' module parameter would yield around 12M
iops; adding it would see a rather dramatic drop to 172k iops.
With this patchset it's back to the expected iops value.

I will consider running this test myself, but do you mean that you get
~12M with shared_tag_bitmap=1 and this patch?

I would always expect shared_tag_bitmap=1 to give a drop for null_blk,
as we move from per-submit queue driver tag to all submit queues sharing
the same driver tagset.

There is a slight drop (less than 1M iops), but that is about to be
expected; however, as there is a rather high fluctuation in the numbers
I didn't include it.

But nothing as dramatic as without this patchset :-)

And I am not sure if you are fixing the IO sched from default.

No, I don't; I'm using whatever it's there (ie mq-deadline).

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		        Kernel Storage Architect
hare@suse.de			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 Nürnberg
GF: F. Imendörffer, HRB 36809 (AG Nürnberg)