Re: slow IO on USB media

On Thu, 26 Dec 2019 16:37:06 +0800 Ming Lei wrote:

quoted

On Wed, Dec 25, 2019 at 10:30:57PM -0500, Theodore Y. Ts'o wrote:

quoted

On Thu, Dec 26, 2019 at 10:27:02AM +0800, Ming Lei wrote:

quoted

Maybe we need to be careful for HDD., since the request count in scheduler
queue is double of in-flight request count, and in theory NCQ should only
cover all in-flight 32 requests. I will find a sata HDD., and see if
performance drop can be observed in the similar 'cp' test.

Please try to measure it, but I'd be really surprised if it's
significant with with modern HDD's.

Just find one machine with AHCI SATA, and run the following xfs
overwrite test:

#!/bin/bash
DIR=$1
echo 3 > /proc/sys/vm/drop_caches
fio --readwrite=write --filesize=5g --overwrite=1 --filename=$DIR/fiofile \
        --runtime=60s --time_based --ioengine=psync --direct=0 --bs=4k
		--iodepth=128 --numjobs=2 --group_reporting=1 --name=overwrite

FS is xfs, and disk is LVM over AHCI SATA with NCQ(depth 32), because the
machine is picked up from RH beaker, and it is the only disk in the box.

#lsblk
NAME                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                               8:0    0 931.5G  0 disk
=E2=94=9C=E2=94=80sda1                            8:1    0     1G  0 part /boot
=E2=94=94=E2=94=80sda2                            8:2    0 930.5G  0 part
  =E2=94=9C=E2=94=80rhel_hpe--ml10gen9--01-root 253:0    0    50G  0 lvm  /
  =E2=94=9C=E2=94=80rhel_hpe--ml10gen9--01-swap 253:1    0   3.9G  0 lvm  [SWAP]
  =E2=94=94=E2=94=80rhel_hpe--ml10gen9--01-home 253:2    0 876.6G  0 lvm  /home


kernel: 3a7ea2c483a53fc("scsi: provide mq_ops->busy() hook") which is
the previous commit of f664a3cc17b7 ("scsi: kill off the legacy IO path").

            |scsi_mod.use_blk_mq=N |scsi_mod.use_blk_mq=Y |
-----------------------------------------------------------
throughput: |244MB/s               |169MB/s               |
-----------------------------------------------------------

Similar result can be observed on v5.4 kernel(184MB/s) with same test
steps.

The simple diff makes direct issue of requests take pending requests
also into account and goes the nornal enqueue-and-dequeue path if any
pending requests exist.

Then it sorts requests regardless of the number of hard queues in a
bid to make requests as sequencial as they are. Let's see if the
added sorting cost can make any sense.

--->8---

--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c

@@ -410,6 +410,11 @@ run:
 		blk_mq_run_hw_queue(hctx, async);
 }
 
+static inline bool blk_mq_sched_hctx_has_pending_rq(struct blk_mq_hw_ctx *hctx)
+{
+	return sbitmap_any_bit_set(&hctx->ctx_map);
+}
+
 void blk_mq_sched_insert_requests(struct blk_mq_hw_ctx *hctx,
 				  struct blk_mq_ctx *ctx,
 				  struct list_head *list, bool run_queue_async)

@@ -433,7 +438,8 @@ void blk_mq_sched_insert_requests(struct
 		 * busy in case of 'none' scheduler, and this way may save
 		 * us one extra enqueue & dequeue to sw queue.
 		 */
-		if (!hctx->dispatch_busy && !e && !run_queue_async) {
+		if (!hctx->dispatch_busy && !e && !run_queue_async &&
+		    !blk_mq_sched_hctx_has_pending_rq(hctx)) {
 			blk_mq_try_issue_list_directly(hctx, list);
 			if (list_empty(list))
 				goto out;
--- a/block/blk-mq.c
+++ b/block/blk-mq.c

@@ -1692,7 +1692,7 @@ void blk_mq_flush_plug_list(struct blk_p
 
 	list_splice_init(&plug->mq_list, &list);
 
-	if (plug->rq_count > 2 && plug->multiple_queues)
+	if (plug->rq_count > 1)
 		list_sort(NULL, &list, plug_rq_cmp);
 
 	plug->rq_count = 0;