Re: Slow initial resync in RAID6 with 36 SAS drives
flat view
From: Song Liu <song@kernel.org>
Date: 2021-09-03 00:58:20
Subsystem:
block layer, the rest · Maintainers:
Jens Axboe, Linus Torvalds
On Tue, Aug 31, 2021 at 10:19 PM Song Liu [off-list ref] wrote:
On Wed, Aug 25, 2021 at 3:06 AM Marcin Wanat [off-list ref] wrote:quoted
On Thu, Aug 19, 2021 at 11:28 AM Marcin Wanat [off-list ref] wrote:quoted
Sorry, this will be a long email with everything I find to be relevant. I have a mdraid6 array with 36 hdd SAS drives each able to doquoted
200MB/s, but I am unable to get more than 38MB/s resync speed on afast system (48cores/96GB ram) with no other load.I have done a bit more research on 24 NVMe drives server and found that resync speed bottleneck affect RAID6 with >16 drives:Sorry for the late response. This is interesting behavior. I don't really know why this is the case at the moment. Let me try to reproduce this first. Thanks, Song
The issue is caused by blk_plug logic. Something like the following should fix it. Marcin, could you please give it a try? Thanks, Song
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 2c4ac51e54eba..fdb945be85753 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c@@ -2251,7 +2251,7 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio) else last = list_entry_rq(plug->mq_list.prev); - if (request_count >= BLK_MAX_REQUEST_COUNT || (last && + if (request_count >= blk_plug_max_rq_count(plug) || (last && blk_rq_bytes(last) >= BLK_PLUG_FLUSH_SIZE)) { blk_flush_plug_list(plug, false); trace_block_plug(q);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index b5c033cf5f26f..2e3c07e959c14 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h@@ -1239,6 +1239,13 @@ extern void blk_start_plug(struct blk_plug *); extern void blk_finish_plug(struct blk_plug *); extern void blk_flush_plug_list(struct blk_plug *, bool); +static inline unsigned short blk_plug_max_rq_count(struct blk_plug *plug) +{ + if (plug->multiple_queues) + return BLK_MAX_REQUEST_COUNT * 4; + return BLK_MAX_REQUEST_COUNT; +} + static inline void blk_flush_plug(struct task_struct *tsk) { struct blk_plug *plug = tsk->plug;