Re: Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1
From: Jinpu Wang <jinpu.wang@cloud.ionos.com>
Date: 2019-08-07 06:37:05
Also in:
lkml
On Wed, Aug 7, 2019 at 1:40 AM NeilBrown [off-list ref] wrote:
On Tue, Aug 06 2019, Jinpu Wang wrote:quoted
On Tue, Aug 6, 2019 at 9:54 AM Jinpu Wang [off-list ref] wrote:quoted
On Tue, Aug 6, 2019 at 1:46 AM NeilBrown [off-list ref] wrote:quoted
On Mon, Aug 05 2019, Jinpu Wang wrote:quoted
Hi Neil, For the md higher write IO latency problem, I bisected it to these commits: 4ad23a97 MD: use per-cpu counter for writes_pending 210f7cd percpu-refcount: support synchronous switch to atomic mode. Do you maybe have an idea? How can we fix it?Hmmm.... not sure.Hi Neil, Thanks for reply, detailed result in line.Thanks for the extra testing. ...quoted
[ 105.133299] md md0 in_sync is 0, sb_flags 2, recovery 3, external 0, safemode 0, recovery_cp 524288... ahh - the resync was still happening. That explains why set_in_sync() is being called so often. If you wait for sync to complete (or create the array with --assume-clean) you should see more normal behaviour.
I've updated my tests accordingly, thanks for the hint.
This patch should fix it. I think we can do better but it would be more complex so no suitable for backports to -stable. Once you confirm it works, I'll send it upstream with a Reported-and-Tested-by from you. Thanks, NeilBrown
Thanks a lot, Neil, my quick test show, yes, it fixed the problem for me. I will run more tests to be sure, will report back the test result. Regards, Jack Wang
quoted hunk ↗ jump to hunk
diff --git a/drivers/md/md.c b/drivers/md/md.c index 24638ccedce4..624cf1ac43dc 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c@@ -8900,6 +8900,7 @@ void md_check_recovery(struct mddev *mddev) if (mddev_trylock(mddev)) { int spares = 0; + bool try_set_sync = mddev->safemode != 0; if (!mddev->external && mddev->safemode == 1) mddev->safemode = 0;@@ -8945,7 +8946,7 @@ void md_check_recovery(struct mddev *mddev) } } - if (!mddev->external && !mddev->in_sync) { + if (try_set_sync && !mddev->external && !mddev->in_sync) { spin_lock(&mddev->lock); set_in_sync(mddev); spin_unlock(&mddev->lock);