Re: [PATCH] writeback: fix writeback cache thrashing
From: Fengguang Wu <hidden>
Date: 2013-01-05 03:18:21
Also in:
linux-fsdevel, lkml
Hi Namjae, On Sun, Dec 30, 2012 at 02:59:50PM +0900, Namjae Jeon wrote:
From: Namjae Jeon <redacted>
Consider Process A: huge I/O on sda
doing heavy write operation - dirty memory becomes more
than dirty_background_ratio
on HDD - flusher thread flush-8:0
Consider Process B: small I/O on sdb
doing while [1]; read 1024K + rewrite 1024K + sleep 2sec
on Flash device - flusher thread flush-8:16
As Process A is a heavy dirtier, dirty memory becomes more
than dirty_background_thresh. Due to this, below check becomes
true(checking global_page_state in over_bground_thresh)
for all bdi devices(even for very small dirtied bdi - sdb):
In this case, even small cached data on 'sdb' is forced to flush
and writeback cache thrashing happens.
When we added debug prints inside above 'if' condition and ran
above Process A(heavy dirtier on bdi with flush-8:0) and
Process B(1024K frequent read/rewrite on bdi with flush-8:16)
we got below prints:
[Test setup: ARM dual core CPU, 512 MB RAM]
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56064 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56704 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84720 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 94720 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 384 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 960 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92160 KB[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 256 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 768 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 256 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 320 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB
Yeah, that IO pattern is not good. Perhaps it's 6 small IOs in /one/ second? However that's not quite in line with "sleep 2sec" in your workload description. Note that I assume flush-8:0 works on a hard disk, so each flush-8:0 line indicates roughly 1 second interval elapsed. It would be much more clear if the printk timestamps are turned on (CONFIG_PRINTK_TIME=y).
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92032 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 91968 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 1024 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 64 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 576 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84352 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 192 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 512 KB [over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE = 0 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92608 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92544 KB As mentioned in above log, when global dirty memory > global background_thresh small cached data is also forced to flush by flush-8:16. If removing global background_thresh checking code, we can reduce cache thrashing of frequently used small data. And It will be great if we can reserve a portion of writeback cache using min_ratio.
After applying patch: $ echo 5 > /sys/block/sdb/bdi/min_ratio $ cat /sys/block/sdb/bdi/min_ratio 5
The below log looks all perfect. However the min_ratio setup is a problem. If possible, I'd like the final patch being able to work reasonably well with min_ratio=0 (the system default), too.
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56064 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 56704 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84160 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 96960 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 94080 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93120 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93120 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 91520 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 89600 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93696 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 93696 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 72960 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90624 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90624 KB [over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 90688 KB
quoted hunk ↗ jump to hunk
As mentioned in the above logs, once cache is reserved for Process B, and patch is applied there is less writeback cache thrashing on sdb by frequent forced writeback by flush-8:16 in over_bground_thresh. After all, small cached data will be flushed by periodic writeback once every dirty_writeback_interval. Suggested-by: Wanpeng Li <redacted> Signed-off-by: Namjae Jeon <redacted> Signed-off-by: Vivek Trivedi <redacted> Cc: Fengguang Wu <redacted> Cc: Jan Kara <jack@suse.cz> Cc: Dave Chinner <redacted> --- fs/fs-writeback.c | 4 ---- 1 file changed, 4 deletions(-)diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 310972b..070b773 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c@@ -756,10 +756,6 @@ static bool over_bground_thresh(struct backing_dev_info *bdi) global_dirty_limits(&background_thresh, &dirty_thresh); - if (global_page_state(NR_FILE_DIRTY) + - global_page_state(NR_UNSTABLE_NFS) > background_thresh) - return true; -
That global test should be kept in some form (see Jan's proposal). Because the below per-bdi test can be inaccurate in various ways: - bdi_stat() may have errors up to bdi_stat_error() - bdi_dirty_limit() may be arbitrarily shifted by min_ratio etc. - bdi_dirty_limit() may be totally wrong due to the estimation in bdi_writeout_fraction() is in its initial value 0, or is still trying to catch up with sudden workload changes.
if (bdi_stat(bdi, BDI_RECLAIMABLE) > bdi_dirty_limit(bdi, background_thresh)) return true;
I suspect even removing the global test as in your patch, the above bdi test will still mostly return true for your described workload, due to bdi_dirty_limit() returning a value close to 0, because the writeout fraction of sdb is close to 0. You cleverly avoided this in your test by raising min_ratio to 5. However I'd suggest to test with min_ratio=0 and try solutions that can work well in such default configuration. Thanks, Fengguang -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>