Re: [PATCH 10/18] writeback: dirty position control - bdi reserve area
From: Wu Fengguang <hidden>
Date: 2011-09-29 12:15:10
Also in:
linux-mm, lkml
On Wed, Sep 28, 2011 at 10:02:05PM +0800, Wu Fengguang wrote:
quoted hunk ↗ jump to hunk
Hi Peter, On Sun, Sep 18, 2011 at 10:47:51PM +0800, Wu Fengguang wrote:quoted
quoted
BTW, I also compared the IO-less patchset and the vanilla kernel's JBOD performance. Basically, the performance is lightly improved under large memory, and reduced a lot in small memory servers. vanillla IO-less --------------------------------------------------------------------------------[...]quoted
26508063 17706200 -33.2% JBOD-10HDD-thresh=100M/xfs-100dd-1M-16p-5895M-100M 23767810 23374918 -1.7% JBOD-10HDD-thresh=100M/xfs-10dd-1M-16p-5895M-100M 28032891 20659278 -26.3% JBOD-10HDD-thresh=100M/xfs-1dd-1M-16p-5895M-100M 26049973 22517497 -13.6% JBOD-10HDD-thresh=100M/xfs-2dd-1M-16p-5895M-100M There are still some itches in JBOD..OK, in the dirty_bytes=100M case, I find that the bdi threshold _and_ writeout bandwidth may drop close to 0 in long periods. This change may avoid one bdi being stuck: /* * bdi reserve area, safeguard against dirty pool underrun and disk idle * * It may push the desired control point of global dirty pages higher * than setpoint. It's not necessary in single-bdi case because a * minimal pool of @freerun dirty pages will already be guaranteed. */ - x_intercept = min(write_bw, freerun); + x_intercept = min(write_bw + MIN_WRITEBACK_PAGES, freerun);After lots of experiments, I end up with this bdi reserve point + x_intercept = bdi_thresh / 2 + MIN_WRITEBACK_PAGES; together with this chunk to avoid a bdi stuck in bdi_thresh=0 state:@@ -590,6 +590,7 @@ static unsigned long bdi_position_ratio( */ if (unlikely(bdi_thresh > thresh)) bdi_thresh = thresh; + bdi_thresh = max(bdi_thresh, (limit - dirty) / 8); /* * scale global setpoint to bdi's: * bdi_setpoint = setpoint * bdi_thresh / threshThe above changes are good enough to keep reasonable amount of bdi dirty pages, so the bdi underrun flag ("[PATCH 11/18] block: add bdi flag to indicate risk of io queue underrun") is dropped. I also tried various bdi freerun patches, however the results are not satisfactory. Basically the bdi reserve area approach (this patch) yields noticeably more smooth/resilient behavior than the freerun/underrun approaches. I noticed that the bdi underrun flag could lead to sudden surge of dirty pages (especially if not safeguarded by the dirty_exceeded condition) in the very small window.. To dig performance increases/drops out of the large number of test results, I wrote a convenient script (attached) to compare the vmstat:nr_written numbers between 2+ set of test runs. It helped a lot for fine tuning the parameters for different cases. The current JBOD performance numbers are encouraging: $ ./compare.rb JBOD*/*-vanilla+ JBOD*/*-bgthresh3+ 3.1.0-rc4-vanilla+ 3.1.0-rc4-bgthresh3+ ------------------------ ------------------------ 52934365 +3.2% 54643527 JBOD-10HDD-thresh=100M/ext4-100dd-1M-24p-16384M-100M:10-X 45488896 +18.2% 53785605 JBOD-10HDD-thresh=100M/ext4-10dd-1M-24p-16384M-100M:10-X 47217534 +12.2% 53001031 JBOD-10HDD-thresh=100M/ext4-1dd-1M-24p-16384M-100M:10-X 32286924 +25.4% 40492312 JBOD-10HDD-thresh=100M/xfs-10dd-1M-24p-16384M-100M:10-X 38676965 +14.2% 44177606 JBOD-10HDD-thresh=100M/xfs-1dd-1M-24p-16384M-100M:10-X 59662173 +11.1% 66269621 JBOD-10HDD-thresh=800M/ext4-10dd-1M-24p-16384M-800M:10-X 57510438 +2.3% 58855181 JBOD-10HDD-thresh=800M/ext4-1dd-1M-24p-16384M-800M:10-X 63691922 +64.0% 104460352 JBOD-10HDD-thresh=800M/xfs-100dd-1M-24p-16384M-800M:10-X 51978567 +16.0% 60298210 JBOD-10HDD-thresh=800M/xfs-10dd-1M-24p-16384M-800M:10-X 47641062 +6.4% 50681038 JBOD-10HDD-thresh=800M/xfs-1dd-1M-24p-16384M-800M:10-X
[snip] I forgot to mention one important change that lead to the increased JBOD performance: the per-bdi background threshold as in the below patch. One thing puzzled me is that in JBOD case, the per-disk writeout performance is smaller than the corresponding single-disk case even when they have comparable bdi_thresh. So I wrote the attached tracing patch and find that in single disk case, bdi_writeback is always kept high while in JBOD case, it could drop low from time to time and correspondingly bdi_reclaimable could sometimes rush high. The fix is to watch bdi_reclaimable and kick background writeback as soon as it goes high. This resembles the global background threshold but in per-bdi manner. The trick is, as long as bdi_reclaimable does not go high, bdi_writeback naturally won't go low because bdi_reclaimable+bdi_writeback ~= bdi_thresh. With enough writeback pages, good performance is maintained. Thanks, Fengguang ---
--- linux-next.orig/fs/fs-writeback.c 2011-09-25 10:08:43.000000000 +0800
+++ linux-next/fs/fs-writeback.c 2011-09-25 15:36:41.000000000 +0800@@ -678,14 +678,18 @@ long writeback_inodes_wb(struct bdi_writ return nr_pages - work.nr_pages; } -static inline bool over_bground_thresh(void) +static bool over_bground_thresh(struct backing_dev_info *bdi) { unsigned long background_thresh, dirty_thresh; global_dirty_limits(&background_thresh, &dirty_thresh); - return (global_page_state(NR_FILE_DIRTY) + - global_page_state(NR_UNSTABLE_NFS) > background_thresh); + if (global_page_state(NR_FILE_DIRTY) + + global_page_state(NR_UNSTABLE_NFS) > background_thresh) + return true; + + return bdi_stat(bdi, BDI_RECLAIMABLE) > + bdi_dirty_limit(bdi, background_thresh); } /*
@@ -747,7 +751,7 @@ static long wb_writeback(struct bdi_writ * For background writeout, stop when we are below the * background dirty threshold */ - if (work->for_background && !over_bground_thresh()) + if (work->for_background && !over_bground_thresh(wb->bdi)) break; if (work->for_kupdate) {
@@ -831,7 +835,7 @@ static unsigned long get_nr_dirty_pages( static long wb_check_background_flush(struct bdi_writeback *wb) { - if (over_bground_thresh()) { + if (over_bground_thresh(wb->bdi)) { struct wb_writeback_work work = { .nr_pages = LONG_MAX,
Attachments
- trace-bdi-dirty-state.patch [text/x-diff] 2122 bytes · preview