Re: too big min_free_kbytes

Sorry for the long delay in replying. I've been out the last week and am
not properly back until tomorrow.

On Mon, Jan 24, 2011 at 04:00:34PM +0100, Andrea Arcangeli wrote:

eOn Mon, Jan 24, 2011 at 11:56:46AM +0800, Shaohua Li wrote:

quoted

Hi,
With transparent huge page, min_free_kbytes is set too big.
Before:
Node 0, zone    DMA32
  pages free     1812
        min      1424
        low      1780
        high     2136
        scanned  0
        spanned  519168
        present  511496

After:
Node 0, zone    DMA32
  pages free     482708
        min      11178
        low      13972
        high     16767
        scanned  0
        spanned  519168
        present  511496
This caused different performance problems in our test. I wonder why we
set the value so big.

It's to enable Mel's anti-frag that keeps pageblocks with movable and
unmovable stuff separated, same as "hugeadm
--set-recommended-min_free_kbytes".

It's not so much "make it work" as "make it work better". The effect can
be measured by recording the mm_page_alloc_extfrag event. The more times
it occurs, the worse fragmentation can get. The event also reports
whether it is severe or not.

Now that I checked, I'm seeing quite too much free memory with only 4G
of ram... You can see the difference with a "cp /dev/sda /dev/null" in
background interleaving these two commands:

There is more than just min_free_kbytes happening here. The high
watermark goes to 16M-ish but the amount of free memory is *way* above
that watermark. Something is causing page reclaim to be a lot more
agressive than it should be.

Is there a difference with THP enabled and disabled but leaving
min_free_kbytes alone? My preliminary theory is that 2M pages are being
requested and kswapd is being woken up when it shouldn't
(__GFP_NO_KSWAPD not specified when it should be). Unfortunately I do
not have access to source at the moment to double check.

echo always >/sys/kernel/mm/transparent_hugepage/enabled
echo 1000 > /proc/sys/vm/min_free_kbytes

The setting of min_free_kbytes to 67584 leads to 716MB of memory
free. Setting to 1000 leads to 20MB free. I'm afraid losing 716MB on a
4G system is way excessive regardless of THP...

Agreed.

can't we just have a
version of anti-frag that reserves a lot fewers pageblocks?

Anti-frag doesn't really take any additional special action due to
min_free_kbytes and it shouldn't be clearing out pageblocks
aggressively like this. I think it would also be worth checking how
often the mm_vmscan_kswapd_wake and mm_vmscan_wakeup_kswapd trace events
are triggering. If mm_vmscan_wakeup_kswapd is triggering a lot, a stack
trace of the most common triggering event might give a clue as to what
is going wrong.

Anti-frag
is quite important to avoid slab to fragment everything. I don't think
we can leave it like this.

For now you can workaround with the above echo 1000 > ...

Agreed. I'll try find time to investigate before the week is out but
after being offline for a week, I've a lot of catching up to do.

-- 
Mel Gorman
Linux Technology Center
IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>