On Tue, Feb 23, 2016 at 08:19:32PM +0000, Mel Gorman wrote:

On Tue, Feb 23, 2016 at 12:04:16PM -0800, Johannes Weiner wrote:

quoted

On Tue, Feb 23, 2016 at 03:04:23PM +0000, Mel Gorman wrote:

quoted

In many benchmarks, there is an obvious difference in the number of
allocations from each zone as the fair zone allocation policy is removed
towards the end of the series. For example, this is the allocation stats
when running blogbench that showed no difference in headling performance

                          mmotm-20160209   nodelru-v2
DMA allocs                           0           0
DMA32 allocs                   7218763      608067
Normal allocs                 12701806    18821286
Movable allocs                       0           0

According to the mmotm numbers, your DMA32 zone is over a third of
available memory, yet in the nodelru-v2 kernel sees only 3% of the
allocations.

In this case yes but blogbench is not scaled to memory size and is not
reclaim intensive. If you look, you'll see the total number of overall
allocations is very similar. During that test, there is a small amount of
kswapd scan activity (but not reclaim which is odd) at the start of the
test for nodelru but that's about it.

Yes, if fairness enforcement is now done by reclaim, then workloads
without reclaim will show skewed placement as the Normal zone is again
filled up first before moving on to the next zone.

That is fine. But what about the balance in reclaiming workloads?

quoted

That's an insanely high level of aging inversion, where
the lifetime of a cache entry is again highly dependent on placement.

The aging is now indepdant of what zone the page was allocated from because
it's node-based LRU reclaim. That may mean that the occupancy of individual
zones is now different but it should only matter if there is a large number
of address-limited requests.

The problem is that kswapd will stay awake and continuously draw
subsequent allocations into a single zone, thus utilizing only a
fraction of available memory. A DMA32-limited kswapd wakeups can
reclaim cache in DMA32 continuously if the allocator continously
places new cache pages in that zone. It looks like that is what
happened in the stutter benchmark.

Sure, it doesn't matter in that benchmark, because the pages are used
only once. But if it had an actual cache workingset bigger than DMA32
but smaller than DMA32+Normal, it would be thrashing unnecessarily.

If kswapd were truly balancing the pages in a node equally, regardless
of zone placement, then in the long run we should see zone allocations
converge to a share that is in proportion to each zone's size. As far
as I can see, that is not quite happening yet.

quoted

The fact that this doesn't make a performance difference in the
specific benchmarks you ran only proves just that: these specific
benchmarks don't care. IMO, benchmarking is not enough here. If this
is truly supposed to be unproblematic, then I think we need a reasoned
explanation. I can't imagine how it possibly could be, though.

The basic explanation is that reclaim is on a per-node basis and we
no longer balance all zones, just one that is necessary to satisfy the
original request that wokeup kswapd.

quoted

If reclaim can't guarantee a balanced zone utilization then the
allocator has to keep doing it. :(

That's the key issue - the main reason balanced zone utilisation is
necessary is because we reclaim on a per-zone basis and we must avoid
page aging anomalies. If we balance such that one eligible zone is above
the watermark then it's less of a concern.

Yes, but only if there can't be extended reclaim stretches that prefer
the pages of a single zone. Yet it looks like this is still possible.

I wonder if that were fixed by dropping patch 7/27? Potentially it
would need a bit more work than that. I.e. could we make kswapd
balance only for the highest classzone in the system, and thus make
address-limited allocations fend for themselves in direct reclaim?

This way, we would avoid that pathological interaction between kswapd
and the allocator, and kswapd would be guaranteed to balance fairly.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>