On Thu, Feb 9, 2017 at 4:46 PM, Vinayak Menon [off-list ref] wrote:

Hi,

We were trying to implement the per app memory cgroup that Johannes
suggested (https://lkml.org/lkml/2014/12/19/358) and later discussed during
Minchan's proposal of per process reclaim

Per app memory cgroups are interesting, but also quite aggressive. Could
you please describe what tasks/workload you have?

(https://lkml.org/lkml/2016/6/13/570). The test was done on Android target
with 2GB of RAM and cgroupv1. The first test done was to just create per
app cgroups without modifying any cgroup controls. 2 kinds of tests were
done which gives similar kind of observation. One was to just open
applications in sequence and repeat this N times (20 apps, so around 20
memcgs max at a time). Another test was to create around 20 cgroups and
perform a make (not kernel, another less heavy source) in each of them.

It is observed that because of the creation of memcgs per app, the per
memcg LRU size is so low and results in kswapd priority drop. This results
in sudden increase in scan at lower priorities. Because of this, kswapd
consumes around 3 times more time (and thus less pageoutrun), and due to
the lag in reclaiming memory direct reclaims are more and consumes around
2.5 times more time.

That does not sound good! Have you been able to test this with older
kernels to see if this is a regression?

Another observation is that the reclaim->generation check in
mem_cgroup_iter results in kswapd breaking the memcg lru reclaim loop in
shrink_zone (this is 4.4 kernel) often. This also contributes to the
priority drop. A test was done to skip the reclaim generation check in
mem_cgroup_iter and allow concurrent reclaimers to run at same priority.
This improved the results reducing the kswapd priority drops (and thus time
spent in kswapd, allocstalls etc). But this problem could be a side effect
of kswapd running for long and reclaiming slow resulting in many parallel
direct reclaims.

Some of the stats are shown below
                            base        per-app-memcg

pgalloc_dma                 4982349     5043134

pgfree                      5249224     5303701

pgactivate                  83480       117088

pgdeactivate                152407      1021799

pgmajfault                  421         31698

pgrefill_dma                156884      1027744

pgsteal_kswapd_dma          128449      97364

pgsteal_direct_dma          101012      229924

pgscan_kswapd_dma           132716      109750

pgscan_direct_dma           104141      265515

slabs_scanned               58782       116886

pageoutrun                  57          16

allocstall                  1283        3540


After this, offloading some of the job to soft reclaim was tried with the
assumption that it will result in lesser priority drops. The problem is in
determining the right value to be set for soft reclaim. For e.g. one of the
main motives behind using memcg in Android is to set different swappiness
to tasks depending on their importance (foreground, background etc.). In
such a case we actually do not want to set any soft limits. And in the
second case when we want to use soft reclaim to offload some work from
kswapd_shrink_zone on to mem_cgroup_soft_limit_reclaim, it becomes tricky
to set the soft limit values. I was trying out with different percentage of
task RSS for setting soft limit, but this actually results in excessive
scanning by mem_cgroup_soft_limit_reclaim, which as I understand  is
because of always using scan priority of 0. This in turn increases the time
spent in kswapd. It reduces the kswapd priority drop though.

Soft limit setting can be tricky, but my advise is to set it based on how much
you see a particular cgroup using when the system is under memory pressure.

Is there a way to mitigate this problem of small lru sizes, priority drop
and kswapd cpu consumption.

I've not investigated or heard of this problem before, so I am not
sure if I have
a solution for you.

Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>