Re: What's the progress of per cgroup background reclaim?

2012/3/15 Ying Han [off-list ref]:

On Wed, Mar 14, 2012 at 12:53 AM, Zhu Yanhai [off-list ref] wrote:

quoted

Hi all,
Just a quick question, could you please tell me what's the current
status of the development of per cgroup background reclaim? This topic
seems to be silent after Han Ying's patchset V7
(http://lwn.net/Articles/440073/) and Kame's async reclaim patchset V3
(https://lkml.org/lkml/2011/5/26/20), and I can't find it in
memcg-devel tree either.
Is anyone still working on this?

There were some discussions on going w/ per-memcg kswapd thread or
workqueue by that time. And now I think we agree to go w/ the
per-memcg thread model.  I haven't done much work since then, and one
of questions is to demonstrate the need of this feature.

I am glad you are asking, do you have workload showing problems w/o it?

Yes, the background is we have a cluster of about 3k-4k servers, all
running JVMs. Because the load of each Java application is small, we
gave each of them a small GC heap, which was 1.5GB or so. Then to take
full use of the huge memory of the servers, we setup several XEN based
virtual machines on each physical box, each XEN VM had one single JVM
running in it.
Now we are trying to switch to a LXC/cgroup based solution,  at the
first step we have built a small experimental cluster online, the
containers equipped with memcg are sizing to the same size with the
XEN VMs (we haven't enabled other controllers since the major need for
resources came from the memory, while the pressure against CPU and IO
usually is smaller). As soon as they are online, we noticed that the
latency recorded in the client side had periodic peaks. We also
noticed that the memory.failcnt counter periodic increased. By
enabling kmem:mm_directreclaim_reclaimall,
kmem:mm_vmscan_direct_reclaim_begin, kmem:mm_vmscan_direct_reclaim_end
in trace events, we can see that kmem:mm_directreclaim_reclaimall came
out regularly, without  kmem:mm_vmscan_direct_reclaim_begin/end seen.
That means the caller of do_try_to_free_pages is
try_to_free_mem_cgroup_pages, not the global caller try_to_free_pages.
To make load balance in the cluster wide, we tend to dispatch the Java
apps average over the containers on different physical boxes, that's
to say unless the cluster is to be filled up, the number of active
containers on each box won't be large. I think under such scenario the
global pressure can never be high, so kswapd keep sleeping most of the
time, however the local pressure within one single cgroup maybe very
high, which results to frequent direct reclaim.
The kernel we are using is a custom RHEL6U2 kernel. We have a kernel
team here so it's fine to backport the upstream solution if there is
any.

--
Thanks,
Zhu Yanhai

--Ying

quoted

Thanks,
Zhu Yanhai