On Wed, 2014-01-08 at 16:54 -0500, Debabrata Banerjee wrote:

Actually I have more data on this:

Could you please stop polluting this thread ?

1. __GFP_NORETRY really does help and should go into stable tree.

Not at all. You are free to patch your kernel if you want.

It helps you workload, and breaks all others.

If compaction is never triggered, we'll never be able to get high order
pages, and performance goes back to what we had 2 years ago.

That discussion does not belong to this thread, again.

2. You may want to consider GFP_NOKSWAPD, because even in the
GFP_ATOMIC case you are waking up kswapd to do reclaims on a
continuous basis even when you don't enter direct reclaim.

3. mlocking memory had very little to do with it, that was a
red-herring. I tested out the problem scenario with no mlocks. You
simply need memory pressure from page_cache, and mm ends up constantly
reclaiming and trying to keep another 1-2GB free on our systems (8GB
phys ~4GB left for kernel, ~3GB optimally used for page_cache).

4. I think perhaps using a kmem_cache allocation for this buffer is
the right way to make this work. I am experimenting with a patch to do
this.

Seriously... I think you missed whole point of having frag allocation on
pages, not kmem_cache.