Re: [PATCH v3 RFC 14/14] mm: speedup page alloc for MPOL_PREFERRED_MANY by adding a NO_SLOWPATH gfp bit

On Wed, Mar 03, 2021 at 01:32:11PM +0100, Michal Hocko wrote:

On Wed 03-03-21 20:18:33, Feng Tang wrote:

quoted

On Wed, Mar 03, 2021 at 08:07:17PM +0800, Tang, Feng wrote:

quoted

Hi Michal,

On Wed, Mar 03, 2021 at 12:39:57PM +0100, Michal Hocko wrote:

quoted

On Wed 03-03-21 18:20:58, Feng Tang wrote:

quoted

When doing broader test, we noticed allocation slowness in one test
case that malloc memory with size which is slightly bigger than free
memory of targeted nodes, but much less then the total free memory
of system.

The reason is the code enters the slowpath of __alloc_pages_nodemask(),
which takes quite some time. As alloc_pages_policy() will give it a 2nd
try with NULL nodemask, so there is no need to enter the slowpath for
the first try. Add a new gfp bit to skip the slowpath, so that user cases
like this can leverage.

With it, the malloc in such case is much accelerated as it never enters
the slowpath.

Adding a new gfp_mask bit is generally not liked, and another idea is to
add another nodemask to struct 'alloc_context', so it has 2: 'preferred-nmask'
and 'fallback-nmask', and they will be tried in turn if not NULL, with
it we can call __alloc_pages_nodemask() only once.

Yes, it is very much disliked. Is there any reason why you cannot use
GFP_NOWAIT for that purpose?

I did try that at the first place, but it didn't obviously change the slowness.
I assumed the direct claim was still involved as GFP_NOWAIT only impact kswapd
reclaim.

I assume you haven't really created gfp mask correctly. What was the
exact gfp mask you have used?

The testcase is a malloc with multi-preferred-node policy, IIRC, the gfp
mask is HIGHUSER_MOVABLE originally, and code here ORs (__GFP_RETRY_MAYFAIL | __GFP_NOWARN).

As GFP_WAIT == __GFP_KSWAPD_RECLAIM, in this test case, the bit is already set.

quoted

One thing I tried which can fix the slowness is:

+	gfp_mask &= ~(__GFP_DIRECT_RECLAIM | __GFP_KSWAPD_RECLAIM);

which explicitly clears the 2 kinds of reclaim. And I thought it's too
hacky and didn't mention it in the commit log.

Clearing __GFP_DIRECT_RECLAIM would be the right way to achieve
GFP_NOWAIT semantic. Why would you want to exclude kswapd as well? 

When I tried gfp_mask &= ~__GFP_DIRECT_RECLAIM, the slowness couldn't
be fixed.

Thanks,
Feng

-- 
Michal Hocko
SUSE Labs