On Tue, Jul 30, 2024 at 4:37 PM Nico Pache [off-list ref] wrote:

Hi Zi Yan,
On Mon, Jul 29, 2024 at 7:26 PM Zi Yan [off-list ref] wrote:

quoted

+Kirill

On 29 Jul 2024, at 18:27, Nico Pache wrote:

quoted

We've seen cases were customers switching from RHEL7 to RHEL8 see a
significant increase in the memory footprint for the same workloads.

Through our investigations we found that a large contributing factor to
the increase in RSS was an increase in THP usage.

Any knob is changed from RHEL7 to RHEL8 to cause more THP usage?

IIRC, most of the systems tuning is the same. We attributed the
increase in THP usage to a combination of improvements in the kernel,
and improvements in the libraries (better alignments). That allowed
THP allocations to succeed at a higher rate. I can go back and confirm
this tomorrow though.

quoted

quoted

For workloads like MySQL, or when using allocators like jemalloc, it is
often recommended to set /transparent_hugepages/enabled=never. This is
in part due to performance degradations and increased memory waste.

This series introduces enabled=defer, this setting acts as a middle
ground between always and madvise. If the mapping is MADV_HUGEPAGE, the
page fault handler will act normally, making a hugepage if possible. If
the allocation is not MADV_HUGEPAGE, then the page fault handler will
default to the base size allocation. The caveat is that khugepaged can
still operate on pages thats not MADV_HUGEPAGE.

Why? If user does not explicitly want huge page, why bother providing huge
pages? Wouldn't it increase memory footprint?

So we have "always", which will always try to allocate a THP when it
can. This setting gives good performance in a lot of conditions, but
tends to waste memory. Additionally applications DON'T need to be
modified to take advantage of THPs.

We have "madvise" which will only satisfy allocations that are
MADV_HUGEPAGE, this gives you granular control, and a lot of times
these madvises come from libraries. Unlike "always" you DO need to
modify your application if you want to use THPs.

Then we have "never", which of course, never allocates THPs.

Ok. back to your question, like "madvise", "defer" gives you the
benefits of THPs when you specifically know you want them
(madv_hugepage), but also benefits applications that dont specifically
ask for them (or cant be modified to ask for them), like "always"
does. The applications that dont ask for THPs must wait for khugepaged
to get them (avoid insertions at PF time)-- this curbs a lot of memory
waste, and gives an increased tunability over "always". Another added
benefit is that khugepaged will most likely not operate on short lived
allocations, meaning that only longstanding memory will be collapsed
to THPs.

The memory waste can be tuned with max_ptes_none... lets say you want
~90% of your PMD to be full before collapsing into a huge page. simply
set max_ptes_none=64. or no waste, set max_ptes_none=0, requiring the
512 pages to be present before being collapsed.

quoted

quoted

This allows for two things... one, applications specifically designed to
use hugepages will get them, and two, applications that don't use
hugepages can still benefit from them without aggressively inserting
THPs at every possible chance. This curbs the memory waste, and defers
the use of hugepages to khugepaged. Khugepaged can then scan the memory
for eligible collapsing.

khugepaged would replace application memory with huge pages without specific
goal. Why not use a user space agent with process_madvise() to collapse
huge pages? Admin might have more knobs to tweak than khugepaged.

The benefits of "always" are that no userspace agent is needed, and
applications dont have to be modified to use madvise(MADV_HUGEPAGE) to
benefit from THPs. This setting hopes to gain some of the same
benefits without the significant waste of memory and an increased
tunability.

future changes I have in the works are to make khugepaged more
"smart". Moving it away from the round robin fashion it currently
operates in, to instead make smart and informed decisions of what
memory to collapse (and potentially split).

Hopefully that helped explain the motivation for this new setting!

Any last comments before I resend this?

Ive been made aware of
https://lore.kernel.org/all/20240730125346.1580150-1-usamaarif642@gmail.com/T/#u (local)
which introduces THP splitting. These are both trying to achieve the
same thing through different means. Our approach leverages khugepaged
to promote pages, while Usama's uses the reclaim path to demote
hugepages and shrink the underlying memory.

I will leave it up to reviewers to determine which is better; However,
we can't have both, as we'd be introducing trashing conditions.

Cheers,
-- Nico

Cheer!
-- Nico

quoted

quoted

Admins may want to lower max_ptes_none, if not, khugepaged may
aggressively collapse single allocations into hugepages.

RFC note
==========
Im not sure if im missing anything related to the mTHP
changes. I think now that we have hugepage_pmd_enabled in
commit 00f58104202c ("mm: fix khugepaged activation policy") everything
should work as expected.

Nico Pache (2):
  mm: defer THP insertion to khugepaged
  mm: document transparent_hugepage=defer usage

 Documentation/admin-guide/mm/transhuge.rst | 18 ++++++++++---
 include/linux/huge_mm.h                    | 15 +++++++++--
 mm/huge_memory.c                           | 31 +++++++++++++++++++---
 3 files changed, 55 insertions(+), 9 deletions(-)

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <redacted>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Barry Song <baohua@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Lance Yang <redacted>
Cc: Peter Xu <peterx@redhat.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Rafael Aquini <redacted>
Cc: Andrea Arcangeli <redacted>
Cc: Jonathan Corbet <corbet@lwn.net>
--
2.45.2

--
Best Regards,
Yan, Zi