Re: [PATCH v5] mm: Proactive compaction

On Thu, May 28, 2020 at 4:32 PM Khalid Aziz [off-list ref] wrote:

This looks good to me. I like the idea overall of controlling
aggressiveness of compaction with a single tunable for the whole
system. I wonder how an end user could arrive at what a reasonable
value would be for this based upon their workload. More comments below.

Tunables like the one this patch introduces, and similar ones like 'swappiness'
will always require some experimentations from the user.

On Mon, 2020-05-18 at 11:14 -0700, Nitin Gupta wrote:

quoted

For some applications, we need to allocate almost all memory as
hugepages. However, on a running system, higher-order allocations can
fail if the memory is fragmented. Linux kernel currently does on-
demand
compaction as we request more hugepages, but this style of compaction
incurs very high latency. Experiments with one-time full memory
compaction (followed by hugepage allocations) show that kernel is
able
to restore a highly fragmented memory state to a fairly compacted
memory
state within <1 sec for a 32G system. Such data suggests that a more
proactive compaction can help us allocate a large fraction of memory
as
hugepages keeping allocation latencies low.

For a more proactive compaction, the approach taken here is to define
a new tunable called 'proactiveness' which dictates bounds for
external
fragmentation wrt HUGETLB_PAGE_ORDER order which kcompactd tries to
maintain.

The tunable is exposed through sysctl:
  /proc/sys/vm/compaction_proactiveness

It takes value in range [0, 100], with a default of 20.

Looking at the code, setting this to 100 would mean system would
continuously strive to drive level of fragmentation down to 0 which can
not be reasonable and would bog the system down. A cap lower than 100
might be a good idea to keep kcompactd from dragging system down.

Yes, I understand that a value of 100 would be a continuous compaction
storm but I still don't want to artificially cap the tunable. The interpretation
of this tunable can change in future, and a range of [0, 100] seems
more intuitive than, say [0, 90]. Still, I think a word of caution should
be added to its documentation (admin-guide/sysctl/vm.rst).

quoted

quoted

Total 2M hugepages allocated = 383859 (749G worth of hugepages out of
762G total free => 98% of free memory could be allocated as
hugepages)

- With 5.6.0-rc3 + this patch, with proactiveness=20

echo 20 | sudo tee /sys/kernel/mm/compaction/node-*/proactiveness

Should be "echo 20 | sudo tee /proc/sys/vm/compaction_proactiveness"

oops... I forgot to update the patch description. This is from the v4 patch
which used sysfs but v5 switched to using sysctl.

quoted

quoted

diff --git a/Documentation/admin-guide/sysctl/vm.rst

b/Documentation/admin-guide/sysctl/vm.rst
index 0329a4d3fa9e..e5d88cabe980 100644

--- a/Documentation/admin-guide/sysctl/vm.rst
+++ b/Documentation/admin-guide/sysctl/vm.rst

@@ -119,6 +119,19 @@ all zones are compacted such that free memory is

available in contiguous
 blocks where possible. This can be important for example in the
allocation of
 huge pages although processes will also directly compact memory as
required.

+compaction_proactiveness
+========================
+
+This tunable takes a value in the range [0, 100] with a default
value of
+20. This tunable determines how aggressively compaction is done in
the
+background. Setting it to 0 disables proactive compaction.
+
+Note that compaction has a non-trivial system-wide impact as pages
+belonging to different processes are moved around, which could also
lead
+to latency spikes in unsuspecting applications. The kernel employs
+various heuristics to avoid wasting CPU cycles if it detects that
+proactive compaction is not being effective.
+

Value of 100 would cause kcompactd to try to bring fragmentation down
to 0. If hugepages are being consumed and released continuously by the
workload, it is possible that kcompactd keeps making progress (and
hence passes the test "proactive_defer = score < prev_score ?")
continuously but can not reach a fragmentation score of 0 and hence
gets stuck in compact_zone() for a long time. Page migration for
compaction is not inexpensive. Maybe either cap the value to something
less than 100 or set a floor for wmark_low above 0.

Some more guidance regarding the value for this tunable might be
helpful here, something along the lines of what does a value of 100
mean in terms of how kcompactd will behave. It can then give end user a
better idea of what they are getting at what cost. You touch upon the
cost above. Just add some more details so an end user can get a better
idea of size of the cost for higher values of this tunable.

I like the idea of capping wmark_low to say, 5 to prevent admins from
overloading the system. Similarly, wmark_high should be capped at
say, 95 to allow tunable values below 10 to have any effect: currently
such low tunable values would give wmark_high=100 which would
cause proactive compaction to never get triggered.

Finally, I see your concern about lack of guidance on extreme values
of the tunable. I will address this in the next (v6) iteration.

Thanks,
Nitin