Re: [RFC PATCH 00/18] mm: arm64: Add kernel replication feature

Hi,

On Fri, Aug 28, 2026 at 12:11:40AM +0800, Nikita Panov wrote:

Current status:

There were several prior submissions with some sort of replication
for NUMA systems, including one from our side for the x86_64 platform.
In the last couple years, several research articles related to solving
locality issues on NUMA machines through replication emerged as well.

[1]  - arm64 kernel text replication
[2]  - x86 NUMA-aware kernel replication
[3]  - x86 kernel text replication
[4]  - NUMA replication of user data
[5]  - Mitosis: Transparently Self-Replicating Page-Tables for Large-Memory Machines
[6]  - WASP: Workload-Aware Self-Replicating Page-Tables for NUMA Servers
[7]  - PaCaR: Improved Buffered I/O Locality on NUMA Systems with Page Cache Replication
[8]  - Memory page replication for Linux on X86 processors

(What happened to 9-11? :P)

[12] - Optimize this_cpu_*() ops for non-x86 (ARM64 for this series)

As of today, none of it was merged into mainline.

I mean :) maybe take that as a hint? If a number of series trying to do X get
rejected by upstream, that is maybe suggestive of barking up the wrong tree?

However, after thorough re-evaluation, we were not able to observe
performance improvement for the x86 platform, so we have decided to stop this
direction and switch on arm64.

I mean, why? You should provide details here, this is quite hand-wavey. You are
also proposing core mm changes for something that seems specific to unique
hardware as far as I can tell, which is a big ask.

<snip>

Known problems:

1. Other combinations of base page size and va size (especially with 16K pages)
   should be adapted and verified.

Umm, yeah this is basic stuff for upstreamability :)

2. Replicated translation tables for the vmalloc region are not local right now.
   Allocation performed with default memory policy, so translation tables
   for kernel modules will not be local. However,
   replicated text and rodata of the modules are local.
   In general, vmalloc patch should be cleaned up.

Again, this is really more of an alpha pre-RFC I'd say.

3. Any modifications of kernel PGD level. These modifications
   should be synchronized across all replicated tables.
   Right now, for example, memory hotplug/hotunplug
   lacks this support, vmemmap and kasan regions for
   added memory might not be observed correctly. This could be fixed
   by patching all places in the kernel where swapper_pg_dir
   is modified, or by "lazy" propagation on kernel faults in the pgd-level.
   Propagation approach will not help in the case of pgd_clear()
   on swapper_pg_dir though.

Yeah OK this suggests to me you've got the locking and synchronisation all wrong
and it's worrying :)

In general I really oppose anything that adds additional kernel page tables or
complicates kernel page table handling.

We already have singificant complexity and bugs/races emerging from people doing
odd things with kernel page tables on assumption that it's 'safe'.

You'll need very compelling evidence to justify anything that touches such
sensitive stuff.

The code is also fiddling with PGD assignment in a way that could interact badly
with how these PGDs are synchronised. These things are very subtle, and even if
it's limited to one arch the core mm code is not.

Overall, this patch set in an early PoC stage and require some improvements.

Overhead:

Memory overhead for the kernel itself is about 30MB per NUMA node
  on our deployment. For kernel modules - depends on their sizes, but text
  and ro-data are not that big.
CPU overhead - replication performed on the boot stage. After boot
  only "rare" operations are slowed down -
  module loading, text patching, kernel table pgd-level modifications.

Hmm. I wonder if they're as rare as you think though? It all depends also on how
slowed down they are, how that manifests, etc.

Performance evaluation:

Our local testing was performed on
Kunpeng 920, 128 CPU, 4 nodes, 100Gb for each node.

Thanks for providing details of the hardware used!

Microbenchmark:
Kernel module with a huge text section (~50MB) filled with CPU-bound
instructions. For each NUMA node thread is spawned, each thread in a loop
executes isntructions. Total execution time of each thread is measured.
The insmod call bound to node 0 through numactl (less time is better).

So wait, you bound it to node 0, then rely on kernel text replication to improve
performance due to a bad hint?

That seems like you could fix the issue by binding correctly? :)

node                 0        1        2        3
Before time, s     5.567    7.598   13.294   18.905
After  time, s     5.469    6.960    6.777    5.531

Diff               ~0%      -8.5%    -49%     -70%
In this benchmark, interconnect was not used by any other actors,
so microbenchmark numbers might be significantly improved.

This benchmark seems entirely synthetic and it seems odd to me, prima facie, to
implement a feature to correct for incorrect NUMA binding?

Maybe I'm missing something though.

Customer's evaluation:
We were provided with the following feedback on this patch set
directly from our customers. Unfortunately, we do not have details
regarding how these measurements were done other than it was
a production setup.
Evaulation was performed on Kunpeng 920 and 920B platforms:
CEPH distributed storage +5%
StarRocksDB              +5%

This isn't hugely encouraging.

Couple more words about patch set and technology:

This patchset was merged into the innovative branch of
the openEuler distributive 1.5 year ago (openEuler-25.03)
and was actively tested in production environment [9], [10].
In addition, besides the kernel part, we have published
user space replication (for translation tables and rodata) as well,
but it is very complex and experimental
even compared to this patch set [11]. With replication in user
space, we were able to achieve the following numbers in
performance improvement:
MySQL + sysbench      1-6%
Spark TPC-H           4-20%
Phoronix test-suite   0-25%

These seem very vague and wide-ranged, I'm not sure they're really saying much
at all?

Discussion:

The main question we'd like to discuss is the following:
Should the kernel replication feature be merged into the Linux
somewhere in the future? In any form, not specifically this patch set,
but the core concept itself.

I will leave the broader topic to the NUMA experts.

If the answer is yes, please share your thoughts on this patch set. What else
should be fixed (or reimplemented and redsigned completly) in this patch
for mainline in your opinion? We'd be glad to do it, and in that case
I'll send an updated version in the near future.

Glancing thorugh, The patch set seems very far from being upstreamable:

	if (kernel_replication_enabled)
		pr_info("WARNING! WARNING! WARNING! Kernel replication enabled WARNING! WARNING! WARNING!\n");

For instance... this really shouts some alpha effort.

Annnd :) this:

	if (rwsem_is_locked(&mm->mmap_lock))
		locked = true;
	else
		mmap_read_lock(mm);

is just utterly, utterly broken.

rwsem_is_locked() can be raced at any time, you don't own the lock so it can
just be unlocked underneath you.

That you do that suggests to me you've not thought about locks correctly _at
all_ here.

And as I said above, locking issues around page table walking and manipulation
is very subtle and difficult to manage correctly.

Then there's stuff like this:

		/* TODO: remove last condition and do something better
		 * In the case of a folded P4D level, pgd_none and pgd_huge
		 * always return 0, so we might start to replicate empty entries.
		 * We obviously want to avoid this, so the last check is performed here.
		 */
		if (pgd_none(*orig_pgd) || pgd_val(*orig_pgd) == 0)
			goto skip;

This whole block seems confused, and it's nitty but you're using completely
incorrect comment style for the linux kernel which again doesn't fill me with
confidence that you've really thought things through or understand mm code
correctly.

Your replicate_memory() function seems to not synchronise _at all_, but you do
for some reason synchronise on dumping memory, bizarely.

You write a ton of duplicative page table code that doesn't seem to handle huge
pages at all, again doesn't seem to be performing any locking correctly at all,
and yeah the list goes on.

In general the code looks like an alpha experimental thing and a million miles
away from anything even vaguely upstreamable.

And in general for this kind of thing - the devil is in the detail.

So if you want to assert that something is viable, you need code that at
least looks _vaguely_ upstreamable and demonstates understanding of the
issues at play here, and you are not doing so.

In conclusion:

- 12 (or is it 9? :) attempts have been made at this kind of thing and all
  were rejected - this isn't an encouraging sign that the approach is
  viable.

- You've already found it has limited use, and your benchmark numbers seem
  either entirely synthetic or bordeline not statistically significant.

- The code is, as discussed, not even vaguely close to being upstreamable.

So overall it seems to me that perhaps better NUMA policy decisions could
solve your problems.

But yeah, what's presented in this series doesn't seem like a worthwhile
road to travel down to me.

--
Cheers, Lorenzo