Here are the results of my research.
One doc is an overview fo the data and the other is a pdf of the raw data.

https://drive.google.com/file/d/1DE8QMri1Rsr7L27fORHFCmwgrMtdfPfu/view?usp=share_link

https://drive.google.com/file/d/1UwnTeqsKB0jgpnZodJ0_cM2bOHx5aR_v/view?usp=share_link

On Thu, Apr 27, 2023 at 4:29 AM Vlastimil Babka [off-list ref] wrote:

On 4/5/23 21:54, Binder Makin wrote:

quoted

I'm still running tests to explore some of these questions.
The machines I am using are roughly as follows.

Intel dual socket 56 total cores
192-384GB ram
LEVEL1_ICACHE_SIZE                 32768
LEVEL1_DCACHE_SIZE                 32768
LEVEL2_CACHE_SIZE                  1048576
LEVEL3_CACHE_SIZE                  40370176

Amd dual socket 128 total cores
1TB ram
LEVEL1_ICACHE_SIZE                 32768
LEVEL1_DCACHE_SIZE                 32768
LEVEL2_CACHE_SIZE                  524288
LEVEL3_CACHE_SIZE                  268435456

Arm single socket 64 total cores
256GB rma
LEVEL1_ICACHE_SIZE                 65536
LEVEL1_DCACHE_SIZE                 65536
LEVEL2_CACHE_SIZE                  1048576
LEVEL3_CACHE_SIZE                  33554432

So with "some artifact of different cache layout" I didn't mean the
different cache sizes of the processors, but possible differences how
objects end up placed in memory by SLAB vs SLUB causing them to collide in
the cache of cause false sharing less or more. This kind of interference can
make interpreting (micro)benchmark results hard.

Anyway, how I'd hope to approach this topic would be that SLAB removal is
proposed, and anyone who opposes that because they can't switch from SLAB to
SLUB would describe why they can't. I'd hope the "why" to be based on
testing with actual workloads, not just benchmarks. Benchmarks are then of
course useful if they can indeed distill the reason why the actual workload
regresses, as then anyone can reproduce that locally and develop/test fixes
etc. My hope is that if some kind of regression is found (e.g. due to lack
of percpu array in SLUB), it can be dealt with by improving SLUB.

Historically I recall that we (SUSE) objected somwhat to SLAB removal as our
distro kernels were using it, but we have switched since. Then networking
had concerns (possibly related to the lack percpu array) but seems bulk
allocations helped and they use SLUB these days [1]. And IIRC Google was
also sticking to SLAB, which led to some attempts to augment SLUB for those
workloads years ago, but those were never finished. So I'd be curious if we
should restart those effors or can just remove SLAB now.

[1] https://lore.kernel.org/all/93665604-5420-be5d-2104-17850288b955@redhat.com/ (local)