On Tue, Jan 17, 2023 at 03:54:34PM +0100, Christoph Lameter wrote:

On Tue, 17 Jan 2023, Jesper Dangaard Brouer wrote:

quoted

When running different network performance microbenchmarks, I started
to notice that performance was reduced (slightly) when machines had
longer uptimes. I believe the cause was 'skbuff_head_cache' got
aliased/merged into the general slub for 256 bytes sized objects (with
my kernel config, without CONFIG_HARDENED_USERCOPY).

Well that is a common effect that we see in multiple subsystems. This is
due to general memory fragmentation. Depending on the prior load the
performance could actually be better after some runtime if the caches are
populated avoiding the page allocator etc.

The merging could actually be beneficial since there may be more partial
slabs to allocate from and thus avoiding expensive calls to the page
allocator.

I wish we had some effective way of memory defragmentation.

If general memory fragmentation is actual cause of this problem, 
it may be worsening by [1] due to assumption that all allocations
are done in the same order as s->oo, when accounting and limiting the number
of percpu slabs.

[1] https://lore.kernel.org/linux-mm/76c63237-c489-b942-bdd9-5720042f52a9@suse.cz (local)