On 30.11.21 12:51, liupeng (DM) wrote:

Hi David,

 

Hi!

We met a performance regression(lmbench bw_mmap_rd/bw_mem) on our server
with

patch 7fef431be9c9(mm/page_alloc: place pages to tail in
__free_pages_core()),

 

Case A.  ./bw_mmap_rd -P 1 512m mmap_only out.file

Case B.  ./bw_mem -P 1 512m frd

 

mode0: IMC Interleave = auto, Channel Interleave = auto, Rank Interleave
= auto

                     without                        with        regression

case A           10535.14                       9645.02      8.4%

case B           10526.88                       9797.63      6.9%

also we found different memory interleaving have different results

 

mode1: IMC Interleave = 1-way, Channel Interleave = 2-way, Rank
Interleave =2-way

                      without                        with         
regression

case A           10543.01                       10643.12       -0.9%

case B           10971.33                       10853.99       1%

 

The 7fef431be9c9 changes the different memory layout,  it seems that the
issue

It doesn't change the physical memory layout. It changes the order in
which physical memory will get allocated.

is related with memory interleave. But the patch does lead to
performance regression.

 

Any suggestion about this regression?

Does it happen only on specific HW?

Commit 7fef431be9c9 discovered a bunch of BUGs already, including
physical memory that has different performance characteristics due to
firmware bugs, which indicated some physical memory e.g., as uncached to
the OS.

That commit triggers allocation of physical memory in a different order,
which sometimes makes "bad physical memory" getting allocated earlier
than later in some benchmarks. Such effects are usually only visible on
a freshly started system; once the system was busy for some time and the
free page lists are nondeterministic, you might observe similar results.

I wrote a kernel module that was able to identify bad physical memory
regions in some affected setups:
	https://github.com/davidhildenbrand/kstream/blob/main/README.md

You can try running it after a freshly booted system to eventually
identify some "bad" physical memory areas. But it might not be the root
cause. Maybe it's just some caching effects?

-- 
Thanks,

David / dhildenb