On Thu, Oct 28, 2021 at 2:55 AM Joakim Tjernlund
[off-list ref] wrote:

On Wed, 2021-10-27 at 10:12 -0700, Yang Shi wrote:

quoted

On Wed, Oct 27, 2021 at 5:48 AM Joakim Tjernlund
[off-list ref] wrote:

quoted

Using a fairly minimal kernel I get a big difference in Reserved memory, using:
    qemu-system-aarch64 -machine virt -kernel arch/arm64/boot/Image -cpu cortex-a53  -nographic -m 32M

I get this:
Linux version 5.11.0
Memory: 24052K/32768K available (3520K kernel code, 684K rwdata, 548K rodata, 320K init, 245K bss, 8716K reserved, 0K cma-reserved)

Versus:

Linux version 5.10.75
Memory: 11836K/32768K available (3518K kernel code, 684K rwdata, 540K rodata, 320K init, 244K bss, 20932K reserved, 0K cma-reserved)

Reserving 20 MB RAM on a 32 MB is way too much, even 8Mb is a bit much for us.

This might be caused by the new lockless printk buffer patch
introduced in 5.10. We noticed this spike on 5.10 kernel too. It may
have much more memory reserved for meta data and the total amount may
be as much as 3x of log_buf_len. The amount varies on different
platforms with different configs. I didn't dig too much into the
details to figure out how it gets the number.

You could try to adjust CONFIG_LOG_BUF_SHIFT or set log_buf_len from
kernel commandline to limit the printk buffer size.

I did try this by changing LOG_BUF_SHIFT but the resulting change was just minor.

Thanks for following up, then you may run into a different problem from us.

To get some info about meminfo/zoneinfo I booted with an inird FS and got this:
  Memory: 9576K/32768K available (3518K kernel code, 460K rwdata, 540K rodata, 320K init, 180K bss, 23192K reserved, 0K cma-reserved)
The initrd only added some 3 MB's
Here are meminfo/zoneinfo, not sure if they are useful though.

They don't show who the reserved memory is consumed by.

If possible you may run git bisect to figure out what commit caused
this problem.

 cat /proc/meminfo
MemTotal:          12416 kB
MemFree:            4644 kB
MemAvailable:       4064 kB
Buffers:               0 kB
Cached:             4788 kB
SwapCached:            0 kB
Active:             2500 kB
Inactive:           2524 kB
Active(anon):       2500 kB
Inactive(anon):     2524 kB
Active(file):          0 kB
Inactive(file):        0 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:           248 kB
Mapped:             1844 kB
Shmem:              4788 kB
KReclaimable:          0 kB
Slab:               1712 kB
SReclaimable:          0 kB
SUnreclaim:         1712 kB
KernelStack:         384 kB
PageTables:           80 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:        6208 kB
Committed_AS:       5636 kB
VmallocTotal:   262930368 kB
VmallocUsed:         324 kB
VmallocChunk:          0 kB
Percpu:               76 kB
HardwareCorrupted:     0 kB

 cat /proc/zoneinfo
Node 0, zone   Normal
  per-node stats
      nr_inactive_anon 631
      nr_active_anon 625
      nr_inactive_file 0
      nr_active_file 0
      nr_unevictable 0
      nr_slab_reclaimable 0
      nr_slab_unreclaimable 429
      nr_isolated_anon 0
      nr_isolated_file 0
      workingset_nodes 0
      workingset_refault_anon 0
      workingset_refault_file 0
      workingset_activate_anon 0
      workingset_activate_file 0
      workingset_restore_anon 0
      workingset_restore_file 0
      workingset_nodereclaim 0
      nr_anon_pages 62
      nr_mapped    461
      nr_file_pages 1197
      nr_dirty     0
      nr_writeback 0
      nr_writeback_temp 0
      nr_shmem     1197
      nr_shmem_hugepages 0
      nr_shmem_pmdmapped 0
      nr_file_hugepages 0
      nr_file_pmdmapped 0
      nr_anon_transparent_hugepages 0
      nr_vmscan_write 0
      nr_vmscan_immediate_reclaim 0
      nr_dirtied   0
      nr_written   0
      nr_kernel_misc_reclaimable 0
      nr_foll_pin_acquired 0
      nr_foll_pin_released 0
      nr_kernel_stack 384
  pages free     1162
        min      97
        low      121
        high     145
        spanned  8192
        present  8192
        managed  3104
        protection: (0, 0)
      nr_free_pages 1162
      nr_zone_inactive_anon 631
      nr_zone_active_anon 625
      nr_zone_inactive_file 0
      nr_zone_active_file 0
      nr_zone_unevictable 0
      nr_zone_write_pending 0
      nr_mlock     0
      nr_page_table_pages 19
      nr_bounce    0
      nr_free_cma  0
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 2
  node_unreclaimable:  0
  start_pfn:           262144
Node 0, zone  Movable
  pages free     0
        min      0
        low      0
        high     0
        spanned  0
        present  0
        managed  0
        protection: (0, 0)