Greetings!
At boot with a KASAN-enabled v6.14-rc4 kernel on my PowerMac G4 DP I get:
[...]
vmalloc_node_range for size 4198400 failed: Address range restricted to 0xf1000000 - 0xf5110000
swapon: vmalloc error: size 4194304, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=openrc.swap,mems_allowed=0
CPU: 0 UID: 0 PID: 870 Comm: swapon Tainted: G W 6.14.0-rc4-PMacG4 #6
Tainted: [W]=WARN
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[f2ffb9d0] [c14cfd88] dump_stack_lvl+0x70/0x8c (unreliable)
[f2ffb9f0] [c04fb9b8] warn_alloc+0x154/0x2b8
[f2ffbab0] [c04de94c] __vmalloc_node_range_noprof+0x154/0x958
[f2ffbb80] [c04df23c] __vmalloc_node_noprof+0xec/0xf4
[f2ffbbc0] [c0558524] swap_cgroup_swapon+0x70/0x198
[f2ffbbf0] [c051e8d8] sys_swapon+0x1838/0x3624
[f2ffbce0] [c001e574] system_call_exception+0x2dc/0x420
[f2ffbf30] [c00291ac] ret_from_syscall+0x0/0x2c
--- interrupt: c00 at 0x2612ec
NIP: 002612ec LR: 00534108 CTR: 001e8310
REGS: f2ffbf40 TRAP: 0c00 Tainted: G W (6.14.0-rc4-PMacG4)
MSR: 0000d032 <EE,PR,ME,IR,DR,RI> CR: 24002444 XER: 00000000
GPR00: 00000057 afe3ef20 a7a95540 01b2bdd0 00000000 24002444 fe5ff7e1 00247c24
GPR08: 0000d032 0000fa89 01b2d568 001e8310 24002448 0054fe14 02921154 00000000
GPR16: 00000000 00534b50 afe3f0ac afe3f0b0 00000000 00000000 0055001c afe3f0d0
GPR24: afe3f0b0 00000003 00000000 00001000 01b2bdd0 00000002 005579ec 01b2d570
NIP [002612ec] 0x2612ec
LR [00534108] 0x534108
Mem-Info:
active_anon:1989 inactive_anon:0 isolated_anon:0
active_file:6407 inactive_file:5879 isolated_file:0
unevictable:0 dirty:0 writeback:0
slab_reclaimable:1538 slab_unreclaimable:22927
mapped:2753 shmem:107 pagetables:182
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:433110 free_pcp:472 free_cma:0
Node 0 active_anon:7972kB inactive_anon:0kB active_file:25652kB inactive_file:23496kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:10908kB dirty:0kB writeback:0kB shmem:464kB writeback_tmp:0kB kernel_stack:1568kB pagetables:724kB sec_pagetables:0kB all_unreclaimable? no
DMA free:591772kB boost:0kB min:3380kB low:4224kB high:5068kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:11056kB unevictable:0kB writepending:0kB present:786432kB managed:716492kB mlocked:0kB bounce:0kB free_pcp:1680kB local_pcp:1180kB free_cma:0kB
lowmem_reserve[]: 0 0 1184 0
DMA: 127*4kB (UE) 66*8kB (UME) 37*16kB (UE) 78*32kB (UME) 10*64kB (UE) 4*128kB (UME) 3*256kB (UM) 6*512kB (UM) 5*1024kB (ME) 4*2048kB (M) 139*4096kB (M) = 591772kB
12404 total pagecache pages
0 pages in swap cache
Free swap = 0kB
Total swap = 0kB
524288 pages RAM
327680 pages HighMem/MovableOnly
42061 pages reserved
Memory allocations:
85.3 MiB 6104 mm/slub.c:2423 func:alloc_slab_page
38.5 MiB 9862 mm/readahead.c:187 func:ractl_alloc_folio
9.47 MiB 2425 mm/filemap.c:1970 func:__filemap_get_folio
7.96 MiB 2037 mm/kasan/shadow.c:304 func:kasan_populate_vmalloc_pte
7.87 MiB 2125 mm/execmem.c:44 func:execmem_vmalloc
5.01 MiB 1283 mm/memory.c:1063 func:folio_prealloc
4.00 MiB 1 fs/btrfs/zstd.c:366 [btrfs] func:zstd_alloc_workspace
3.86 MiB 247 lib/stackdepot.c:627 func:stack_depot_save_flags
3.62 MiB 412 mm/slub.c:2425 func:alloc_slab_page
3.09 MiB 18430 fs/kernfs/dir.c:624 func:__kernfs_new_node
couldn't allocate enough memory for swap_cgroup
swap_cgroup can be disabled by swapaccount=0 boot option
[...]
Does only happen with CONFIG_KASAN_INLINE=y but not with CONFIG_KASAN_OUTLINE=y.
Kernel .config attached.
Regards,
Erhard
Erhard Furtner [off-list ref] writes:
Greetings!
At boot with a KASAN-enabled v6.14-rc4 kernel on my PowerMac G4 DP I get:
[...]
vmalloc_node_range for size 4198400 failed: Address range restricted to 0xf1000000 - 0xf5110000
swapon: vmalloc error: size 4194304, vm_struct allocation failed, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=openrc.swap,mems_allowed=0
Did we exhaust the vmalloc area completely?
CPU: 0 UID: 0 PID: 870 Comm: swapon Tainted: G W 6.14.0-rc4-PMacG4 #6
Tainted: [W]=WARN
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[f2ffb9d0] [c14cfd88] dump_stack_lvl+0x70/0x8c (unreliable)
[f2ffb9f0] [c04fb9b8] warn_alloc+0x154/0x2b8
[f2ffbab0] [c04de94c] __vmalloc_node_range_noprof+0x154/0x958
[f2ffbb80] [c04df23c] __vmalloc_node_noprof+0xec/0xf4
[f2ffbbc0] [c0558524] swap_cgroup_swapon+0x70/0x198
[f2ffbbf0] [c051e8d8] sys_swapon+0x1838/0x3624
[f2ffbce0] [c001e574] system_call_exception+0x2dc/0x420
Since only the swapon failed, I think you might still have the console
up right? So this is mostly a vmalloc allocation failure report?
quoted hunk
[f2ffbf30] [c00291ac] ret_from_syscall+0x0/0x2c
--- interrupt: c00 at 0x2612ec
NIP: 002612ec LR: 00534108 CTR: 001e8310
REGS: f2ffbf40 TRAP: 0c00 Tainted: G W (6.14.0-rc4-PMacG4)
MSR: 0000d032 <EE,PR,ME,IR,DR,RI> CR: 24002444 XER: 00000000
GPR00: 00000057 afe3ef20 a7a95540 01b2bdd0 00000000 24002444 fe5ff7e1 00247c24
GPR08: 0000d032 0000fa89 01b2d568 001e8310 24002448 0054fe14 02921154 00000000
GPR16: 00000000 00534b50 afe3f0ac afe3f0b0 00000000 00000000 0055001c afe3f0d0
GPR24: afe3f0b0 00000003 00000000 00001000 01b2bdd0 00000002 005579ec 01b2d570
NIP [002612ec] 0x2612ec
LR [00534108] 0x534108
Mem-Info:
active_anon:1989 inactive_anon:0 isolated_anon:0
active_file:6407 inactive_file:5879 isolated_file:0
unevictable:0 dirty:0 writeback:0
slab_reclaimable:1538 slab_unreclaimable:22927
mapped:2753 shmem:107 pagetables:182
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:433110 free_pcp:472 free_cma:0
Node 0 active_anon:7972kB inactive_anon:0kB active_file:25652kB inactive_file:23496kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:10908kB dirty:0kB writeback:0kB shmem:464kB writeback_tmp:0kB kernel_stack:1568kB pagetables:724kB sec_pagetables:0kB all_unreclaimable? no
DMA free:591772kB boost:0kB min:3380kB low:4224kB high:5068kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:11056kB unevictable:0kB writepending:0kB present:786432kB managed:716492kB mlocked:0kB bounce:0kB free_pcp:1680kB local_pcp:1180kB free_cma:0kB
lowmem_reserve[]: 0 0 1184 0
DMA: 127*4kB (UE) 66*8kB (UME) 37*16kB (UE) 78*32kB (UME) 10*64kB (UE) 4*128kB (UME) 3*256kB (UM) 6*512kB (UM) 5*1024kB (ME) 4*2048kB (M) 139*4096kB (M) = 591772kB
12404 total pagecache pages
0 pages in swap cache
Free swap = 0kB
Total swap = 0kB
524288 pages RAM
327680 pages HighMem/MovableOnly
42061 pages reserved
Though above are mainly the physical mem info printed, but vmalloc can
also fail sometimes (e.g. this report), it is nice if we can print how
much of vmalloc space is free out of vmalloc total in show_mem() here.
Maybe linux-mm can tell if we should add this diff change for future?
diff --git a/mm/show_mem.c b/mm/show_mem.c
index 43afb56abbd3..b3af59fced02 100644
--- a/mm/show_mem.c
+++ b/mm/show_mem.c
@@ -14,6 +14,7 @@
#include <linux/mmzone.h>
#include <linux/swap.h>
#include <linux/vmstat.h>
+#include <linux/vmalloc.h>
#include "internal.h"
#include "swap.h"
@@ -416,6 +417,8 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)
printk("%lu pages RAM\n", total);
printk("%lu pages HighMem/MovableOnly\n", highmem);
printk("%lu pages reserved\n", reserved);
+ printk("%lu pages Vmalloc Total\n", (unsigned long)VMALLOC_TOTAL >> PAGE_SHIFT);
+ printk("%lu pages Vmalloc Used\n", vmalloc_nr_pages());
#ifdef CONFIG_CMA
printk("%lu pages cma reserved\n", totalcma_pages);
#endif
But meanwhile below data can give more details about the vmalloc area.
1. cat /proc/vmallocinfo
2. cat /proc/meminfo
-ritesh
Memory allocations:
85.3 MiB 6104 mm/slub.c:2423 func:alloc_slab_page
38.5 MiB 9862 mm/readahead.c:187 func:ractl_alloc_folio
9.47 MiB 2425 mm/filemap.c:1970 func:__filemap_get_folio
7.96 MiB 2037 mm/kasan/shadow.c:304 func:kasan_populate_vmalloc_pte
7.87 MiB 2125 mm/execmem.c:44 func:execmem_vmalloc
5.01 MiB 1283 mm/memory.c:1063 func:folio_prealloc
4.00 MiB 1 fs/btrfs/zstd.c:366 [btrfs] func:zstd_alloc_workspace
3.86 MiB 247 lib/stackdepot.c:627 func:stack_depot_save_flags
3.62 MiB 412 mm/slub.c:2425 func:alloc_slab_page
3.09 MiB 18430 fs/kernfs/dir.c:624 func:__kernfs_new_node
couldn't allocate enough memory for swap_cgroup
swap_cgroup can be disabled by swapaccount=0 boot option
[...]
Does only happen with CONFIG_KASAN_INLINE=y but not with CONFIG_KASAN_OUTLINE=y.
Kernel .config attached.
Regards,
Erhard
On Thu, 27 Feb 2025 10:42:44 +0530
Ritesh Harjani (IBM) [off-list ref] wrote:
Since only the swapon failed, I think you might still have the console
up right? So this is mostly a vmalloc allocation failure report?
Correct. The machine stays usable via VNC in my case, even after the vmalloc allocation failure.
quoted hunk
Though above are mainly the physical mem info printed, but vmalloc can
also fail sometimes (e.g. this report), it is nice if we can print how
much of vmalloc space is free out of vmalloc total in show_mem() here.
Maybe linux-mm can tell if we should add this diff change for future?
diff --git a/mm/show_mem.c b/mm/show_mem.c
index 43afb56abbd3..b3af59fced02 100644
--- a/mm/show_mem.c
+++ b/mm/show_mem.c
@@ -14,6 +14,7 @@
#include <linux/mmzone.h>
#include <linux/swap.h>
#include <linux/vmstat.h>
+#include <linux/vmalloc.h>
#include "internal.h"
#include "swap.h"
@@ -416,6 +417,8 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx)
printk("%lu pages RAM\n", total);
printk("%lu pages HighMem/MovableOnly\n", highmem);
printk("%lu pages reserved\n", reserved);
+ printk("%lu pages Vmalloc Total\n", (unsigned long)VMALLOC_TOTAL >> PAGE_SHIFT);
+ printk("%lu pages Vmalloc Used\n", vmalloc_nr_pages());
#ifdef CONFIG_CMA
printk("%lu pages cma reserved\n", totalcma_pages);
#endif
I applied your patch and now get this output:
[...]
vmalloc_node_range for size 1196032 failed: Address range restricted to 0xf1000000 - 0xf5110000
modprobe: vmalloc error: size 1189956, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 1 UID: 0 PID: 1853 Comm: modprobe Tainted: G W 6.14.0-rc4-PMacG4-dirty #9
Tainted: [W]=WARN
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[f427b850] [c14cfe28] dump_stack_lvl+0x70/0x8c (unreliable)
[f427b870] [c04fba74] warn_alloc+0x154/0x2b8
[f427b930] [c04dea08] __vmalloc_node_range_noprof+0x154/0x958
[f427ba00] [c04df2f8] __vmalloc_node_noprof+0xec/0xf4
[f427ba40] [c062da4c] kernel_read_file+0x514/0x7a0
[f427baf0] [c01ee5d4] init_module_from_file+0xb0/0x108
[f427bbd0] [c01eed04] sys_finit_module+0x390/0x7a4
[f427bca0] [c001e574] system_call_exception+0x2dc/0x420
[f427bf30] [c00291ac] ret_from_syscall+0x0/0x2c
--- interrupt: c00 at 0x321334
NIP: 00321334 LR: 005c9340 CTR: 002ad944
REGS: f427bf40 TRAP: 0c00 Tainted: G W (6.14.0-rc4-PMacG4-dirty)
MSR: 0000d032 <EE,PR,ME,IR,DR,RI> CR: 2822242c XER: 20000000
GPR00: 00000161 afb2c9e0 a7a03540 00000001 005d6a48 00000000 afb2c9b5 0000007f
GPR08: 00000000 af3da4b8 00000000 00000008 00000003 005efe10 2422242c 00000000
GPR16: 00040000 00000000 00000000 00000000 00750610 00000000 00751a30 00000000
GPR24: 00750610 00000000 005d6a48 00751330 00040000 00000000 005f7aac 00750610
NIP [00321334] 0x321334
LR [005c9340] 0x5c9340
Mem-Info:
active_anon:16715 inactive_anon:0 isolated_anon:0
active_file:25985 inactive_file:12645 isolated_file:0
unevictable:0 dirty:23 writeback:0
slab_reclaimable:2378 slab_unreclaimable:22874
mapped:14774 shmem:155 pagetables:355
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:388018 free_pcp:426 free_cma:0
Node 0 active_anon:66860kB inactive_anon:0kB active_file:103940kB inactive_file:50580kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:59096kB dirty:92kB writeback:0kB shmem:620kB writeback_tmp:0kB kernel_stack:3008kB pagetables:1420kB sec_pagetables:0kB all_unreclaimable? no
DMA free:566884kB boost:0kB min:3380kB low:4224kB high:5068kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:28264kB unevictable:0kB writepending:80kB present:786432kB managed:716496kB mlocked:0kB bounce:0kB free_pcp:1704kB local_pcp:1304kB free_cma:0kB
lowmem_reserve[]: 0 0 1184 0
DMA: 43*4kB (U) 29*8kB (UM) 21*16kB (UE) 41*32kB (UME) 9*64kB (UME) 6*128kB (UE) 3*256kB (UE) 3*512kB (UM) 6*1024kB (UME) 5*2048kB (UM) 133*4096kB (M) = 566852kB
38772 total pagecache pages
0 pages in swap cache
Free swap = 8388604kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
42060 pages reserved
16656 pages Vmalloc Total
6771 pages Vmalloc Used
Memory allocations:
124 MiB 31813 mm/readahead.c:187 func:ractl_alloc_folio
88.6 MiB 7937 mm/slub.c:2423 func:alloc_slab_page
50.3 MiB 12883 ./include/linux/highmem.h:229 func:vma_alloc_zeroed_movable_folio
26.5 MiB 6781 mm/filemap.c:1970 func:__filemap_get_folio
14.5 MiB 3714 mm/memory.c:1063 func:folio_prealloc
12.3 MiB 3280 mm/execmem.c:44 func:execmem_vmalloc
9.22 MiB 2360 mm/kasan/shadow.c:304 func:kasan_populate_vmalloc_pte
5.44 MiB 348 lib/stackdepot.c:627 func:stack_depot_save_flags
4.40 MiB 531 mm/slub.c:2425 func:alloc_slab_page
4.01 MiB 1025 mm/swap_cgroup.c:140 func:swap_cgroup_swapon
vmalloc_node_range for size 1196032 failed: Address range restricted to 0xf1000000 - 0xf5110000
vmalloc_node_range for size 1196032 failed: Address range restricted to 0xf1000000 - 0xf5110000
[...]
Looks like there would be still Vmalloc pages free though? 6771 used vs. 16656 total.
Also it seems more clear I get this vmalloc error at the module loading stage during bootup. For my original report the rather big btrfs module got pulled in at boot so I got the vmalloc failure instantly.
Now the machine boots up ok, but as soon as some module gets loaded, e.g. nfs via mounting a remote directory I get the failure. This is the case for both INLINE_KASAN and OUTLINE_KASAN. Maybe for my original report the btrfs module was not loaded for my OUTLINE_KASAN boot so I got no hit early. Doesn't seem to be 100% deterministic what modules get loaded in what order.
But meanwhile below data can give more details about the vmalloc area.
1. cat /proc/vmallocinfo
2. cat /proc/meminfo
Did that, with "no_hash_pointers" used and for both INLINE and OUTLINE KASAN as suggested by Christophe. Both after I get the vmalloc failure after mounting a remote NFS directory. See attached files.
For completeness also the OUTLINE_KASAN vmalloc failure:
[...]
vmalloc_node_range for size 839680 failed: Address range restricted to 0xf1000000 - 0xf5110000
modprobe: vmalloc error: size 832096, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 0 UID: 0 PID: 1857 Comm: modprobe Tainted: G W 6.14.0-rc4-PMacG4-dirty #10
Tainted: [W]=WARN
Hardware name: PowerMac3,6 7455 0x80010303 PowerMac
Call Trace:
[f34db980] [c09e7f34] dump_stack_lvl+0x70/0x8c (unreliable)
[f34db9a0] [c02887a4] warn_alloc+0x154/0x208
[f34dba60] [c027a308] __vmalloc_node_range_noprof+0x144/0x530
[f34dbb10] [c027a770] __vmalloc_node_noprof+0x7c/0x84
[f34dbb50] [c032a0d4] kernel_read_file+0x314/0x440
[f34dbbf0] [c0102918] init_module_from_file+0xb0/0x108
[f34dbcd0] [c0102d7c] sys_finit_module+0x1d0/0x37c
[f34dbda0] [c00138b8] system_call_exception+0x154/0x1b8
[f34dbf30] [c001a1ac] ret_from_syscall+0x0/0x2c
--- interrupt: c00 at 0x491334
NIP: 00491334 LR: 00739340 CTR: 0041d944
REGS: f34dbf40 TRAP: 0c00 Tainted: G W (6.14.0-rc4-PMacG4-dirty)
MSR: 0000d032 <EE,PR,ME,IR,DR,RI> CR: 2822242c XER: 20000000
GPR00: 00000161 af942800 a7fae540 00000001 00746a48 00000000 af9427d5 0000007f
GPR08: 00000000 aea582d8 00000000 00000008 00000003 0075fe10 2422242c 00000000
GPR16: 00040000 00000000 00000000 00000000 00ee8610 00000000 00ee9a30 00000000
GPR24: 00ee8610 00000000 00746a48 00ee9330 00040000 00000000 00767aac 00ee8610
NIP [00491334] 0x491334
LR [00739340] 0x739340
Mem-Info:
active_anon:16937 inactive_anon:0 isolated_anon:0
active_file:25969 inactive_file:11637 isolated_file:0
unevictable:0 dirty:21 writeback:0
slab_reclaimable:2396 slab_unreclaimable:23326
mapped:14784 shmem:155 pagetables:353
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:392772 free_pcp:777 free_cma:0
Node 0 active_anon:67748kB inactive_anon:0kB active_file:103876kB inactive_file:46548kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:59136kB dirty:84kB writeback:0kB shmem:620kB writeback_tmp:0kB kernel_stack:3216kB pagetables:1412kB sec_pagetables:0kB all_unreclaimable? no
DMA free:576744kB boost:0kB min:3412kB low:4264kB high:5116kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:28408kB unevictable:0kB writepending:80kB present:786432kB managed:729040kB mlocked:0kB bounce:0kB free_pcp:3108kB local_pcp:1788kB free_cma:0kB
lowmem_reserve[]: 0 0 1184 0
DMA: 0*4kB 2*8kB (UM) 5*16kB (UM) 36*32kB (UM) 15*64kB (UME) 2*128kB (ME) 3*256kB (UM) 2*512kB (UM) 3*1024kB (ME) 4*2048kB (UM) 137*4096kB (M) = 576672kB
37789 total pagecache pages
0 pages in swap cache
Free swap = 8388604kB
Total swap = 8388604kB
524288 pages RAM
327680 pages HighMem/MovableOnly
38924 pages reserved
16656 pages Vmalloc Total
5814 pages Vmalloc Used
Memory allocations:
120 MiB 30794 mm/readahead.c:187 func:ractl_alloc_folio
90.4 MiB 8145 mm/slub.c:2423 func:alloc_slab_page
50.6 MiB 12941 ./include/linux/highmem.h:229 func:vma_alloc_zeroed_movable_folio
26.6 MiB 6819 mm/filemap.c:1970 func:__filemap_get_folio
15.0 MiB 3837 mm/memory.c:1063 func:folio_prealloc
8.71 MiB 2230 mm/kasan/shadow.c:304 func:kasan_populate_vmalloc_pte
8.34 MiB 2276 mm/execmem.c:44 func:execmem_vmalloc
5.14 MiB 329 lib/stackdepot.c:627 func:stack_depot_save_flags
4.42 MiB 531 mm/slub.c:2425 func:alloc_slab_page
4.01 MiB 1025 mm/swap_cgroup.c:140 func:swap_cgroup_swapon
vmalloc_node_range for size 839680 failed: Address range restricted to 0xf1000000 - 0xf5110000
vmalloc_node_range for size 839680 failed: Address range restricted to 0xf1000000 - 0xf5110000
[...]
Regards,
Erhard