After you tried the above patch instead of the problem patch,
do one more test and apply the below on current linux-next.
Please, say which of the patches makes your kernel bootable again.
After you tried the above patch instead of the problem patch,
do one more test and apply the below on current linux-next.
Please, say which of the patches makes your kernel bootable again.
@@ -334,7 +334,7 @@ static int memcg_expand_one_shrinker_map(struct mem_cgroup *memcg,if(!old)return0;-new=kvmalloc_node(sizeof(*new)+size,GFP_KERNEL,nid);+new=kmalloc_node(sizeof(*new)+size,GFP_KERNEL,nid);if(!new)return-ENOMEM;
@@ -378,7 +378,7 @@ static int memcg_alloc_shrinker_maps(struct mem_cgroup *memcg)mutex_lock(&memcg_shrinker_map_mutex);size=memcg_shrinker_map_size;for_each_node(nid){-map=kvzalloc_node(sizeof(*map)+size,GFP_KERNEL,nid);+map=kzalloc_node(sizeof(*map)+size,GFP_KERNEL,nid);if(!map){memcg_free_shrinker_maps(memcg);ret=-ENOMEM;
Hm, though vmalloc() itself should not bring a problem, since it's already used
in profile_init(), which goes before cgroup_init() in start_kernel(), and
CONFIG_PROFILING=y is enabled in every powerpc defconfig.
Ok, I'm waiting for results of your tests.
After you tried the above patch instead of the problem patch,
do one more test and apply the below on current linux-next.
Please, say which of the patches makes your kernel bootable again.
With this original patch I can boot the machine successfully.
Ok, thanks.
I think, there is no a problem in the commited patch, since mem_cgroup_css_alloc()
is called from the place, where any memory allocations have to be allowed. This
is one of the reason, memory_cgrp_subsys.early_init is 0, and all nodes allocations
should be availeble there.
The problem is not in vmalloc() itself, since the second patch with kmalloc_node()
also fails on your setup. Maybe, the reproduction depends on amount of allocated
memory. For me this looks like a problem in powerpc, but it would be interesting
to hear some comments from powerpc guys.
For now we may replace the commited patch with v2 (https://patchwork.kernel.org/patch/11360007/)
containing workaround, which we have in another alloc_mem_cgroup_per_node_info() allocations.
Kirill
After you tried the above patch instead of the problem patch,
do one more test and apply the below on current linux-next.
Please, say which of the patches makes your kernel bootable again.
With this incremental patch applied on top of current linux-next, machine fails to boot
Your calltrace points to a standard system call path. I do not see any
reason why that commit should cause any problems. Do you see the
same when applying the patch you managed to bisect to on top of Linus
tree? Just to rule out any other potential problems in linux-next?
This all smells like a corrupted slab allocator. Which allocator do
you use?
After you tried the above patch instead of the problem patch,
do one more test and apply the below on current linux-next.
Please, say which of the patches makes your kernel bootable again.
With this incremental patch applied on top of current linux-next, machine fails to boot
Your calltrace points to a standard system call path. I do not see any
reason why that commit should cause any problems. Do you see the
same when applying the patch you managed to bisect to on top of Linus
tree? Just to rule out any other potential problems in linux-next?
Yes, I can recreate the same problem with the patch applied on top of
5.6.0-rc2.
CONFIG_SLUB is enabled in my case. I have attached the .config.
The LPAR has 34GB of memory allocated.
[ 8.766078] BUG: Kernel NULL pointer dereference on read at 0x000073b0
[ 8.766083] Faulting instruction address: 0xc0000000003d38a4
[ 8.766089] Oops: Kernel access of bad area, sig: 11 [#1]
[ 8.766093] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
[ 8.766098] Modules linked in:
[ 8.766103] CPU: 12 PID: 1 Comm: systemd Not tainted 5.6.0-rc2-autotest+ #2
[ 8.766107] NIP: c0000000003d38a4 LR: c0000000003d3e44 CTR: 0000000000000000
[ 8.766113] REGS: c0000008b37836e0 TRAP: 0300 Not tainted (5.6.0-rc2-autotest+)
[ 8.766118] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 24004844 XER: 00000000
[ 8.766125] CFAR: c00000000000dec4 DAR: 00000000000073b0 DSISR: 40000000 IRQMASK: 1
[ 8.766125] GPR00: c0000000003d3e44 c0000008b3783970 c00000000155d500 c0000008b301f500
[ 8.766125] GPR04: 0000000000000dc0 0000000000000002 c0000000003443f8 c0000008bac98620
[ 8.766125] GPR08: 00000008b9bf0000 0000000000000001 0000000000000000 0000000000000000
[ 8.766125] GPR12: 0000000024004844 c00000001ec5d200 0000000000000000 0000000000000000
[ 8.766125] GPR16: c000000007be2048 c000000001595818 c000000001750c98 0000000000000002
[ 8.766125] GPR20: c000000001750ca8 c000000001624470 0000000fffffffe0 5deadbeef0000122
[ 8.766125] GPR24: 0000000000000001 0000000000000dc0 0000000000000002 c0000000003443f8
[ 8.766125] GPR28: c0000008b301f500 c0000008bac98620 0000000000000000 c00c000002286fc0
[ 8.766172] NIP [c0000000003d38a4] ___slab_alloc+0x1f4/0x760
[ 8.766177] LR [c0000000003d3e44] __slab_alloc+0x34/0x60
[ 8.766181] Call Trace:
[ 8.766184] [c0000008b3783970] [c0000000003d39e4] ___slab_alloc+0x334/0x760 (unreliable)
[ 8.766191] [c0000008b3783a50] [c0000000003d3e44] __slab_alloc+0x34/0x60
[ 8.766196] [c0000008b3783a80] [c0000000003d5250] __kmalloc_node+0x110/0x490
[ 8.766203] [c0000008b3783b00] [c0000000003443f8] kvmalloc_node+0x58/0x110
[ 8.766208] [c0000008b3783b40] [c0000000003fcf58] mem_cgroup_css_online+0x108/0x270
[ 8.766215] [c0000008b3783ba0] [c000000000236078] online_css+0x48/0xd0
[ 8.766220] [c0000008b3783bd0] [c00000000023eebc] cgroup_apply_control_enable+0x2ec/0x4d0
[ 8.766226] [c0000008b3783cb0] [c000000000242728] cgroup_mkdir+0x228/0x5f0
[ 8.766232] [c0000008b3783d20] [c00000000051ab48] kernfs_iop_mkdir+0xb8/0x170
[ 8.766238] [c0000008b3783d50] [c00000000043a7c0] vfs_mkdir+0x110/0x230
[ 8.766243] [c0000008b3783da0] [c00000000043e8a0] do_mkdirat+0xb0/0x1a0
[ 8.766249] [c0000008b3783e20] [c00000000000b278] system_call+0x5c/0x68
[ 8.766253] Instruction dump:
[ 8.766257] 7c421378 e95f0000 714a0001 4082fff0 4bffff64 60000000 60000000 faa10088
[ 8.766264] 3ea2000c 3ab56f70 7b4a1f24 7d55502a <e94a73b0> 2faa0000 409e0394 3d02002a
[ 8.766271] ---[ end trace d651c32e3d9219fb ]---
[ 8.768347]
[ 9.768359] Kernel panic - not syncing: Fatal exception
Thanks
-Sachin
After you tried the above patch instead of the problem patch,
do one more test and apply the below on current linux-next.
Please, say which of the patches makes your kernel bootable again.
With this incremental patch applied on top of current linux-next, machine fails to boot
Your calltrace points to a standard system call path. I do not see any
reason why that commit should cause any problems. Do you see the
same when applying the patch you managed to bisect to on top of Linus
tree? Just to rule out any other potential problems in linux-next?
Yes, I can recreate the same problem with the patch applied on top of
5.6.0-rc2.
And just to make sure. This was with http://lkml.kernel.org/r/fff0e636-4c36-ed10-281c-8cdb0687c839@virtuozzo.com
right?
If yes, is it possible that the specific node is somehow crippled (e.g.
some nodes don't have any memory and thus the allocator blows up)? In
other words what is the numa topology? (numactl -H)
If yes, is it possible that the specific node is somehow crippled (e.g.
some nodes don't have any memory and thus the allocator blows up)? In
other words what is the numa topology? (numactl -H)
If yes, is it possible that the specific node is somehow crippled (e.g.
some nodes don't have any memory and thus the allocator blows up)? In
other words what is the numa topology? (numactl -H)
Here is the o/p of numactl
# numactl -H
available: 2 nodes (0-1)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
OK, so what I expected. The node0 is memory less or simply not present
at all. Fun!
Anyway, I do not think it is expected that kmalloc_node just blows up
on those nodes. The page allocator simply falls back to the closest
node. Something for kmalloc maintainers I believe.
A short summary. kmalloc_node blows up when trying to allocate from a
memory less node.
From: Christopher Lameter <hidden> Date: 2020-02-22 03:38:12
On Tue, 18 Feb 2020, Michal Hocko wrote:
Anyway, I do not think it is expected that kmalloc_node just blows up
on those nodes. The page allocator simply falls back to the closest
node. Something for kmalloc maintainers I believe.
That is the case for an unconstrained allocation. kmalloc_node means that
you want memory from that node. And If there is no such node then it is an
error.
A short summary. kmalloc_node blows up when trying to allocate from a
memory less node.
Use kmalloc instead? And set a memory allocation policy?
From: Michal Hocko <mhocko@kernel.org> Date: 2020-02-24 08:58:16
On Sat 22-02-20 03:38:11, Cristopher Lameter wrote:
On Tue, 18 Feb 2020, Michal Hocko wrote:
quoted
Anyway, I do not think it is expected that kmalloc_node just blows up
on those nodes. The page allocator simply falls back to the closest
node. Something for kmalloc maintainers I believe.
That is the case for an unconstrained allocation. kmalloc_node means that
you want memory from that node. And If there is no such node then it is an
error.
Hmm, nasty. Is there any reason why kmalloc_node behaves differently
from the page allocator?
quoted
A short summary. kmalloc_node blows up when trying to allocate from a
memory less node.
Use kmalloc instead? And set a memory allocation policy?
The current code (memcg_expand_one_shrinker_map resp. memcg_alloc_shrinker_maps)
already use kvmalloc. Kirill's patch wanted to make those data structure
on the respective node and kvmalloc_node sounded like the right thing to
do. It comes as a surprise that the kernel simply blows up on a memory
less node rather than falling back to a close node gracefully. I suspect
this already happens when the target node is out of memory, right?
How would a memory allocation policy help in this case btw.?
--
Michal Hocko
SUSE Labs
From: Christopher Lameter <hidden> Date: 2020-02-26 18:25:30
On Mon, 24 Feb 2020, Michal Hocko wrote:
Hmm, nasty. Is there any reason why kmalloc_node behaves differently
from the page allocator?
The page allocator will do the same thing if you pass GFP_THISNODE and
insist on allocating memory from a node that does not exist.
quoted
quoted
A short summary. kmalloc_node blows up when trying to allocate from a
memory less node.
Use kmalloc instead? And set a memory allocation policy?
The current code (memcg_expand_one_shrinker_map resp. memcg_alloc_shrinker_maps)
already use kvmalloc. Kirill's patch wanted to make those data structure
on the respective node and kvmalloc_node sounded like the right thing to
do. It comes as a surprise that the kernel simply blows up on a memory
less node rather than falling back to a close node gracefully. I suspect
this already happens when the target node is out of memory, right?
No. If the target node is out of memory then direct reclaim is going to be
invovked.
How would a memory allocation policy help in this case btw.?
From: Michal Hocko <mhocko@kernel.org> Date: 2020-02-26 18:41:56
On Wed 26-02-20 18:25:28, Cristopher Lameter wrote:
On Mon, 24 Feb 2020, Michal Hocko wrote:
quoted
Hmm, nasty. Is there any reason why kmalloc_node behaves differently
from the page allocator?
The page allocator will do the same thing if you pass GFP_THISNODE and
insist on allocating memory from a node that does not exist.
I do not think that the page allocator would blow up even with
GFP_THISNODE. The allocation would just fail on memory less node.
Besides that kmalloc_node shouldn't really have an implicit GFP_THISNODE
semantic right? At least I do not see anything like that documented
anywhere.
--
Michal Hocko
SUSE Labs
From: Christopher Lameter <hidden> Date: 2020-02-26 18:44:15
On Wed, 26 Feb 2020, Michal Hocko wrote:
Besides that kmalloc_node shouldn't really have an implicit GFP_THISNODE
semantic right? At least I do not see anything like that documented
anywhere.
Kmalloc_node does not support memory policies etc. Only kmalloc does.
kmalloc_node is mostly used by subsystems that have determined the active
nodes and want a targeted allocation on those nodes.
From: Michal Hocko <mhocko@kernel.org> Date: 2020-02-26 19:02:14
On Wed 26-02-20 18:44:13, Cristopher Lameter wrote:
On Wed, 26 Feb 2020, Michal Hocko wrote:
quoted
Besides that kmalloc_node shouldn't really have an implicit GFP_THISNODE
semantic right? At least I do not see anything like that documented
anywhere.
Kmalloc_node does not support memory policies etc. Only kmalloc does.
kmalloc_node is mostly used by subsystems that have determined the active
nodes and want a targeted allocation on those nodes.
I am sorry but I have hard time to follow your responses here. They open
more questions than they answer for me. The primary point here is that
kmalloc_node on a memory less node blows up and panics the kernel. I
strongly believe this is a bug. We cannot really make all callers of
kmalloc_node and co. to be hotplug aware.
Another question is the semantic of kmalloc_node when the node cannot
satisfy the request. I have always thought that the allocation would
simply fall back to any other node unless __GFP_THISNODE is explicitly
specified.
--
Michal Hocko
SUSE Labs
From: David Rientjes <rientjes@google.com> Date: 2020-02-26 20:32:00
On Wed, 26 Feb 2020, Michal Hocko wrote:
On Wed 26-02-20 18:44:13, Cristopher Lameter wrote:
quoted
On Wed, 26 Feb 2020, Michal Hocko wrote:
quoted
Besides that kmalloc_node shouldn't really have an implicit GFP_THISNODE
semantic right? At least I do not see anything like that documented
anywhere.
Kmalloc_node does not support memory policies etc. Only kmalloc does.
kmalloc_node is mostly used by subsystems that have determined the active
nodes and want a targeted allocation on those nodes.
I am sorry but I have hard time to follow your responses here. They open
more questions than they answer for me. The primary point here is that
kmalloc_node on a memory less node blows up and panics the kernel. I
strongly believe this is a bug. We cannot really make all callers of
kmalloc_node and co. to be hotplug aware.
Another question is the semantic of kmalloc_node when the node cannot
satisfy the request. I have always thought that the allocation would
simply fall back to any other node unless __GFP_THISNODE is explicitly
specified.
Am I right in classifying this as a trade-off between an
unlikely(!node_state(nid, N_MEMORY)) directly in kmalloc_node() vs fixing
up a caller passing a memoryless nid?
Seems like we wouldn't want to penalize kmalloc_node() for making such a
check for 99.99% of allocators that don't need it and would rather do a
node_to_mem_node(nid) or numa_mem_id() in the caller?
From: Michal Hocko <mhocko@kernel.org> Date: 2020-02-26 20:52:20
On Wed 26-02-20 12:31:56, David Rientjes wrote:
On Wed, 26 Feb 2020, Michal Hocko wrote:
quoted
On Wed 26-02-20 18:44:13, Cristopher Lameter wrote:
quoted
On Wed, 26 Feb 2020, Michal Hocko wrote:
quoted
Besides that kmalloc_node shouldn't really have an implicit GFP_THISNODE
semantic right? At least I do not see anything like that documented
anywhere.
Kmalloc_node does not support memory policies etc. Only kmalloc does.
kmalloc_node is mostly used by subsystems that have determined the active
nodes and want a targeted allocation on those nodes.
I am sorry but I have hard time to follow your responses here. They open
more questions than they answer for me. The primary point here is that
kmalloc_node on a memory less node blows up and panics the kernel. I
strongly believe this is a bug. We cannot really make all callers of
kmalloc_node and co. to be hotplug aware.
Another question is the semantic of kmalloc_node when the node cannot
satisfy the request. I have always thought that the allocation would
simply fall back to any other node unless __GFP_THISNODE is explicitly
specified.
Am I right in classifying this as a trade-off between an
unlikely(!node_state(nid, N_MEMORY)) directly in kmalloc_node() vs fixing
up a caller passing a memoryless nid?
The thing is that any check for node online/populated followed by the
allocation is inherently racy without using memory hotplug locking
around that and I am pretty sure this is a step into a wrong direction.
Is there any problem to initialize slub internal data structures for all
possible nodes? This wouldn't require any checks into hot paths.
--
Michal Hocko
SUSE Labs
On Wed 26-02-20 18:25:28, Cristopher Lameter wrote:
quoted
On Mon, 24 Feb 2020, Michal Hocko wrote:
quoted
Hmm, nasty. Is there any reason why kmalloc_node behaves differently
from the page allocator?
The page allocator will do the same thing if you pass GFP_THISNODE and
insist on allocating memory from a node that does not exist.
I do not think that the page allocator would blow up even with
GFP_THISNODE. The allocation would just fail on memory less node.
Besides that kmalloc_node shouldn't really have an implicit GFP_THISNODE
semantic right? At least I do not see anything like that documented
anywhere.
Seems like SLAB at least behaves like the page allocator. See
____cache_alloc_node() where it basically does:
page = cache_grow_begin(cachep, gfp_exact_node(flags), nodeid);
...
if (!page)
fallback_alloc(cachep, flags)
gfp_exact_node() adds __GFP_THISNODE among other things, so the initial
attempt does try to stick only to the given node. But fallback_alloc()
doesn't. In fact, even if kmalloc_node() was called with __GFP_THISNODE
then it wouldn't work as intended, as fallback_alloc() doesn't get the
nodeid, but instead will use numa_mem_id(). That part could probably be
improved.
SLUB's ___slab_alloc() has for example this:
if (node != NUMA_NO_NODE && !node_present_pages(node))
searchnode = node_to_mem_node(node);
That's from Joonsoo's 2014 commit a561ce00b09e ("slub: fall back to
node_to_mem_node() node if allocating on memoryless node"), suggesting
that the scenario in this bug report should work. Perhaps it just got
broken unintentionally later.
And AFAICS the whole path leading to alloc_slab_page() also doesn't add
__GFP_THISNODE, but will keep it if caller passed it, and ultimately it
does:
if (node == NUMA_NO_NODE)
page = alloc_pages(flags, order);
else
page = __alloc_pages_node(node, flags, order);
So yeah looks like SLUB's kmalloc_node() is supposed to behave like the
page allocator's __alloc_pages_node() and respect __GFP_THISNODE but not
enforce it by itself. There's probably just some missing data structure
initialization somewhere right now for memoryless nodes.
if (node == NUMA_NO_NODE)
page = alloc_pages(flags, order);
else
page = __alloc_pages_node(node, flags, order);
So yeah looks like SLUB's kmalloc_node() is supposed to behave like the
page allocator's __alloc_pages_node() and respect __GFP_THISNODE but not
enforce it by itself. There's probably just some missing data structure
initialization somewhere right now for memoryless nodes.
Upon more digging, I think the problem could manifest if
node_to_mem_node(0) (_node_numa_mem_[0]) returned 0 instead of 1,
because it wasn't initialized properly for a memoryless node. Can you
e.g. print it somewhere?
From: Michal Hocko <mhocko@kernel.org> Date: 2020-02-27 12:03:04
On Wed 26-02-20 22:45:52, Vlastimil Babka wrote:
On 2/26/20 7:41 PM, Michal Hocko wrote:
quoted
On Wed 26-02-20 18:25:28, Cristopher Lameter wrote:
quoted
On Mon, 24 Feb 2020, Michal Hocko wrote:
quoted
Hmm, nasty. Is there any reason why kmalloc_node behaves differently
from the page allocator?
The page allocator will do the same thing if you pass GFP_THISNODE and
insist on allocating memory from a node that does not exist.
I do not think that the page allocator would blow up even with
GFP_THISNODE. The allocation would just fail on memory less node.
Besides that kmalloc_node shouldn't really have an implicit GFP_THISNODE
semantic right? At least I do not see anything like that documented
anywhere.
Seems like SLAB at least behaves like the page allocator. See
____cache_alloc_node() where it basically does:
page = cache_grow_begin(cachep, gfp_exact_node(flags), nodeid);
...
if (!page)
fallback_alloc(cachep, flags)
gfp_exact_node() adds __GFP_THISNODE among other things, so the initial
attempt does try to stick only to the given node. But fallback_alloc()
doesn't. In fact, even if kmalloc_node() was called with __GFP_THISNODE
then it wouldn't work as intended, as fallback_alloc() doesn't get the
nodeid, but instead will use numa_mem_id(). That part could probably be
improved.
SLUB's ___slab_alloc() has for example this:
if (node != NUMA_NO_NODE && !node_present_pages(node))
Hmm, just a quick note. Shouldn't this be node_managed_pages? In most
cases the difference is negligible but I can imagine crazy setups where
all present pages are simply consumed.
searchnode = node_to_mem_node(node);
That's from Joonsoo's 2014 commit a561ce00b09e ("slub: fall back to
node_to_mem_node() node if allocating on memoryless node"), suggesting
that the scenario in this bug report should work. Perhaps it just got
broken unintentionally later.
A very good reference. Thanks!
And AFAICS the whole path leading to alloc_slab_page() also doesn't add
__GFP_THISNODE, but will keep it if caller passed it, and ultimately it
does:
if (node == NUMA_NO_NODE)
page = alloc_pages(flags, order);
else
page = __alloc_pages_node(node, flags, order);
So yeah looks like SLUB's kmalloc_node() is supposed to behave like the
page allocator's __alloc_pages_node() and respect __GFP_THISNODE but not
enforce it by itself. There's probably just some missing data structure
initialization somewhere right now for memoryless nodes.
Thanks for the confirmation!
--
Michal Hocko
SUSE Labs
From: Michal Hocko <mhocko@kernel.org> Date: 2020-02-27 12:12:18
On Wed 26-02-20 23:29:24, Vlastimil Babka wrote:
On 2/26/20 10:45 PM, Vlastimil Babka wrote:
quoted
if (node == NUMA_NO_NODE)
page = alloc_pages(flags, order);
else
page = __alloc_pages_node(node, flags, order);
So yeah looks like SLUB's kmalloc_node() is supposed to behave like the
page allocator's __alloc_pages_node() and respect __GFP_THISNODE but not
enforce it by itself. There's probably just some missing data structure
initialization somewhere right now for memoryless nodes.
Upon more digging, I think the problem could manifest if
node_to_mem_node(0) (_node_numa_mem_[0]) returned 0 instead of 1,
because it wasn't initialized properly for a memoryless node. Can you
e.g. print it somewhere?
{
this_cpu_write(_numa_mem_, node);
_node_numa_mem_[numa_node_id()] = node;
+ pr_info("%s %d -> %d\n", __FUNCTION__, numa_node_id(), node);
+ dump_stack();
}
#endif
Btw. it would be also helpful to get
`faddr2line ___slab_alloc+0x334' from your kernel Sachin.
[linux-next]# ./scripts/faddr2line ./vmlinux ___slab_alloc+0x334
___slab_alloc+0x334/0x760:
new_slab_objects at mm/slub.c:2478
(inlined by) ___slab_alloc at mm/slub.c:2628
[linux-next]#
I have also attached boot log with a kernel that include about change.
I see the following o/p during boot:
[ 0.005269] set_numa_mem 1 -> 1
[ 0.005270] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 5.6.0-rc3-next-20200227-autotest+ #6
[ 0.005271] Call Trace:
[ 0.005272] [c0000008b37dfe80] [c000000000b5d948] dump_stack+0xbc/0x104 (unreliable)
[ 0.005274] [c0000008b37dfec0] [c000000000059320] start_secondary+0x600/0x6e0
[ 0.005277] [c0000008b37dff90] [c00000000000ac54] start_secondary_prolog+0x10/0x14
Thanks
-Sachin
{
this_cpu_write(_numa_mem_, node);
_node_numa_mem_[numa_node_id()] = node;
+ pr_info("%s %d -> %d\n", __FUNCTION__, numa_node_id(), node);
+ dump_stack();
}
#endif
Btw. it would be also helpful to get
`faddr2line ___slab_alloc+0x334' from your kernel Sachin.
[linux-next]# ./scripts/faddr2line ./vmlinux ___slab_alloc+0x334
___slab_alloc+0x334/0x760:
new_slab_objects at mm/slub.c:2478
(inlined by) ___slab_alloc at mm/slub.c:2628
[linux-next]#
Hmm that doesn't look relevant, but that address was marked as unreliable, no?
Don't we actually need this one?
[ 8.768727] NIP [c0000000003d55f4] ___slab_alloc+0x1f4/0x760
I have also attached boot log with a kernel that include about change.
I see the following o/p during boot:
[ 0.005269] set_numa_mem 1 -> 1
So there's no "set_numa_mem 0 -> X", specifically not
"set_numa_mem 0 -> 1" which I would have expected. That seems to confirm my
suspicion that the arch code doesn't set up the memoryless node 0 properly.
{
this_cpu_write(_numa_mem_, node);
_node_numa_mem_[numa_node_id()] = node;
+ pr_info("%s %d -> %d\n", __FUNCTION__, numa_node_id(), node);
+ dump_stack();
}
#endif
Btw. it would be also helpful to get
`faddr2line ___slab_alloc+0x334' from your kernel Sachin.
[linux-next]# ./scripts/faddr2line ./vmlinux ___slab_alloc+0x334
___slab_alloc+0x334/0x760:
new_slab_objects at mm/slub.c:2478
(inlined by) ___slab_alloc at mm/slub.c:2628
[linux-next]#
Hmm that doesn't look relevant, but that address was marked as unreliable, no?
Don't we actually need this one?
[ 8.768727] NIP [c0000000003d55f4] ___slab_alloc+0x1f4/0x760
quoted
I have also attached boot log with a kernel that include about change.
I see the following o/p during boot:
[ 0.005269] set_numa_mem 1 -> 1
So there's no "set_numa_mem 0 -> X", specifically not
"set_numa_mem 0 -> 1" which I would have expected. That seems to confirm my
suspicion that the arch code doesn't set up the memoryless node 0 properly.
But that doesn't work because smp_prepare_boot_cpu() is called too
early:
asmlinkage __visible void __init start_kernel(void)
{
...
smp_prepare_boot_cpu(); /* arch-specific boot-cpu hooks */
boot_cpu_hotplug_init();
build_all_zonelists(NULL);
And local_memory_node() uses first_zones_zonelist() which doesn't work
prior to build_all_zonelists() being called.
The patch below might work. Sachin can you test this? I tried faking up
a system with a memoryless node zero but couldn't get it to even start
booting.
cheers
From: Sachin Sant <hidden> Date: 2020-03-12 16:56:41
The patch below might work. Sachin can you test this? I tried faking up
a system with a memoryless node zero but couldn't get it to even start
booting.
The patch did not help. The kernel crashed during
the boot with the same call trace.
BUG_ON() introduced with the patch was not triggered.
Thanks
-Sachin
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2020-03-13 10:48:11
Sachin Sant [off-list ref] writes:
quoted
The patch below might work. Sachin can you test this? I tried faking up
a system with a memoryless node zero but couldn't get it to even start
booting.
The patch did not help. The kernel crashed during
the boot with the same call trace.
BUG_ON() introduced with the patch was not triggered.
OK, that's weird.
I eventually managed to get a memoryless node going in sim, and it
appears to work there.
eg in dmesg:
[ 0.000000][ T0] numa: NODE_DATA [mem 0x2000fffa2f80-0x2000fffa7fff]
[ 0.000000][ T0] numa: NODE_DATA(0) on node 1
[ 0.000000][ T0] numa: NODE_DATA [mem 0x2000fff9df00-0x2000fffa2f7f]
...
[ 0.000000][ T0] Early memory node ranges
[ 0.000000][ T0] node 1: [mem 0x0000000000000000-0x00000000ffffffff]
[ 0.000000][ T0] node 1: [mem 0x0000200000000000-0x00002000ffffffff]
[ 0.000000][ T0] Could not find start_pfn for node 0
[ 0.000000][ T0] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[ 0.000000][ T0] On node 0 totalpages: 0
[ 0.000000][ T0] Initmem setup node 1 [mem 0x0000000000000000-0x00002000ffffffff]
[ 0.000000][ T0] On node 1 totalpages: 131072
# dmesg | grep set_numa
[ 0.000000][ T0] set_numa_mem: mem node for 0 = 1
[ 0.005654][ T0] set_numa_mem: mem node for 1 = 1
So is the problem more than just node zero having no memory?
cheers
* Michael Ellerman [off-list ref] [2020-03-13 21:48:06]:
Sachin Sant [off-list ref] writes:
quoted
quoted
The patch below might work. Sachin can you test this? I tried faking up
a system with a memoryless node zero but couldn't get it to even start
booting.
The patch did not help. The kernel crashed during
the boot with the same call trace.
BUG_ON() introduced with the patch was not triggered.
OK, that's weird.
I eventually managed to get a memoryless node going in sim, and it
appears to work there.
eg in dmesg:
[ 0.000000][ T0] numa: NODE_DATA [mem 0x2000fffa2f80-0x2000fffa7fff]
[ 0.000000][ T0] numa: NODE_DATA(0) on node 1
[ 0.000000][ T0] numa: NODE_DATA [mem 0x2000fff9df00-0x2000fffa2f7f]
...
[ 0.000000][ T0] Early memory node ranges
[ 0.000000][ T0] node 1: [mem 0x0000000000000000-0x00000000ffffffff]
[ 0.000000][ T0] node 1: [mem 0x0000200000000000-0x00002000ffffffff]
[ 0.000000][ T0] Could not find start_pfn for node 0
[ 0.000000][ T0] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[ 0.000000][ T0] On node 0 totalpages: 0
[ 0.000000][ T0] Initmem setup node 1 [mem 0x0000000000000000-0x00002000ffffffff]
[ 0.000000][ T0] On node 1 totalpages: 131072
# dmesg | grep set_numa
[ 0.000000][ T0] set_numa_mem: mem node for 0 = 1
[ 0.005654][ T0] set_numa_mem: mem node for 1 = 1
So is the problem more than just node zero having no memory?
The problem would happen with possible nodes which are not yet present. i.e
no cpus, no memory attached to those nodes.
Please look at
http://lore.kernel.org/lkml/20200312131438.GB3277@linux.vnet.ibm.com/t/#u
for more details.
The summary being: pgdat/Node_Data for such nodes is not allocated. Hence
the node_present_pages(nid) called where nid is a possible but not yet
present node fails. Currently node_present_pages(nid) and node_to_mem_node
don't seem to be equipped to handle possible but not present nodes.
* Michael Ellerman [off-list ref] [2020-03-13 21:48:06]:
quoted
Sachin Sant [off-list ref] writes:
quoted
quoted
The patch below might work. Sachin can you test this? I tried faking up
a system with a memoryless node zero but couldn't get it to even start
booting.
The patch did not help. The kernel crashed during
the boot with the same call trace.
BUG_ON() introduced with the patch was not triggered.
OK, that's weird.
I eventually managed to get a memoryless node going in sim, and it
appears to work there.
eg in dmesg:
[ 0.000000][ T0] numa: NODE_DATA [mem 0x2000fffa2f80-0x2000fffa7fff]
[ 0.000000][ T0] numa: NODE_DATA(0) on node 1
[ 0.000000][ T0] numa: NODE_DATA [mem 0x2000fff9df00-0x2000fffa2f7f]
...
[ 0.000000][ T0] Early memory node ranges
[ 0.000000][ T0] node 1: [mem 0x0000000000000000-0x00000000ffffffff]
[ 0.000000][ T0] node 1: [mem 0x0000200000000000-0x00002000ffffffff]
[ 0.000000][ T0] Could not find start_pfn for node 0
[ 0.000000][ T0] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[ 0.000000][ T0] On node 0 totalpages: 0
[ 0.000000][ T0] Initmem setup node 1 [mem 0x0000000000000000-0x00002000ffffffff]
[ 0.000000][ T0] On node 1 totalpages: 131072
# dmesg | grep set_numa
[ 0.000000][ T0] set_numa_mem: mem node for 0 = 1
[ 0.005654][ T0] set_numa_mem: mem node for 1 = 1
So is the problem more than just node zero having no memory?
The problem would happen with possible nodes which are not yet present. i.e
no cpus, no memory attached to those nodes.
Please look at
http://lore.kernel.org/lkml/20200312131438.GB3277@linux.vnet.ibm.com/t/#u
for more details.
The summary being: pgdat/Node_Data for such nodes is not allocated. Hence
Michael's log shows that his pgdat is still allocated. But perhaps Sachin had
also your 3 patches from the other thread applied, in addition to Michael's
patch. So in his case pgdat for node 0 would indeed be no longer allocated, and
thus SLUB code was crashing in node_present_pages() instead.
the node_present_pages(nid) called where nid is a possible but not yet
present node fails. Currently node_present_pages(nid) and node_to_mem_node
don't seem to be equipped to handle possible but not present nodes.
From: Sachin Sant <hidden> Date: 2020-03-15 01:30:58
On 13-Mar-2020, at 5:05 PM, Vlastimil Babka [off-list ref] wrote:
On 3/13/20 12:12 PM, Srikar Dronamraju wrote:
quoted
* Michael Ellerman [off-list ref] [2020-03-13 21:48:06]:
quoted
Sachin Sant [off-list ref] writes:
quoted
quoted
The patch below might work. Sachin can you test this? I tried faking up
a system with a memoryless node zero but couldn't get it to even start
booting.
The patch did not help. The kernel crashed during
the boot with the same call trace.
BUG_ON() introduced with the patch was not triggered.
OK, that's weird.
I eventually managed to get a memoryless node going in sim, and it
appears to work there.
eg in dmesg:
[ 0.000000][ T0] numa: NODE_DATA [mem 0x2000fffa2f80-0x2000fffa7fff]
[ 0.000000][ T0] numa: NODE_DATA(0) on node 1
[ 0.000000][ T0] numa: NODE_DATA [mem 0x2000fff9df00-0x2000fffa2f7f]
...
[ 0.000000][ T0] Early memory node ranges
[ 0.000000][ T0] node 1: [mem 0x0000000000000000-0x00000000ffffffff]
[ 0.000000][ T0] node 1: [mem 0x0000200000000000-0x00002000ffffffff]
[ 0.000000][ T0] Could not find start_pfn for node 0
[ 0.000000][ T0] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[ 0.000000][ T0] On node 0 totalpages: 0
[ 0.000000][ T0] Initmem setup node 1 [mem 0x0000000000000000-0x00002000ffffffff]
[ 0.000000][ T0] On node 1 totalpages: 131072
# dmesg | grep set_numa
[ 0.000000][ T0] set_numa_mem: mem node for 0 = 1
[ 0.005654][ T0] set_numa_mem: mem node for 1 = 1
So is the problem more than just node zero having no memory?
I tried with just the patch Michael suggested on top of March 13 next tree.
I still see the same failure. Here is a snippet from the log
[ 0.000000] numa: NODE_DATA [mem 0x8bfedc900-0x8bfee3fff]
[ 0.000000] numa: NODE_DATA(0) on node 1
[ 0.000000] numa: NODE_DATA [mem 0x8bfed5200-0x8bfedc8ff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] link-stack-flush: software flush enabled.
[ 0.000000] count-cache-flush: software flush disabled.
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:0 psize:0 block size:8
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:0 psize:2 block size:8
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:0 psize:10 block size:8
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:2 psize:2 block size:8
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:2 psize:10 block size:8
[ 0.000000] PPC64 nvram contains 15360 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x00000008bfffffff]
[ 0.000000] Device empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 1: [mem 0x0000000000000000-0x00000008bfffffff]
[ 0.000000] Could not find start_pfn for node 0
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
[ 0.000000] Initmem setup node 1 [mem 0x0000000000000000-0x00000008bfffffff]
[ 0.000000] percpu: Embedded 11 pages/cpu s624024 r0 d96872 u1048576
[ 0.000000] Built 2 zonelists, mobility grouping on. Total pages: 572880
Have attached the complete boot log.
Thanks
-Sachin
Calling a kmalloc_node on a possible node which is not yet onlined can
lead to panic. Currently node_present_pages() doesn't verify the node is
online before accessing the pgdat for the node. However pgdat struct may
not be available resulting in a crash.
NIP [c0000000003d55f4] ___slab_alloc+0x1f4/0x760
LR [c0000000003d5b94] __slab_alloc+0x34/0x60
Call Trace:
[c0000008b3783960] [c0000000003d5734] ___slab_alloc+0x334/0x760 (unreliable)
[c0000008b3783a40] [c0000000003d5b94] __slab_alloc+0x34/0x60
[c0000008b3783a70] [c0000000003d6fa0] __kmalloc_node+0x110/0x490
[c0000008b3783af0] [c0000000003443d8] kvmalloc_node+0x58/0x110
[c0000008b3783b30] [c0000000003fee38] mem_cgroup_css_online+0x108/0x270
[c0000008b3783b90] [c000000000235aa8] online_css+0x48/0xd0
[c0000008b3783bc0] [c00000000023eaec] cgroup_apply_control_enable+0x2ec/0x4d0
[c0000008b3783ca0] [c000000000242318] cgroup_mkdir+0x228/0x5f0
[c0000008b3783d10] [c00000000051e170] kernfs_iop_mkdir+0x90/0xf0
[c0000008b3783d50] [c00000000043dc00] vfs_mkdir+0x110/0x230
[c0000008b3783da0] [c000000000441c90] do_mkdirat+0xb0/0x1a0
[c0000008b3783e20] [c00000000000b278] system_call+0x5c/0x68
Fix this by verifying the node is online before accessing the pgdat
structure. Fix the same for node_spanned_pages() too.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-mm@kvack.org
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Sachin Sant <redacted>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Christopher Lameter <redacted>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: Joonsoo Kim <redacted>
Cc: Kirill Tkhai <redacted>
Cc: Vlastimil Babka <redacted>
Cc: Srikar Dronamraju <redacted>
Cc: Bharata B Rao <redacted>
Reported-by: Sachin Sant <redacted>
Tested-by: Sachin Sant <redacted>
Signed-off-by: Srikar Dronamraju <redacted>
---
include/linux/mmzone.h | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
Currently while allocating a slab for a offline node, we use its
associated node_numa_mem to search for a partial slab. If we don't find
a partial slab, we try allocating a slab from the offline node using
__alloc_pages_node. However this is bound to fail.
NIP [c00000000039a300] __alloc_pages_nodemask+0x130/0x3b0
LR [c00000000039a3c4] __alloc_pages_nodemask+0x1f4/0x3b0
Call Trace:
[c0000008b36837f0] [c00000000039a3b4] __alloc_pages_nodemask+0x1e4/0x3b0 (unreliable)
[c0000008b3683870] [c0000000003d1ff8] new_slab+0x128/0xcf0
[c0000008b3683950] [c0000000003d6060] ___slab_alloc+0x410/0x820
[c0000008b3683a40] [c0000000003d64a4] __slab_alloc+0x34/0x60
[c0000008b3683a70] [c0000000003d78b0] __kmalloc_node+0x110/0x490
[c0000008b3683af0] [c000000000343a08] kvmalloc_node+0x58/0x110
[c0000008b3683b30] [c0000000003ffd44] mem_cgroup_css_online+0x104/0x270
[c0000008b3683b90] [c000000000234e08] online_css+0x48/0xd0
[c0000008b3683bc0] [c00000000023dedc] cgroup_apply_control_enable+0x2ec/0x4d0
[c0000008b3683ca0] [c0000000002416f8] cgroup_mkdir+0x228/0x5f0
[c0000008b3683d10] [c000000000520360] kernfs_iop_mkdir+0x90/0xf0
[c0000008b3683d50] [c00000000043e400] vfs_mkdir+0x110/0x230
[c0000008b3683da0] [c000000000441ee0] do_mkdirat+0xb0/0x1a0
[c0000008b3683e20] [c00000000000b278] system_call+0x5c/0x68
Mitigate this by allocating the new slab from the node_numa_mem.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-mm@kvack.org
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Sachin Sant <redacted>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Christopher Lameter <redacted>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: Joonsoo Kim <redacted>
Cc: Kirill Tkhai <redacted>
Cc: Vlastimil Babka <redacted>
Cc: Srikar Dronamraju <redacted>
Cc: Bharata B Rao <redacted>
Reported-by: Sachin Sant <redacted>
Tested-by: Sachin Sant <redacted>
Signed-off-by: Srikar Dronamraju <redacted>
---
mm/slub.c | 19 ++++++++-----------
1 file changed, 8 insertions(+), 11 deletions(-)
For a memoryless or offline nodes, node_numa_mem refers to a N_MEMORY
fallback node. Currently kernel has an API set_numa_mem that sets
node_numa_mem for memoryless node. However this API cannot be used for
offline nodes. Hence all offline nodes will have their node_numa_mem set
to 0. However systems can themselves have node 0 as offline i.e
memoryless and cpuless at this time. In such cases,
node_to_mem_node() fails to provide a N_MEMORY fallback node.
Mitigate this by having a new API that sets the default node_numa_mem for
offline nodes to be first_memory_node.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-mm@kvack.org
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Sachin Sant <redacted>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Christopher Lameter <redacted>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: Joonsoo Kim <redacted>
Cc: Kirill Tkhai <redacted>
Cc: Vlastimil Babka <redacted>
Cc: Srikar Dronamraju <redacted>
Cc: Bharata B Rao <redacted>
Reported-by: Sachin Sant <redacted>
Tested-by: Sachin Sant <redacted>
Signed-off-by: Srikar Dronamraju <redacted>
---
include/asm-generic/topology.h | 3 +++
include/linux/topology.h | 7 +++++++
2 files changed, 10 insertions(+)
@@ -147,6 +147,13 @@ static inline int node_to_mem_node(int node)}#endif+#ifndef reset_numa_mem+staticinlinevoidreset_numa_mem(intnode)+{+_node_numa_mem_[node]=first_memory_node;+}+#endif+#ifndef numa_mem_id/* Returns the number of the nearest Node with memory */staticinlineintnuma_mem_id(void)
Currently while allocating a slab for a offline node, we use its
associated node_numa_mem to search for a partial slab. If we don't find
a partial slab, we try allocating a slab from the offline node using
__alloc_pages_node. However this is bound to fail.
NIP [c00000000039a300] __alloc_pages_nodemask+0x130/0x3b0
LR [c00000000039a3c4] __alloc_pages_nodemask+0x1f4/0x3b0
Call Trace:
[c0000008b36837f0] [c00000000039a3b4] __alloc_pages_nodemask+0x1e4/0x3b0 (unreliable)
[c0000008b3683870] [c0000000003d1ff8] new_slab+0x128/0xcf0
[c0000008b3683950] [c0000000003d6060] ___slab_alloc+0x410/0x820
[c0000008b3683a40] [c0000000003d64a4] __slab_alloc+0x34/0x60
[c0000008b3683a70] [c0000000003d78b0] __kmalloc_node+0x110/0x490
[c0000008b3683af0] [c000000000343a08] kvmalloc_node+0x58/0x110
[c0000008b3683b30] [c0000000003ffd44] mem_cgroup_css_online+0x104/0x270
[c0000008b3683b90] [c000000000234e08] online_css+0x48/0xd0
[c0000008b3683bc0] [c00000000023dedc] cgroup_apply_control_enable+0x2ec/0x4d0
[c0000008b3683ca0] [c0000000002416f8] cgroup_mkdir+0x228/0x5f0
[c0000008b3683d10] [c000000000520360] kernfs_iop_mkdir+0x90/0xf0
[c0000008b3683d50] [c00000000043e400] vfs_mkdir+0x110/0x230
[c0000008b3683da0] [c000000000441ee0] do_mkdirat+0xb0/0x1a0
[c0000008b3683e20] [c00000000000b278] system_call+0x5c/0x68
Mitigate this by allocating the new slab from the node_numa_mem.
Are you sure this is really needed and the other 3 patches are not enough for
the current SLUB code to work as needed? It seems you are changing the semantics
here...
If we indeed start having pgdat/NODE_DATA for even offline nodes as Michal
Hocko, then we may not this particular patch.
--
Thanks and Regards
Srikar Dronamraju
Currently while allocating a slab for a offline node, we use its
associated node_numa_mem to search for a partial slab. If we don't find
a partial slab, we try allocating a slab from the offline node using
__alloc_pages_node. However this is bound to fail.
NIP [c00000000039a300] __alloc_pages_nodemask+0x130/0x3b0
LR [c00000000039a3c4] __alloc_pages_nodemask+0x1f4/0x3b0
Call Trace:
[c0000008b36837f0] [c00000000039a3b4] __alloc_pages_nodemask+0x1e4/0x3b0 (unreliable)
[c0000008b3683870] [c0000000003d1ff8] new_slab+0x128/0xcf0
[c0000008b3683950] [c0000000003d6060] ___slab_alloc+0x410/0x820
[c0000008b3683a40] [c0000000003d64a4] __slab_alloc+0x34/0x60
[c0000008b3683a70] [c0000000003d78b0] __kmalloc_node+0x110/0x490
[c0000008b3683af0] [c000000000343a08] kvmalloc_node+0x58/0x110
[c0000008b3683b30] [c0000000003ffd44] mem_cgroup_css_online+0x104/0x270
[c0000008b3683b90] [c000000000234e08] online_css+0x48/0xd0
[c0000008b3683bc0] [c00000000023dedc] cgroup_apply_control_enable+0x2ec/0x4d0
[c0000008b3683ca0] [c0000000002416f8] cgroup_mkdir+0x228/0x5f0
[c0000008b3683d10] [c000000000520360] kernfs_iop_mkdir+0x90/0xf0
[c0000008b3683d50] [c00000000043e400] vfs_mkdir+0x110/0x230
[c0000008b3683da0] [c000000000441ee0] do_mkdirat+0xb0/0x1a0
[c0000008b3683e20] [c00000000000b278] system_call+0x5c/0x68
Mitigate this by allocating the new slab from the node_numa_mem.
Are you sure this is really needed and the other 3 patches are not enough for
the current SLUB code to work as needed? It seems you are changing the semantics
here...
The other 3 patches are not enough because we don't carry the searchnode
when the actual alloc_pages_node gets called.
With only the 3 patches, we see the above Panic, its signature is slightly
different from what Sachin first reported and which I have carried in 1st
patch.
Currently while allocating a slab for a offline node, we use its
associated node_numa_mem to search for a partial slab. If we don't find
a partial slab, we try allocating a slab from the offline node using
__alloc_pages_node. However this is bound to fail.
NIP [c00000000039a300] __alloc_pages_nodemask+0x130/0x3b0
LR [c00000000039a3c4] __alloc_pages_nodemask+0x1f4/0x3b0
Call Trace:
[c0000008b36837f0] [c00000000039a3b4] __alloc_pages_nodemask+0x1e4/0x3b0 (unreliable)
[c0000008b3683870] [c0000000003d1ff8] new_slab+0x128/0xcf0
[c0000008b3683950] [c0000000003d6060] ___slab_alloc+0x410/0x820
[c0000008b3683a40] [c0000000003d64a4] __slab_alloc+0x34/0x60
[c0000008b3683a70] [c0000000003d78b0] __kmalloc_node+0x110/0x490
[c0000008b3683af0] [c000000000343a08] kvmalloc_node+0x58/0x110
[c0000008b3683b30] [c0000000003ffd44] mem_cgroup_css_online+0x104/0x270
[c0000008b3683b90] [c000000000234e08] online_css+0x48/0xd0
[c0000008b3683bc0] [c00000000023dedc] cgroup_apply_control_enable+0x2ec/0x4d0
[c0000008b3683ca0] [c0000000002416f8] cgroup_mkdir+0x228/0x5f0
[c0000008b3683d10] [c000000000520360] kernfs_iop_mkdir+0x90/0xf0
[c0000008b3683d50] [c00000000043e400] vfs_mkdir+0x110/0x230
[c0000008b3683da0] [c000000000441ee0] do_mkdirat+0xb0/0x1a0
[c0000008b3683e20] [c00000000000b278] system_call+0x5c/0x68
Mitigate this by allocating the new slab from the node_numa_mem.
Are you sure this is really needed and the other 3 patches are not enough for
the current SLUB code to work as needed? It seems you are changing the semantics
here...
The other 3 patches are not enough because we don't carry the searchnode
when the actual alloc_pages_node gets called.
With only the 3 patches, we see the above Panic, its signature is slightly
different from what Sachin first reported and which I have carried in 1st
patch.
Currently fallback nodes for offline nodes aren't set. Hence by default
node 0 ends up being the default node. However node 0 might be offline.
Fix this by explicitly setting fallback node. Ensure first_memory_node
is set before kernel does explicit setting of fallback node.
arch/powerpc/mm/numa.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
- for_each_online_node(node) {
+ for_each_node(node) {
+ /*
+ * For all possible but not yet online nodes, ensure their
+ * node_numa_mem is set correctly so that kmalloc_node works
+ * for such nodes.
+ */
+ if (!node_online(node)) {
Change the above line to like below:
+ if (!node_state(node, N_MEMORY)) {
Mitigate this by allocating the new slab from the node_numa_mem.
Are you sure this is really needed and the other 3 patches are not enough for
the current SLUB code to work as needed? It seems you are changing the semantics
here...
The other 3 patches are not enough because we don't carry the searchnode
when the actual alloc_pages_node gets called.
With only the 3 patches, we see the above Panic, its signature is slightly
different from what Sachin first reported and which I have carried in 1st
patch.
Ah, I see. So that's the missing pgdat after your series [1] right?
Yes the pgdat would be missing after my cpuless, memoryless node patchset.
However..
That sounds like an argument for Michal's suggestions that pgdats exist and have
correctly populated zonelists for all possible nodes.
Only the first patch in this series would be affected by pgdat existing or
not. Even if the pgdat existed, the NODE_DATA[nid]->node_present_pages
would be 0. Right? So it would look at node_to_mem_node(). And since node 0 is
cpuless it would return 0. If we pass this node 0 (which is memoryless/cpuless) to
alloc_pages_node. Please note I am only setting node_numa_mem only
for offline nodes. However we could change this to set for all offline and
memoryless nodes.
Mitigate this by allocating the new slab from the node_numa_mem.
Are you sure this is really needed and the other 3 patches are not enough for
the current SLUB code to work as needed? It seems you are changing the semantics
here...
The other 3 patches are not enough because we don't carry the searchnode
when the actual alloc_pages_node gets called.
With only the 3 patches, we see the above Panic, its signature is slightly
different from what Sachin first reported and which I have carried in 1st
patch.
Ah, I see. So that's the missing pgdat after your series [1] right?
Yes the pgdat would be missing after my cpuless, memoryless node patchset.
However..
quoted
That sounds like an argument for Michal's suggestions that pgdats exist and have
correctly populated zonelists for all possible nodes.
Only the first patch in this series would be affected by pgdat existing or
not. Even if the pgdat existed, the NODE_DATA[nid]->node_present_pages
would be 0. Right? So it would look at node_to_mem_node(). And since node 0 is
cpuless it would return 0.
I thought the point was to return 1 for node 0.
If we pass this node 0 (which is memoryless/cpuless) to
alloc_pages_node. Please note I am only setting node_numa_mem only
for offline nodes. However we could change this to set for all offline and
memoryless nodes.
That would indeed make sense.
But I guess that alloc_pages would still crash as the result of
numa_to_mem_node() is not passed down to alloc_pages() without this patch. In
__alloc_pages_node() we currently have "The node must be valid and online" so
offline nodes don't have zonelists. Either they get them, or we indeed need
something like this patch. But in order to not make get_any_partial() dead code,
the final replacement of invalid node with a valid one should be done in
alloc_slab_page() I guess?
I.e. here in this if(), now node will never equal NUMA_NO_NODE (thanks to the
hunk below), thus the get_any_partial() call becomes dead code?
Very true.
Would it be okay if we remove the node != NUMA_NO_NODE
if (object || node != NUMA_NO_NODE)
return object;
will now become
if (object)
return object;
--
Thanks and Regards
Srikar Dronamraju
If we pass this node 0 (which is memoryless/cpuless) to
alloc_pages_node. Please note I am only setting node_numa_mem only
for offline nodes. However we could change this to set for all offline and
memoryless nodes.
That would indeed make sense.
But I guess that alloc_pages would still crash as the result of
numa_to_mem_node() is not passed down to alloc_pages() without this patch. In
__alloc_pages_node() we currently have "The node must be valid and online" so
offline nodes don't have zonelists. Either they get them, or we indeed need
something like this patch. But in order to not make get_any_partial() dead code,
the final replacement of invalid node with a valid one should be done in
alloc_slab_page() I guess?
I am posting v2 with this change.
quoted
quoted
node_to_mem_node() could be just a shortcut for the first zone's node in the
zonelist, so that fallback follows the topology.