From: David Rientjes <rientjes@google.com> Date: 2014-03-12 02:02:20
On Tue, 11 Mar 2014, Nishanth Aravamudan wrote:
I have a P7 system that has no node0, but a node0 shows up in numactl
--hardware, which has no cpus and no memory (and no PCI devices):
numactl --hardware
available: 4 nodes (0-3)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 1 size: 0 MB
node 1 free: 0 MB
node 2 cpus:
node 2 size: 7935 MB
node 2 free: 7716 MB
node 3 cpus:
node 3 size: 8395 MB
node 3 free: 8015 MB
node distances:
node 0 1 2 3
0: 10 20 10 20
1: 20 10 20 20
2: 10 20 10 20
3: 20 20 20 10
This is because we statically initialize N_ONLINE to be [0] in
mm/page_alloc.c:
[N_ONLINE] = { { [0] = 1UL } },
I'm not sure what the architectural requirements are here, but at least
on this test system, removing this initialization, it boots fine and is
running. I've not yet tried stress tests, but it's survived the
beginnings of kernbench so far.
numactl --hardware
available: 3 nodes (1-3)
node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 1 size: 0 MB
node 1 free: 0 MB
node 2 cpus:
node 2 size: 7935 MB
node 2 free: 7479 MB
node 3 cpus:
node 3 size: 8396 MB
node 3 free: 8375 MB
node distances:
node 1 2 3
1: 10 20 20
2: 20 10 20
3: 20 20 10
Perhaps we could put in a ARCH_DOES_NOT_NEED_NODE0 and only define it on
powerpc for now, conditionalizing the above initialization on that?
I don't know if anything has recently changed in the past year or so, but
I've booted x86 machines with a hacked BIOS so that all memory on node 0
is hotpluggable and offline, so I believe this is possible on x86 as well.
On 11.03.2014 [19:02:17 -0700], David Rientjes wrote:
On Tue, 11 Mar 2014, Nishanth Aravamudan wrote:
quoted
I have a P7 system that has no node0, but a node0 shows up in numactl
--hardware, which has no cpus and no memory (and no PCI devices):
numactl --hardware
available: 4 nodes (0-3)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 1 size: 0 MB
node 1 free: 0 MB
node 2 cpus:
node 2 size: 7935 MB
node 2 free: 7716 MB
node 3 cpus:
node 3 size: 8395 MB
node 3 free: 8015 MB
node distances:
node 0 1 2 3
0: 10 20 10 20
1: 20 10 20 20
2: 10 20 10 20
3: 20 20 20 10
This is because we statically initialize N_ONLINE to be [0] in
mm/page_alloc.c:
[N_ONLINE] = { { [0] = 1UL } },
I'm not sure what the architectural requirements are here, but at least
on this test system, removing this initialization, it boots fine and is
running. I've not yet tried stress tests, but it's survived the
beginnings of kernbench so far.
numactl --hardware
available: 3 nodes (1-3)
node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 1 size: 0 MB
node 1 free: 0 MB
node 2 cpus:
node 2 size: 7935 MB
node 2 free: 7479 MB
node 3 cpus:
node 3 size: 8396 MB
node 3 free: 8375 MB
node distances:
node 1 2 3
1: 10 20 20
2: 20 10 20
3: 20 20 10
Perhaps we could put in a ARCH_DOES_NOT_NEED_NODE0 and only define it on
powerpc for now, conditionalizing the above initialization on that?
I don't know if anything has recently changed in the past year or so, but
I've booted x86 machines with a hacked BIOS so that all memory on node 0
is hotpluggable and offline, so I believe this is possible on x86 as well.
Good to know, thanks! This is also certainly not very common on powerpc,
but it is possible -- and the topology ends up being inaccurate because
of the static initialization.
Thanks,
Nish
On 12.03.2014 [08:41:40 -0500], Christoph Lameter wrote:
On Tue, 11 Mar 2014, Nishanth Aravamudan wrote:
quoted
I have a P7 system that has no node0, but a node0 shows up in numactl
--hardware, which has no cpus and no memory (and no PCI devices):
Well as you see from the code there has been so far the assumption that
node 0 has memory. I have never run a machine that has no node 0 memory.
Do you mean beyond the initialization? I didn't see anything obvious so
far in the code itself that assumes a given node has memory (in the
sense of the nid). What are your thoughts about how best to support
this?
Thanks,
Nish
On 13.03.2014 [09:49:49 -0700], Nishanth Aravamudan wrote:
On 12.03.2014 [08:41:40 -0500], Christoph Lameter wrote:
quoted
On Tue, 11 Mar 2014, Nishanth Aravamudan wrote:
quoted
I have a P7 system that has no node0, but a node0 shows up in numactl
--hardware, which has no cpus and no memory (and no PCI devices):
Well as you see from the code there has been so far the assumption that
node 0 has memory. I have never run a machine that has no node 0 memory.
Do you mean beyond the initialization? I didn't see anything obvious so
far in the code itself that assumes a given node has memory (in the
sense of the nid). What are your thoughts about how best to support
this?
Ah, I found one path that is problematic on powerpc:
I'm seeing a panic at boot with this change on an LPAR which actually
has no Node 0. Here's what I think is happening:
start_kernel
...
-> setup_per_cpu_areas
-> pcpu_embed_first_chunk
-> pcpu_fc_alloc
-> ___alloc_bootmem_node(NODE_DATA(cpu_to_node(cpu), ...
-> smp_prepare_boot_cpu
-> set_numa_node(boot_cpuid)
So we panic on the NODE_DATA call. It seems that ia64, at least, uses
pcpu_alloc_first_chunk rather than embed. x86 has some code to handle
early calls of cpu_to_node (early_cpu_to_node) and sets the mapping for
all CPUs in setup_per_cpu_areas().
Thoughts? Does that mean we need something similar to x86 for powerpc?
Thanks,
Nish
From: Christoph Lameter <hidden> Date: 2014-05-21 14:16:33
On Mon, 19 May 2014, Nishanth Aravamudan wrote:
I'm seeing a panic at boot with this change on an LPAR which actually
has no Node 0. Here's what I think is happening:
start_kernel
...
-> setup_per_cpu_areas
-> pcpu_embed_first_chunk
-> pcpu_fc_alloc
-> ___alloc_bootmem_node(NODE_DATA(cpu_to_node(cpu), ...
-> smp_prepare_boot_cpu
-> set_numa_node(boot_cpuid)
So we panic on the NODE_DATA call. It seems that ia64, at least, uses
pcpu_alloc_first_chunk rather than embed. x86 has some code to handle
early calls of cpu_to_node (early_cpu_to_node) and sets the mapping for
all CPUs in setup_per_cpu_areas().
Maybe we can switch ia64 too embed? Tejun: Why are there these
dependencies?
Thoughts? Does that mean we need something similar to x86 for powerpc?
Hello,
On Wed, May 21, 2014 at 09:16:27AM -0500, Christoph Lameter wrote:
On Mon, 19 May 2014, Nishanth Aravamudan wrote:
quoted
I'm seeing a panic at boot with this change on an LPAR which actually
has no Node 0. Here's what I think is happening:
start_kernel
...
-> setup_per_cpu_areas
-> pcpu_embed_first_chunk
-> pcpu_fc_alloc
-> ___alloc_bootmem_node(NODE_DATA(cpu_to_node(cpu), ...
-> smp_prepare_boot_cpu
-> set_numa_node(boot_cpuid)
So we panic on the NODE_DATA call. It seems that ia64, at least, uses
pcpu_alloc_first_chunk rather than embed. x86 has some code to handle
early calls of cpu_to_node (early_cpu_to_node) and sets the mapping for
all CPUs in setup_per_cpu_areas().
Maybe we can switch ia64 too embed? Tejun: Why are there these
dependencies?
quoted
Thoughts? Does that mean we need something similar to x86 for powerpc?
I'm missing context to properly understand what's going on but the
specific allocator in use shouldn't matter. e.g. x86 can use both
embed and page allocators. If the problem is that the arch is
accessing percpu memory before percpu allocator is initialized and the
problem was masked before somehow, the right thing to do would be
removing those premature percpu accesses. If early percpu variables
are really necessary, doing similar early_percpu thing as in x86 would
be necessary.
Thanks.
--
tejun
Hi Tejun,
On 21.05.2014 [14:58:12 -0400], Tejun Heo wrote:
Hello,
On Wed, May 21, 2014 at 09:16:27AM -0500, Christoph Lameter wrote:
quoted
On Mon, 19 May 2014, Nishanth Aravamudan wrote:
quoted
I'm seeing a panic at boot with this change on an LPAR which actually
has no Node 0. Here's what I think is happening:
start_kernel
...
-> setup_per_cpu_areas
-> pcpu_embed_first_chunk
-> pcpu_fc_alloc
-> ___alloc_bootmem_node(NODE_DATA(cpu_to_node(cpu), ...
-> smp_prepare_boot_cpu
-> set_numa_node(boot_cpuid)
So we panic on the NODE_DATA call. It seems that ia64, at least, uses
pcpu_alloc_first_chunk rather than embed. x86 has some code to handle
early calls of cpu_to_node (early_cpu_to_node) and sets the mapping for
all CPUs in setup_per_cpu_areas().
Maybe we can switch ia64 too embed? Tejun: Why are there these
dependencies?
quoted
Thoughts? Does that mean we need something similar to x86 for powerpc?
I'm missing context to properly understand what's going on but the
specific allocator in use shouldn't matter. e.g. x86 can use both
embed and page allocators. If the problem is that the arch is
accessing percpu memory before percpu allocator is initialized and the
problem was masked before somehow, the right thing to do would be
removing those premature percpu accesses. If early percpu variables
are really necessary, doing similar early_percpu thing as in x86 would
be necessary.
For context: I was looking at why N_ONLINE was statically setting Node 0
to be online, whether or not the topology is that way -- I've been
getting several bugs lately where Node 0 is online, but has no CPUs and
no memory on it, on powerpc.
On powerpc, setup_per_cpu_areas calls into ___alloc_bootmem_node using
NODE_DATA(cpu_to_node(cpu)).
Currently, cpu_to_node() in arch/powerpc/include/asm/topology.h does:
/*
* During early boot, the numa-cpu lookup table might not have been
* setup for all CPUs yet. In such cases, default to node 0.
*/
return (nid < 0) ? 0 : nid;
And so early at boot, if node 0 is not present, we end up accessing an
unitialized NODE_DATA(). So this seems buggy (I'll contact the powerpc
deveopers separately on that).
I recently submitted patches to have powerpc turn on
USE_PERCPU_NUMA_NODEID and HAVE_MEMORYLESS_NODES. But then, cpu_to_node
will be accessing percpu data in setup_per_cpu_areas, which seems like a
no-no. And more specifically, since we haven't yet run
smp_prepare_boot_cpu() at this point, cpu_to_node has not yet been
initialized to provide a sane value.
Thanks,
Nish
From: David Rientjes <rientjes@google.com> Date: 2014-06-09 21:48:00
On Wed, 21 May 2014, Nishanth Aravamudan wrote:
For context: I was looking at why N_ONLINE was statically setting Node 0
to be online, whether or not the topology is that way -- I've been
getting several bugs lately where Node 0 is online, but has no CPUs and
no memory on it, on powerpc.
On powerpc, setup_per_cpu_areas calls into ___alloc_bootmem_node using
NODE_DATA(cpu_to_node(cpu)).
Currently, cpu_to_node() in arch/powerpc/include/asm/topology.h does:
/*
* During early boot, the numa-cpu lookup table might not have been
* setup for all CPUs yet. In such cases, default to node 0.
*/
return (nid < 0) ? 0 : nid;
And so early at boot, if node 0 is not present, we end up accessing an
unitialized NODE_DATA(). So this seems buggy (I'll contact the powerpc
deveopers separately on that).
I think what this really wants to do is NODE_DATA(cpu_to_mem(cpu)) and I
thought ppc had the cpu-to-local-memory-node mappings correct?
On 09.06.2014 [14:47:57 -0700], David Rientjes wrote:
On Wed, 21 May 2014, Nishanth Aravamudan wrote:
quoted
For context: I was looking at why N_ONLINE was statically setting Node 0
to be online, whether or not the topology is that way -- I've been
getting several bugs lately where Node 0 is online, but has no CPUs and
no memory on it, on powerpc.
On powerpc, setup_per_cpu_areas calls into ___alloc_bootmem_node using
NODE_DATA(cpu_to_node(cpu)).
Currently, cpu_to_node() in arch/powerpc/include/asm/topology.h does:
/*
* During early boot, the numa-cpu lookup table might not have been
* setup for all CPUs yet. In such cases, default to node 0.
*/
return (nid < 0) ? 0 : nid;
And so early at boot, if node 0 is not present, we end up accessing an
unitialized NODE_DATA(). So this seems buggy (I'll contact the powerpc
deveopers separately on that).
I think what this really wants to do is NODE_DATA(cpu_to_mem(cpu)) and I
thought ppc had the cpu-to-local-memory-node mappings correct?
Except cpu_to_mem relies on the mapping being defined, but early in
boot, specifically, it isn't yet (at least not necessarily).
-Nish
On Tue, Jun 10, 2014 at 04:31:57PM -0700, Nishanth Aravamudan wrote:
quoted
I think what this really wants to do is NODE_DATA(cpu_to_mem(cpu)) and I
thought ppc had the cpu-to-local-memory-node mappings correct?
Except cpu_to_mem relies on the mapping being defined, but early in
boot, specifically, it isn't yet (at least not necessarily).
Can't ppc NODE_DATA simply return dummy generic node_data during early
boot? Populating it with just enough to make early boot work
shouldn't be too hard, right?
Thanks.
--
tejun
Hello,
On Wed, May 21, 2014 at 09:16:27AM -0500, Christoph Lameter wrote:
quoted
On Mon, 19 May 2014, Nishanth Aravamudan wrote:
quoted
I'm seeing a panic at boot with this change on an LPAR which actually
has no Node 0. Here's what I think is happening:
start_kernel
...
-> setup_per_cpu_areas
-> pcpu_embed_first_chunk
-> pcpu_fc_alloc
-> ___alloc_bootmem_node(NODE_DATA(cpu_to_node(cpu), ...
-> smp_prepare_boot_cpu
-> set_numa_node(boot_cpuid)
So we panic on the NODE_DATA call. It seems that ia64, at least, uses
pcpu_alloc_first_chunk rather than embed. x86 has some code to handle
early calls of cpu_to_node (early_cpu_to_node) and sets the mapping for
all CPUs in setup_per_cpu_areas().
Maybe we can switch ia64 too embed? Tejun: Why are there these
dependencies?
quoted
Thoughts? Does that mean we need something similar to x86 for powerpc?
I'm missing context to properly understand what's going on but the
specific allocator in use shouldn't matter. e.g. x86 can use both
embed and page allocators. If the problem is that the arch is
accessing percpu memory before percpu allocator is initialized and the
problem was masked before somehow, the right thing to do would be
removing those premature percpu accesses. If early percpu variables
are really necessary, doing similar early_percpu thing as in x86 would
be necessary.
The early access is in the arch's pcpu_alloc_bootmem. On x86, rather
than using NODE_DATA(cpu_to_node), it uses (in pcpu_alloc_bootmem),
early_cpu_to_node(cpu) with their custom logic.
The issue is that cpu_to_node, if USE_PERCPU_NUMA_NODE_ID is defined
(which it is for NUMA powerpc, x86, ia64), is that cpu_to_node uses the
percpu area, which data isn't initialized yet.
So I guess powerpc needs the same treatment as x86.
Thanks,
Nish
On Tue, Jun 10, 2014 at 04:31:57PM -0700, Nishanth Aravamudan wrote:
quoted
quoted
I think what this really wants to do is NODE_DATA(cpu_to_mem(cpu)) and I
thought ppc had the cpu-to-local-memory-node mappings correct?
Except cpu_to_mem relies on the mapping being defined, but early in
boot, specifically, it isn't yet (at least not necessarily).
Can't ppc NODE_DATA simply return dummy generic node_data during early
boot? Populating it with just enough to make early boot work
shouldn't be too hard, right?
So the problem is this, whether we use cpu_to_mem() or cpu_to_node()
here, neither is setup yet because of the ordering between percpu setup
and the actual writing of the percpu data (that is actually storing what
node/local memory is relative to a given CPU).
The NODE_DATA is all correct, but since we are calling cpu_to_{mem,node}
before it really holds valid data, it falsely says 0, which is not
necessarily even an online node.
So, I think we need to do the same thing as x86 and have an early
mapping setup and configured before the percpu areas are.
Thanks,
Nish