[RFC PATCH v3 1/3] powerpc: Fix warning reported by verify_cpu_node_mapping()

With commit 2fabf084b6ad ("powerpc: reorder per-cpu NUMA information's
initialization"), during boottime, cpu_numa_callback() is called
earlier(before their online) for each cpu, and verify_cpu_node_mapping()
uses cpu_to_node() to check whether siblings are in the same node.

It skips the checking for siblings that are not online yet. So the only
check done here is for the bootcpu, which is online at that time. But
the per-cpu numa_node cpu_to_node() uses hasn't been set up yet (which
will be set up in smp_prepare_cpus()).

So I saw something like following reported:
[    0.000000] CPU thread siblings 1/2/3 and 0 don't belong to the same
node!

As we don't actually do the checking during this early stage, so maybe
we could directly call numa_setup_cpu() in do_init_bootmem().

Cc: Nishanth Aravamudan <redacted>
Cc: Nathan Fontenot <redacted>
Signed-off-by: Li Zhong <redacted>
---
 arch/powerpc/mm/numa.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index d7737a5..9918c02 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1128,8 +1128,7 @@ void __init do_init_bootmem(void)
 	 * early in boot, cf. smp_prepare_cpus().
 	 */
 	for_each_possible_cpu(cpu) {
-		cpu_numa_callback(&ppc64_numa_nb, CPU_UP_PREPARE,
-				  (void *)(unsigned long)cpu);
+		numa_setup_cpu((unsigned long)cpu);
 	}
 }
 

-- 
1.9.1

As Nish suggested, it makes more sense to init the numa node informatiion
for present cpus at boottime, which could also avoid WARN_ON(1) in
numa_setup_cpu().

With this change, we also need to change the smp_prepare_cpus() to set up
numa information only on present cpus.

For those possible, but not present cpus, their numa information
will be set up after they are started, as the original code did before commit
2fabf084b6ad.

Cc: Nishanth Aravamudan <redacted>
Cc: Nathan Fontenot <redacted>
Signed-off-by: Li Zhong <redacted>
---
 arch/powerpc/kernel/smp.c | 10 ++++++++--
 arch/powerpc/mm/numa.c    |  2 +-
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index a0738af..dc0e774 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c

@@ -379,8 +379,11 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 		/*
 		 * numa_node_id() works after this.
 		 */
-		set_cpu_numa_node(cpu, numa_cpu_lookup_table[cpu]);
-		set_cpu_numa_mem(cpu, local_memory_node(numa_cpu_lookup_table[cpu]));
+		if (cpu_present(cpu)) {
+			set_cpu_numa_node(cpu, numa_cpu_lookup_table[cpu]);
+			set_cpu_numa_mem(cpu,
+				local_memory_node(numa_cpu_lookup_table[cpu]));
+		}
 	}
 
 	cpumask_set_cpu(boot_cpuid, cpu_sibling_mask(boot_cpuid));

@@ -728,6 +731,9 @@ void start_secondary(void *unused)
 	}
 	traverse_core_siblings(cpu, true);
 
+	set_numa_node(numa_cpu_lookup_table[cpu]);
+	set_numa_mem(local_memory_node(numa_cpu_lookup_table[cpu]));
+
 	smp_wmb();
 	notify_cpu_starting(cpu);
 	set_cpu_online(cpu, true);

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 9918c02..3a9061e 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1127,7 +1127,7 @@ void __init do_init_bootmem(void)
 	 * even before we online them, so that we can use cpu_to_{node,mem}
 	 * early in boot, cf. smp_prepare_cpus().
 	 */
-	for_each_possible_cpu(cpu) {
+	for_each_present_cpu(cpu) {
 		numa_setup_cpu((unsigned long)cpu);
 	}
 }

-- 
1.9.1

this patches changes some error handling logics in numa_setup_cpu(),
when cpu node is not found, so:

if the cpu is possible, but not present, -1 is kept in numa_cpu_lookup_table,
so later, if the cpu is added, we could set correct numa information for it.

if the cpu is present, then we set the first online node to
numa_cpu_lookup_table instead of 0 ( in case 0 might not be an online node? )

Cc: Nishanth Aravamudan <redacted>
Cc: Nathan Fontenot <redacted>
Signed-off-by: Li Zhong <redacted>
---
 arch/powerpc/mm/numa.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 3a9061e..ec32d46 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -538,7 +538,7 @@ static int of_drconf_to_nid_single(struct of_drconf_cell *drmem,
  */
 static int numa_setup_cpu(unsigned long lcpu)
 {
-	int nid;
+	int nid = -1;
 	struct device_node *cpu;
 
 	/*

@@ -555,19 +555,21 @@ static int numa_setup_cpu(unsigned long lcpu)
 
 	if (!cpu) {
 		WARN_ON(1);
-		nid = 0;
-		goto out;
+		if (cpu_present(lcpu))
+			goto out_present;
+		else
+			goto out;
 	}
 
 	nid = of_node_to_nid_single(cpu);
 
+out_present:
 	if (nid < 0 || !node_online(nid))
 		nid = first_online_node;
-out:
-	map_cpu_to_node(lcpu, nid);
 
+	map_cpu_to_node(lcpu, nid);
 	of_node_put(cpu);
-
+out:
 	return nid;
 }
 

-- 
1.9.1

On 27.08.2014 [17:33:59 +0800], Li Zhong wrote:

With commit 2fabf084b6ad ("powerpc: reorder per-cpu NUMA information's
initialization"), during boottime, cpu_numa_callback() is called
earlier(before their online) for each cpu, and verify_cpu_node_mapping()
uses cpu_to_node() to check whether siblings are in the same node.

It skips the checking for siblings that are not online yet. So the only
check done here is for the bootcpu, which is online at that time. But
the per-cpu numa_node cpu_to_node() uses hasn't been set up yet (which
will be set up in smp_prepare_cpus()).

So I saw something like following reported:
[    0.000000] CPU thread siblings 1/2/3 and 0 don't belong to the same
node!

As we don't actually do the checking during this early stage, so maybe
we could directly call numa_setup_cpu() in do_init_bootmem().

Cc: Nishanth Aravamudan <redacted>
Cc: Nathan Fontenot <redacted>
Signed-off-by: Li Zhong <redacted>

Acked-by: Nishanth Aravamudan <redacted>

On 27.08.2014 [17:34:00 +0800], Li Zhong wrote:

As Nish suggested, it makes more sense to init the numa node informatiion
for present cpus at boottime, which could also avoid WARN_ON(1) in
numa_setup_cpu().

With this change, we also need to change the smp_prepare_cpus() to set up
numa information only on present cpus.

For those possible, but not present cpus, their numa information
will be set up after they are started, as the original code did before commit
2fabf084b6ad.

Cc: Nishanth Aravamudan <redacted>
Cc: Nathan Fontenot <redacted>
Signed-off-by: Li Zhong <redacted>

Acked-by: Nishanth Aravamudan <redacted>

On 27.08.2014 [17:34:01 +0800], Li Zhong wrote:

this patches changes some error handling logics in numa_setup_cpu(),
when cpu node is not found, so:

if the cpu is possible, but not present, -1 is kept in numa_cpu_lookup_table,
so later, if the cpu is added, we could set correct numa information for it.

if the cpu is present, then we set the first online node to
numa_cpu_lookup_table instead of 0 ( in case 0 might not be an online node? )

Not currently possible (Node 0 is always online), but I'm working on
changing that :)

Cc: Nishanth Aravamudan <redacted>
Cc: Nathan Fontenot <redacted>
Signed-off-by: Li Zhong <redacted>

Acked-by: Nishanth Aravamudan <redacted>

On 03/09/14 13:02, Nishanth Aravamudan wrote:

On 27.08.2014 [17:34:00 +0800], Li Zhong wrote:

quoted

As Nish suggested, it makes more sense to init the numa node informatiion
for present cpus at boottime, which could also avoid WARN_ON(1) in
numa_setup_cpu().

Hit this on a Power8 LPAR. With the patchset applied the warnings no 
longer present.

quoted

With this change, we also need to change the smp_prepare_cpus() to set up
numa information only on present cpus.

For those possible, but not present cpus, their numa information
will be set up after they are started, as the original code did before commit
2fabf084b6ad.

Cc: Nishanth Aravamudan <redacted>
Cc: Nathan Fontenot <redacted>
Signed-off-by: Li Zhong <redacted>

Acked-by: Nishanth Aravamudan <redacted>

Tested-by: Cyril Bur <redacted>

quoted

---
  arch/powerpc/kernel/smp.c | 10 ++++++++--
  arch/powerpc/mm/numa.c    |  2 +-
  2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index a0738af..dc0e774 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c

@@ -379,8 +379,11 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
  		/*
  		 * numa_node_id() works after this.
  		 */
-		set_cpu_numa_node(cpu, numa_cpu_lookup_table[cpu]);
-		set_cpu_numa_mem(cpu, local_memory_node(numa_cpu_lookup_table[cpu]));
+		if (cpu_present(cpu)) {
+			set_cpu_numa_node(cpu, numa_cpu_lookup_table[cpu]);
+			set_cpu_numa_mem(cpu,
+				local_memory_node(numa_cpu_lookup_table[cpu]));
+		}
  	}

  	cpumask_set_cpu(boot_cpuid, cpu_sibling_mask(boot_cpuid));

@@ -728,6 +731,9 @@ void start_secondary(void *unused)
  	}
  	traverse_core_siblings(cpu, true);

+	set_numa_node(numa_cpu_lookup_table[cpu]);
+	set_numa_mem(local_memory_node(numa_cpu_lookup_table[cpu]));
+
  	smp_wmb();
  	notify_cpu_starting(cpu);
  	set_cpu_online(cpu, true);

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 9918c02..3a9061e 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1127,7 +1127,7 @@ void __init do_init_bootmem(void)
  	 * even before we online them, so that we can use cpu_to_{node,mem}
  	 * early in boot, cf. smp_prepare_cpus().
  	 */
-	for_each_possible_cpu(cpu) {
+	for_each_present_cpu(cpu) {
  		numa_setup_cpu((unsigned long)cpu);
  	}
  }
--

1.9.1

_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Ben & Michael,

What's the status of these patches?

Thanks,
Nish

On 27.08.2014 [17:33:59 +0800], Li Zhong wrote:

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

Ben & Michael,

What's the status of these patches?

Waiting for somebody to review them ? :-)

Cheers,
Ben.

Thanks,
Nish

On 27.08.2014 [17:33:59 +0800], Li Zhong wrote:

quoted

With commit 2fabf084b6ad ("powerpc: reorder per-cpu NUMA information's
initialization"), during boottime, cpu_numa_callback() is called
earlier(before their online) for each cpu, and verify_cpu_node_mapping()
uses cpu_to_node() to check whether siblings are in the same node.

It skips the checking for siblings that are not online yet. So the only
check done here is for the bootcpu, which is online at that time. But
the per-cpu numa_node cpu_to_node() uses hasn't been set up yet (which
will be set up in smp_prepare_cpus()).

So I saw something like following reported:
[    0.000000] CPU thread siblings 1/2/3 and 0 don't belong to the same
node!

As we don't actually do the checking during this early stage, so maybe
we could directly call numa_setup_cpu() in do_init_bootmem().

Cc: Nishanth Aravamudan <redacted>
Cc: Nathan Fontenot <redacted>
Signed-off-by: Li Zhong <redacted>
---
 arch/powerpc/mm/numa.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index d7737a5..9918c02 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1128,8 +1128,7 @@ void __init do_init_bootmem(void)
 	 * early in boot, cf. smp_prepare_cpus().
 	 */
 	for_each_possible_cpu(cpu) {
-		cpu_numa_callback(&ppc64_numa_nb, CPU_UP_PREPARE,
-				  (void *)(unsigned long)cpu);
+		numa_setup_cpu((unsigned long)cpu);
 	}
 }

-- 
1.9.1

On 03.10.2014 [07:28:19 +1000], Benjamin Herrenschmidt wrote:

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Waiting for somebody to review them ? :-)

Heh, I acked them about a month ago, I can give a stronger Reviewed-by,
if you prefer?

-Nish

On Thu, 2014-10-02 at 14:53 -0700, Nishanth Aravamudan wrote:

On 03.10.2014 [07:28:19 +1000], Benjamin Herrenschmidt wrote:

quoted

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Waiting for somebody to review them ? :-)

Heh, I acked them about a month ago, I can give a stronger Reviewed-by,
if you prefer?

Nah, nothing other than having me or mpe pick them up. I've been letting
Michael handle this merge window, so maybe he should take it :-)

Cheers,
Ben.

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

Ben & Michael,

What's the status of these patches?

Been in my next for a week :)

https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next

cheers

On 03.10.2014 [10:50:20 +1000], Michael Ellerman wrote:

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Been in my next for a week :)

https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next

Ah ok, thanks -- I wasn't following your tree, my fault. Do we want
these to go back to 3.17-stable, as they fix some annoying splats during
boot (non-fatal afaict, though)?

-Nish

On Fri, 2014-10-03 at 16:26 -0700, Nishanth Aravamudan wrote:

On 03.10.2014 [10:50:20 +1000], Michael Ellerman wrote:

quoted

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Been in my next for a week :)

https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next

Ah ok, thanks -- I wasn't following your tree, my fault. 

Not really your fault, I hadn't announced my trees existence :)

Do we want these to go back to 3.17-stable, as they fix some annoying splats
during boot (non-fatal afaict, though)?

Up to you really, I don't know how often/bad they were. I haven't added CC
stable tags to the commits, so if you want them in stable you should send them
explicitly.

cheers

On 07.10.2014 [17:28:38 +1100], Michael Ellerman wrote:

On Fri, 2014-10-03 at 16:26 -0700, Nishanth Aravamudan wrote:

quoted

On 03.10.2014 [10:50:20 +1000], Michael Ellerman wrote:

quoted

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Been in my next for a week :)

https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next

Ah ok, thanks -- I wasn't following your tree, my fault. 

Not really your fault, I hadn't announced my trees existence :)

quoted

Do we want these to go back to 3.17-stable, as they fix some annoying splats
during boot (non-fatal afaict, though)?

Up to you really, I don't know how often/bad they were. I haven't added CC
stable tags to the commits, so if you want them in stable you should send them
explicitly.

I think they occur every boot, unconditionally, on pseries. Doesn't
prevent boot, just really noisy. I think it'd be good to get them into
-stable.

Li Zhong, can you push them once they get sent upstream?

Thanks,
Nish

On 二, 2014-10-07 at 08:33 -0700, Nishanth Aravamudan wrote:

On 07.10.2014 [17:28:38 +1100], Michael Ellerman wrote:

quoted

On Fri, 2014-10-03 at 16:26 -0700, Nishanth Aravamudan wrote:

quoted

On 03.10.2014 [10:50:20 +1000], Michael Ellerman wrote:

quoted

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Been in my next for a week :)

https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next

Ah ok, thanks -- I wasn't following your tree, my fault. 

Not really your fault, I hadn't announced my trees existence :)

quoted

Do we want these to go back to 3.17-stable, as they fix some annoying splats
during boot (non-fatal afaict, though)?

Up to you really, I don't know how often/bad they were. I haven't added CC
stable tags to the commits, so if you want them in stable you should send them
explicitly.

I think they occur every boot, unconditionally, on pseries. Doesn't
prevent boot, just really noisy. I think it'd be good to get them into
-stable.

Li Zhong, can you push them once they get sent upstream?

Ok, I will send these patches to stable mailing list after it is merged.

Thanks, Zhong

Thanks,
Nish

On 二, 2014-10-07 at 08:33 -0700, Nishanth Aravamudan wrote:

On 07.10.2014 [17:28:38 +1100], Michael Ellerman wrote:

quoted

On Fri, 2014-10-03 at 16:26 -0700, Nishanth Aravamudan wrote:

quoted

On 03.10.2014 [10:50:20 +1000], Michael Ellerman wrote:

quoted

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Been in my next for a week :)

https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next

Ah ok, thanks -- I wasn't following your tree, my fault. 

Not really your fault, I hadn't announced my trees existence :)

quoted

Do we want these to go back to 3.17-stable, as they fix some annoying splats
during boot (non-fatal afaict, though)?

Up to you really, I don't know how often/bad they were. I haven't added CC
stable tags to the commits, so if you want them in stable you should send them
explicitly.

I think they occur every boot, unconditionally, on pseries. Doesn't
prevent boot, just really noisy. I think it'd be good to get them into
-stable.

Li Zhong, can you push them once they get sent upstream?

I guess I only need to send the first two patches to stable? 

Thanks, Zhong

Thanks,
Nish

On Tue, 2014-10-14 at 10:39 +0800, Li Zhong wrote:

On 二, 2014-10-07 at 08:33 -0700, Nishanth Aravamudan wrote:

quoted

On 07.10.2014 [17:28:38 +1100], Michael Ellerman wrote:

quoted

On Fri, 2014-10-03 at 16:26 -0700, Nishanth Aravamudan wrote:

quoted

On 03.10.2014 [10:50:20 +1000], Michael Ellerman wrote:

quoted

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Been in my next for a week :)

https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next

Ah ok, thanks -- I wasn't following your tree, my fault. 

Not really your fault, I hadn't announced my trees existence :)

quoted

Do we want these to go back to 3.17-stable, as they fix some annoying splats
during boot (non-fatal afaict, though)?

Up to you really, I don't know how often/bad they were. I haven't added CC
stable tags to the commits, so if you want them in stable you should send them
explicitly.

I think they occur every boot, unconditionally, on pseries. Doesn't
prevent boot, just really noisy. I think it'd be good to get them into
-stable.

Li Zhong, can you push them once they get sent upstream?

I guess I only need to send the first two patches to stable? 

Probably. It's not clear from the changelog how serious a problem it fixes.

See Documentation/stable_kernel_rules.txt

cheers

On 二, 2014-10-14 at 15:35 +1100, Michael Ellerman wrote:

On Tue, 2014-10-14 at 10:39 +0800, Li Zhong wrote:

quoted

On 二, 2014-10-07 at 08:33 -0700, Nishanth Aravamudan wrote:

quoted

On 07.10.2014 [17:28:38 +1100], Michael Ellerman wrote:

quoted

On Fri, 2014-10-03 at 16:26 -0700, Nishanth Aravamudan wrote:

quoted

On 03.10.2014 [10:50:20 +1000], Michael Ellerman wrote:

quoted

On Thu, 2014-10-02 at 14:13 -0700, Nishanth Aravamudan wrote:

quoted

Ben & Michael,

What's the status of these patches?

Been in my next for a week :)

https://git.kernel.org/cgit/linux/kernel/git/mpe/linux.git/log/?h=next

Ah ok, thanks -- I wasn't following your tree, my fault. 

Not really your fault, I hadn't announced my trees existence :)

quoted

Do we want these to go back to 3.17-stable, as they fix some annoying splats
during boot (non-fatal afaict, though)?

Up to you really, I don't know how often/bad they were. I haven't added CC
stable tags to the commits, so if you want them in stable you should send them
explicitly.

I think they occur every boot, unconditionally, on pseries. Doesn't
prevent boot, just really noisy. I think it'd be good to get them into
-stable.

Li Zhong, can you push them once they get sent upstream?

I guess I only need to send the first two patches to stable? 

Probably. It's not clear from the changelog how serious a problem it fixes.

See Documentation/stable_kernel_rules.txt

OK, I'll send the first two to stable. 
Thanks, Zhong

cheers