[PATCH V13 0/4] powerpc/vphn: Update CPU topology when VPHN enabled

powerpc/numa: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  This patch
addresses some of those problems.

First, it corrects the currently broken capability to set the
topology for shared CPUs in LPARs.  At boot time for shared CPU
lpars, the topology for each CPU was being set to node zero.  Now
when numa_update_cpu_topology() is called appropriately, the Virtual
Processor Home Node (VPHN) capabilities information provided by the
pHyp allows the appropriate node in the shared configuration to be
selected for the CPU.

Next, it updates the initialization checks to independently recognize
PRRN or VPHN support.

Next, during hotplug CPU operations, it resets the timer on topology
update work function to a small value to better ensure that the CPU
topology is detected and configured sooner.

Also, fix an end-of-updates processing problem observed occasionally
in numa_update_cpu_topology().

Signed-off-by: Michael Bringmann <redacted>

Michael Bringmann (4):
  powerpc/vphn: Update CPU topology when VPHN enabled
  powerpc/vphn: Improve recognition of PRRN/VPHN
  powerpc/hotplug: Improve responsiveness of hotplug change
  powerpc/vphn:
---
Changes in V13:
  -- Split into multiple smaller patches

powerpc/vphn: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  This patch
corrects the currently broken capability to set the topology for
shared CPUs in LPARs.  At boot time for shared CPU lpars, the
topology for each CPU was being set to node zero.  Now when
numa_update_cpu_topology() is called appropriately, the Virtual
Processor Home Node (VPHN) capabilities information provided by the
pHyp allows the appropriate node in the shared configuration to be
selected for the CPU.

Signed-off-by: Michael Bringmann <redacted>
---
Changes in V13:
  -- Split patch for improved review
---
 arch/powerpc/mm/numa.c |   31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index b95c584..312f6ee 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1153,6 +1153,8 @@ struct topology_update_data {
 static int vphn_enabled;
 static int prrn_enabled;
 static void reset_topology_timer(void);
+static int topology_inited;
+static int topology_update_needed;
 
 /*
  * Store the current values of the associativity change counters in the

@@ -1246,6 +1248,11 @@ static long vphn_get_associativity(unsigned long cpu,
 			"hcall_vphn() experienced a hardware fault "
 			"preventing VPHN. Disabling polling...\n");
 		stop_topology_update();
+		break;
+	case H_SUCCESS:
+		dbg("VPHN hcall succeeded. Reset polling...\n");
+		timed_topology_update(0);
+		break;
 	}
 
 	return rc;

@@ -1323,8 +1330,11 @@ int numa_update_cpu_topology(bool cpus_locked)
 	struct device *dev;
 	int weight, new_nid, i = 0;
 
-	if (!prrn_enabled && !vphn_enabled)
+	if (!prrn_enabled && !vphn_enabled) {
+		if (!topology_inited)
+			topology_update_needed = 1;
 		return 0;
+	}
 
 	weight = cpumask_weight(&cpu_associativity_changes_mask);
 	if (!weight)

@@ -1363,6 +1373,8 @@ int numa_update_cpu_topology(bool cpus_locked)
 			cpumask_andnot(&cpu_associativity_changes_mask,
 					&cpu_associativity_changes_mask,
 					cpu_sibling_mask(cpu));
+			pr_info("Assoc chg gives same node %d for cpu%d\n",
+					new_nid, cpu);
 			cpu = cpu_last_thread_sibling(cpu);
 			continue;
 		}

@@ -1433,6 +1445,7 @@ int numa_update_cpu_topology(bool cpus_locked)
 
 out:
 	kfree(updates);
+	topology_update_needed = 0;
 	return changed;
 }
 

@@ -1453,6 +1466,13 @@ static void topology_schedule_update(void)
 	schedule_work(&topology_work);
 }
 
+void shared_topology_update(void)
+{
+	if (firmware_has_feature(FW_FEATURE_VPHN) &&
+		   lppaca_shared_proc(get_lppaca()))
+		topology_schedule_update();
+}
+
 static void topology_timer_fn(unsigned long ignored)
 {
 	if (prrn_enabled && cpumask_weight(&cpu_associativity_changes_mask))

@@ -1519,7 +1539,6 @@ int start_topology_update(void)
 	if (firmware_has_feature(FW_FEATURE_PRRN)) {
 		if (!prrn_enabled) {
 			prrn_enabled = 1;
-			vphn_enabled = 0;
 #ifdef CONFIG_SMP
 			rc = of_reconfig_notifier_register(&dt_update_nb);
 #endif

@@ -1527,7 +1546,6 @@ int start_topology_update(void)
 	} else if (firmware_has_feature(FW_FEATURE_VPHN) &&
 		   lppaca_shared_proc(get_lppaca())) {
 		if (!vphn_enabled) {
-			prrn_enabled = 0;
 			vphn_enabled = 1;
 			setup_cpu_associativity_change_counters();
 			init_timer_deferrable(&topology_timer);

@@ -1613,9 +1631,16 @@ static int topology_update_init(void)
 	if (topology_updates_enabled)
 		start_topology_update();
 
+	shared_topology_update();
+
 	if (!proc_create("powerpc/topology_updates", 0644, NULL, &topology_ops))
 		return -ENOMEM;
 
+	topology_inited = 1;
+	if (topology_update_needed)
+		bitmap_fill(cpumask_bits(&cpu_associativity_changes_mask),
+					nr_cpumask_bits);
+
 	return 0;
 }
 device_initcall(topology_update_init);

powerpc/vphn: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  This patch
updates the initialization checks to independently recognize PRRN
or VPHN support.

Signed-off-by: Michael Bringmann <redacted>
---
Changes in V13:
  -- Split patch to improve review
---
 arch/powerpc/mm/numa.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 312f6ee..c08d736 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1543,7 +1543,8 @@ int start_topology_update(void)
 			rc = of_reconfig_notifier_register(&dt_update_nb);
 #endif
 		}
-	} else if (firmware_has_feature(FW_FEATURE_VPHN) &&
+	}
+	if (firmware_has_feature(FW_FEATURE_VPHN) &&
 		   lppaca_shared_proc(get_lppaca())) {
 		if (!vphn_enabled) {
 			vphn_enabled = 1;

@@ -1568,7 +1569,8 @@ int stop_topology_update(void)
 #ifdef CONFIG_SMP
 		rc = of_reconfig_notifier_unregister(&dt_update_nb);
 #endif
-	} else if (vphn_enabled) {
+	}
+	if (vphn_enabled) {
 		vphn_enabled = 0;
 		rc = del_timer_sync(&topology_timer);
 	}

powerpc/hotplug: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  During hotplug
CPU operations, this patch resets the timer on topology update work
function to a small value to better ensure that the CPU topology is
detected and configured sooner.

Signed-off-by: Michael Bringmann <redacted>
---
 arch/powerpc/include/asm/topology.h          |    8 ++++++++
 arch/powerpc/mm/numa.c                       |   21 ++++++++++++++++++++-
 arch/powerpc/platforms/pseries/hotplug-cpu.c |    2 ++
 3 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/include/asm/topology.h b/arch/powerpc/include/asm/topology.h
index dc4e159..beb9bca 100644
--- a/arch/powerpc/include/asm/topology.h
+++ b/arch/powerpc/include/asm/topology.h

@@ -98,6 +98,14 @@ static inline int prrn_is_enabled(void)
 }
 #endif /* CONFIG_NUMA && CONFIG_PPC_SPLPAR */
 
+#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_NEED_MULTIPLE_NODES)
+#if defined(CONFIG_PPC_SPLPAR)
+extern int timed_topology_update(int nsecs);
+#else
+#define	timed_topology_update(nsecs)
+#endif /* CONFIG_PPC_SPLPAR */
+#endif /* CONFIG_HOTPLUG_CPU || CONFIG_NEED_MULTIPLE_NODES */
+
 #include <asm-generic/topology.h>
 
 #ifdef CONFIG_SMP

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index c08d736..3a5b334 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1148,15 +1148,34 @@ struct topology_update_data {
 	int new_nid;
 };
 
+#define TOPOLOGY_DEF_TIMER_SECS	60
+
 static u8 vphn_cpu_change_counts[NR_CPUS][MAX_DISTANCE_REF_POINTS];
 static cpumask_t cpu_associativity_changes_mask;
 static int vphn_enabled;
 static int prrn_enabled;
 static void reset_topology_timer(void);
+static int topology_timer_secs = 1;
 static int topology_inited;
 static int topology_update_needed;
 
 /*
+ * Change polling interval for associativity changes.
+ */
+int timed_topology_update(int nsecs)
+{
+	if (nsecs > 0)
+		topology_timer_secs = nsecs;
+	else
+		topology_timer_secs = TOPOLOGY_DEF_TIMER_SECS;
+
+	if (vphn_enabled)
+		reset_topology_timer();
+
+	return 0;
+}
+
+/*
  * Store the current values of the associativity change counters in the
  * hypervisor.
  */

@@ -1489,7 +1508,7 @@ static void topology_timer_fn(unsigned long ignored)
 static void reset_topology_timer(void)
 {
 	topology_timer.data = 0;
-	topology_timer.expires = jiffies + 60 * HZ;
+	topology_timer.expires = jiffies + topology_timer_secs * HZ;
 	mod_timer(&topology_timer, topology_timer.expires);
 }
 

diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c b/arch/powerpc/platforms/pseries/hotplug-cpu.c
index 6afd1ef..5a7fb1e 100644
--- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
+++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c

@@ -356,6 +356,7 @@ static int dlpar_online_cpu(struct device_node *dn)
 			BUG_ON(get_cpu_current_state(cpu)
 					!= CPU_STATE_OFFLINE);
 			cpu_maps_update_done();
+			timed_topology_update(1);
 			rc = device_online(get_cpu_device(cpu));
 			if (rc)
 				goto out;

@@ -522,6 +523,7 @@ static int dlpar_offline_cpu(struct device_node *dn)
 				set_preferred_offline_state(cpu,
 							    CPU_STATE_OFFLINE);
 				cpu_maps_update_done();
+				timed_topology_update(1);
 				rc = device_offline(get_cpu_device(cpu));
 				if (rc)
 					goto out;

powerpc/vphn: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  This patch
fixes an end-of-updates processing problem observed occasionally
in numa_update_cpu_topology().

Signed-off-by: Michael Bringmann <redacted>
---
 arch/powerpc/mm/numa.c |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 3a5b334..fccf23f 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1410,6 +1410,13 @@ int numa_update_cpu_topology(bool cpus_locked)
 		cpu = cpu_last_thread_sibling(cpu);
 	}
 
+	/*
+	 * Prevent processing of 'updates' from overflowing array
+	 * in cases where last entry filled in a 'next' pointer.
+	 */
+	if (i)
+		updates[i-1].next = NULL;
+
 	pr_debug("Topology update for the following CPUs:\n");
 	if (cpumask_weight(&updated_cpus)) {
 		for (ud = &updates[0]; ud; ud = ud->next) {

On 09/01/2017 10:48 AM, Michael Bringmann wrote:

As mentioned previously, this should either be removed or changed to a
debug statement.

Could we just check vphn_enabled here? The init routine will set this to true
only if the feature is supported and we are using shared processors.

Also, this routine seems a bit odd. The only place it is called from is an
init routine, topology_update_init. Is there a reason this check couldn't just
be in that routine, or it seems like you could just call topology_schedule_update
directly from start_topology_update when vphn is initialized.

-Nathan

On 09/01/2017 10:48 AM, Michael Bringmann wrote:

In patch 1/4, you removed the setting of prrn_enabled and vphn_enabled
to 0. It seems like that update would be part of this patch.

-Nathan

On 09/01/2017 10:48 AM, Michael Bringmann wrote:

powerpc/hotplug: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  During hotplug
CPU operations, this patch resets the timer on topology update work
function to a small value to better ensure that the CPU topology is
detected and configured sooner.

Looking through the changes you've made here I don't see where the
topology timeout ever gets set to the default timeout. When calculating
the next timeout you use topology_timer_secs which is initialized to 1, so
the timer pops every second after initialization. Then after a dlpar cpu
operation the timer is set to pop every second. There is no place that I
see where the timeout is set to the default 60 seconds.

Should this whole thing be wrapped by if (vphn_enabled) ?

-Nathan

On 09/01/2017 10:48 AM, Michael Bringmann wrote:

This really looks like the bug is in the code above this where we
fill in the updates array for each of the sibling cpus. The code
there assumes that if the current update entry is not the end that
there will be more updates and blindly sets the next pointer.

Perhaps correcting the logic in that code to next pointers. Set the
ud pointer to NULL before the outer for_each_cpu() loop. Then in the
inner for_each_cpu(sibling,...) loop update the ud-> next pointer as
the first operation.

		for_each_cpu(sibling, cpu_sibling_mask(cpu)) {
			if (ud)
				ud->next = &updates[i];
			...
		}

Obviously untested, but I think this would prevent setting the next
pointer in the last update entry that is filled out erroneously.
  
-Nathan

 	pr_debug("Topology update for the following CPUs:\n");
 	if (cpumask_weight(&updated_cpus)) {
 		for (ud = &updates[0]; ud; ud = ud->next) {

On 09/06/2017 09:45 AM, Nathan Fontenot wrote:

On 09/01/2017 10:48 AM, Michael Bringmann wrote:

quoted

powerpc/vphn: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  This patch
fixes an end-of-updates processing problem observed occasionally
in numa_update_cpu_topology().

Signed-off-by: Michael Bringmann <redacted>
---
 arch/powerpc/mm/numa.c |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 3a5b334..fccf23f 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1410,6 +1410,13 @@ int numa_update_cpu_topology(bool cpus_locked)
 		cpu = cpu_last_thread_sibling(cpu);
 	}

+	/*
+	 * Prevent processing of 'updates' from overflowing array
+	 * in cases where last entry filled in a 'next' pointer.
+	 */
+	if (i)
+		updates[i-1].next = NULL;
+

This really looks like the bug is in the code above this where we
fill in the updates array for each of the sibling cpus. The code
there assumes that if the current update entry is not the end that
there will be more updates and blindly sets the next pointer.

Perhaps correcting the logic in that code to next pointers. Set the
ud pointer to NULL before the outer for_each_cpu() loop. Then in the
inner for_each_cpu(sibling,...) loop update the ud-> next pointer as
the first operation.

		for_each_cpu(sibling, cpu_sibling_mask(cpu)) {
			if (ud)
				ud->next = &updates[i];
			...
		}

Obviously untested, but I think this would prevent setting the next
pointer in the last update entry that is filled out erroneously.

The above fragment looks to skip initialization of the 'next' pointer
in the first element of the the 'updates'.  That would abort subsequent
evaluation of the array too soon, I believe.  I would like to take another look
to see whether the current check 'if (i < weight) ud->next = &updates[i];'
is having problems due to i being 0-relative and weight being 1-relative.

  
-Nathan

Michael

quoted

 	pr_debug("Topology update for the following CPUs:\n");
 	if (cpumask_weight(&updated_cpus)) {
 		for (ud = &updates[0]; ud; ud = ud->next) {

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com

On 09/06/2017 05:03 PM, Michael Bringmann wrote:

On 09/06/2017 09:45 AM, Nathan Fontenot wrote:

quoted

On 09/01/2017 10:48 AM, Michael Bringmann wrote:

quoted

powerpc/vphn: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  This patch
fixes an end-of-updates processing problem observed occasionally
in numa_update_cpu_topology().

Signed-off-by: Michael Bringmann <redacted>
---
 arch/powerpc/mm/numa.c |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 3a5b334..fccf23f 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1410,6 +1410,13 @@ int numa_update_cpu_topology(bool cpus_locked)
 		cpu = cpu_last_thread_sibling(cpu);
 	}

+	/*
+	 * Prevent processing of 'updates' from overflowing array
+	 * in cases where last entry filled in a 'next' pointer.
+	 */
+	if (i)
+		updates[i-1].next = NULL;
+

This really looks like the bug is in the code above this where we
fill in the updates array for each of the sibling cpus. The code
there assumes that if the current update entry is not the end that
there will be more updates and blindly sets the next pointer.

Perhaps correcting the logic in that code to next pointers. Set the
ud pointer to NULL before the outer for_each_cpu() loop. Then in the
inner for_each_cpu(sibling,...) loop update the ud-> next pointer as
the first operation.

		for_each_cpu(sibling, cpu_sibling_mask(cpu)) {
			if (ud)
				ud->next = &updates[i];
			...
		}

Obviously untested, but I think this would prevent setting the next
pointer in the last update entry that is filled out erroneously.

The above fragment looks to skip initialization of the 'next' pointer
in the first element of the the 'updates'.  That would abort subsequent
evaluation of the array too soon, I believe.  I would like to take another look
to see whether the current check 'if (i < weight) ud->next = &updates[i];'
is having problems due to i being 0-relative and weight being 1-relative.

Another thing to keep in mind is that cpus can be skipped by checks earlier
in the loop. There is not guarantee that we will add 'weight' elements to
the ud list.

-Nathan
 

quoted

  
-Nathan

Michael

quoted

quoted

 	pr_debug("Topology update for the following CPUs:\n");
 	if (cpumask_weight(&updated_cpus)) {
 		for (ud = &updates[0]; ud; ud = ud->next) {

Simplest change IMO:

		for_each_cpu(sibling, cpu_sibling_mask(cpu)) {
			ud = &updates[i++];
+			ud->next = &updates[i];
                        ud->cpu = sibling;
                        ud->new_nid = new_nid;
                        ud->old_nid = numa_cpu_lookup_table[sibling];
                        cpumask_set_cpu(sibling, &updated_cpus);
-                       if (i < weight)
-                               ud->next = &updates[i];
                }
                cpu = cpu_last_thread_sibling(cpu);

	}

	if (i)
		updates[i-1].next = NULL;

Link all of the updates together, and NULL the link pointer in the
last entry to be filled in.  No worries about invalid comparisons.
Reduced code.

Michael


On 09/07/2017 08:35 AM, Nathan Fontenot wrote:

On 09/06/2017 05:03 PM, Michael Bringmann wrote:

quoted

On 09/06/2017 09:45 AM, Nathan Fontenot wrote:

quoted

On 09/01/2017 10:48 AM, Michael Bringmann wrote:

quoted

powerpc/vphn: On Power systems with shared configurations of CPUs
and memory, there are some issues with the association of additional
CPUs and memory to nodes when hot-adding resources.  This patch
fixes an end-of-updates processing problem observed occasionally
in numa_update_cpu_topology().

Signed-off-by: Michael Bringmann <redacted>
---
 arch/powerpc/mm/numa.c |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 3a5b334..fccf23f 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1410,6 +1410,13 @@ int numa_update_cpu_topology(bool cpus_locked)
 		cpu = cpu_last_thread_sibling(cpu);
 	}

+	/*
+	 * Prevent processing of 'updates' from overflowing array
+	 * in cases where last entry filled in a 'next' pointer.
+	 */
+	if (i)
+		updates[i-1].next = NULL;
+

This really looks like the bug is in the code above this where we
fill in the updates array for each of the sibling cpus. The code
there assumes that if the current update entry is not the end that
there will be more updates and blindly sets the next pointer.

Perhaps correcting the logic in that code to next pointers. Set the
ud pointer to NULL before the outer for_each_cpu() loop. Then in the
inner for_each_cpu(sibling,...) loop update the ud-> next pointer as
the first operation.

		for_each_cpu(sibling, cpu_sibling_mask(cpu)) {
			if (ud)
				ud->next = &updates[i];
			...
		}

Obviously untested, but I think this would prevent setting the next
pointer in the last update entry that is filled out erroneously.

The above fragment looks to skip initialization of the 'next' pointer
in the first element of the the 'updates'.  That would abort subsequent
evaluation of the array too soon, I believe.  I would like to take another look
to see whether the current check 'if (i < weight) ud->next = &updates[i];'
is having problems due to i being 0-relative and weight being 1-relative.

Another thing to keep in mind is that cpus can be skipped by checks earlier
in the loop. There is not guarantee that we will add 'weight' elements to
the ud list.

-Nathan

quoted

quoted

  
-Nathan

Michael

quoted

quoted

 	pr_debug("Topology update for the following CPUs:\n");
 	if (cpumask_weight(&updated_cpus)) {
 		for (ud = &updates[0]; ud; ud = ud->next) {

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com