[PATCH 0/4] powerpc/mm: movable hotplug memory nodes

These changes enable onlining memory into ZONE_MOVABLE on power, and the
creation of discrete nodes of movable memory.

Node hotplug is not supported on power [1]. The approach taken instead is to
create a memoryless placeholder node for the designated address range at boot.
Hotplug and onlining of the memory are then done in the usual way.

The numa code on power currently prevents hotplugging to a memoryless node.
This limitation has been questioned before [2], and judging by the response,
there doesn't seem to be a reason we can't remove it. No issues have been
found in light testing.

[1] commit 3af229f ("powerpc/numa: Reset node_possible_map to only node_online_map")
[2] http://lkml.kernel.org/r/CAGZKiBrmkSa1yyhbf5hwGxubcjsE5SmkSMY4tpANERMe2UG4bg@mail.gmail.com
    http://lkml.kernel.org/r/20160511215051.GF22115@arbab-laptop.austin.ibm.com

Reza Arbab (4):
  dt-bindings: add doc for ibm,hotplug-aperture
  powerpc/mm: create numa nodes for hotplug memory
  powerpc/mm: allow memory hotplug into a memoryless node
  mm: enable CONFIG_MOVABLE_NODE on powerpc

 .../bindings/powerpc/opal/hotplug-aperture.txt     | 26 ++++++++++++++++++++++
 Documentation/kernel-parameters.txt                |  2 +-
 arch/powerpc/mm/numa.c                             | 23 ++++++++-----------
 mm/Kconfig                                         |  2 +-
 4 files changed, 37 insertions(+), 16 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/powerpc/opal/hotplug-aperture.txt

-- 
1.8.3.1

Remove the check which prevents us from hotplugging into an empty node.

Signed-off-by: Reza Arbab <redacted>
---
 arch/powerpc/mm/numa.c | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 80d067d..bc70c4f 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -1127,7 +1127,7 @@ static int hot_add_node_scn_to_nid(unsigned long scn_addr)
 int hot_add_scn_to_nid(unsigned long scn_addr)
 {
 	struct device_node *memory = NULL;
-	int nid, found = 0;
+	int nid;
 
 	if (!numa_enabled || (min_common_depth < 0))
 		return first_online_node;

@@ -1143,17 +1143,6 @@ int hot_add_scn_to_nid(unsigned long scn_addr)
 	if (nid < 0 || !node_online(nid))
 		nid = first_online_node;
 
-	if (NODE_DATA(nid)->node_spanned_pages)
-		return nid;
-
-	for_each_online_node(nid) {
-		if (NODE_DATA(nid)->node_spanned_pages) {
-			found = 1;
-			break;
-		}
-	}
-
-	BUG_ON(!found);
 	return nid;
 }
 

-- 
1.8.3.1

Onlining memory into ZONE_MOVABLE requires CONFIG_MOVABLE_NODE.

Enable the use of this config option on PPC64 platforms.

Signed-off-by: Reza Arbab <redacted>
---
 Documentation/kernel-parameters.txt | 2 +-
 mm/Kconfig                          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 46c030a..07fefd8 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt

@@ -2344,7 +2344,7 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			that the amount of memory usable for all allocations
 			is not too small.
 
-	movable_node	[KNL,X86] Boot-time switch to enable the effects
+	movable_node	[KNL,X86,PPC] Boot-time switch to enable the effects
 			of CONFIG_MOVABLE_NODE=y. See mm/Kconfig for details.
 
 	MTD_Partition=	[MTD]

diff --git a/mm/Kconfig b/mm/Kconfig
index 78a23c5..4154638 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig

@@ -153,7 +153,7 @@ config MOVABLE_NODE
 	bool "Enable to assign a node which has only movable memory"
 	depends on HAVE_MEMBLOCK
 	depends on NO_BOOTMEM
-	depends on X86_64
+	depends on X86_64 || PPC64
 	depends on NUMA
 	default n
 	help

-- 
1.8.3.1

When scanning the device tree to initialize the system NUMA topology,
process dt elements with compatible id "ibm,hotplug-aperture" to create
memoryless numa nodes.

These nodes will be filled when hotplug occurs within the associated
address range.

Signed-off-by: Reza Arbab <redacted>
---
 arch/powerpc/mm/numa.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index 75b9cd6..80d067d 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c

@@ -708,6 +708,12 @@ static void __init parse_drconf_memory(struct device_node *memory)
 	}
 }
 
+static const struct of_device_id memory_match[] = {
+	{ .type = "memory" },
+	{ .compatible = "ibm,hotplug-aperture" },
+	{ /* sentinel */ }
+};
+
 static int __init parse_numa_properties(void)
 {
 	struct device_node *memory;

@@ -752,7 +758,7 @@ static int __init parse_numa_properties(void)
 
 	get_n_mem_cells(&n_mem_addr_cells, &n_mem_size_cells);
 
-	for_each_node_by_type(memory, "memory") {
+	for_each_matching_node(memory, memory_match) {
 		unsigned long start;
 		unsigned long size;
 		int nid;

@@ -1080,7 +1086,7 @@ static int hot_add_node_scn_to_nid(unsigned long scn_addr)
 	struct device_node *memory;
 	int nid = -1;
 
-	for_each_node_by_type(memory, "memory") {
+	for_each_matching_node(memory, memory_match) {
 		unsigned long start, size;
 		int ranges;
 		const __be32 *memcell_buf;

-- 
1.8.3.1

Signed-off-by: Reza Arbab <redacted>
---
 .../bindings/powerpc/opal/hotplug-aperture.txt     | 26 ++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/powerpc/opal/hotplug-aperture.txt

diff --git a/Documentation/devicetree/bindings/powerpc/opal/hotplug-aperture.txt b/Documentation/devicetree/bindings/powerpc/opal/hotplug-aperture.txt
new file mode 100644
index 0000000..b8dffaa
--- /dev/null
+++ b/Documentation/devicetree/bindings/powerpc/opal/hotplug-aperture.txt

@@ -0,0 +1,26 @@
+Designated hotplug memory
+-------------------------
+
+This binding describes a region of hotplug memory which is not present at boot,
+allowing its eventual NUMA associativity to be prespecified.
+
+Required properties:
+
+- compatible
+	"ibm,hotplug-aperture"
+
+- reg
+	base address and size of the region (standard definition)
+
+- ibm,associativity
+	NUMA associativity (standard definition)
+
+Example:
+
+A 2 GiB aperture at 0x100000000, to be part of nid 3 when hotplugged:
+
+	hotplug-memory@100000000 {
+		compatible = "ibm,hotplug-aperture";
+		reg = <0x0 0x100000000 0x0 0x80000000>;
+		ibm,associativity = <0x4 0x0 0x0 0x0 0x3>;
+	};

-- 
1.8.3.1

On Wed, Aug 10, 2016 at 08:30:28PM +1000, Michael Ellerman wrote:

Reza Arbab [off-list ref] writes:

quoted

Node hotplug is not supported on power [1].

But maybe it should be?

Doing so will involve, at the very least, reverting the commit I cited, 
3af229f2071f ("powerpc/numa: Reset node_possible_map to only 
node_online_map"), and fixing that issue in a different way.

I'll look into it and see what I can do.

-- 
Reza Arbab

On 08/10/2016 05:30 AM, Michael Ellerman wrote:

Reza Arbab [off-list ref] writes:

quoted

These changes enable onlining memory into ZONE_MOVABLE on power, and the
creation of discrete nodes of movable memory.

Node hotplug is not supported on power [1].

But maybe it should be?

Yes, it should be supported.

I have briefly looked into this recently only to find
this will not be a simple update.

-Nathan 

Reza Arbab [off-list ref] writes:

These changes enable onlining memory into ZONE_MOVABLE on power, and the
creation of discrete nodes of movable memory.

Node hotplug is not supported on power [1].

But maybe it should be?

cheers

On Mon, Aug 08, 2016 at 01:27:20PM -0500, Reza Arbab wrote:

Signed-off-by: Reza Arbab <redacted>
---
 .../bindings/powerpc/opal/hotplug-aperture.txt     | 26 ++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/powerpc/opal/hotplug-aperture.txt

Acked-by: Rob Herring <robh@kernel.org>

On 09/08/16 04:27, Reza Arbab wrote:

+Stewart and Alistair

Looks good to me!

Acked-by: Balbir Singh <bsingharora@gmail.com>

Balbir Singh [off-list ref] writes:

On 09/08/16 04:27, Reza Arbab wrote:

quoted

Signed-off-by: Reza Arbab <redacted>
---
 .../bindings/powerpc/opal/hotplug-aperture.txt     | 26 ++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100644
Documentation/devicetree/bindings/powerpc/opal/hotplug-aperture.txt

Forgive me for being absent on the whole discussion here, but is this an
OPAL specific binding? If so, shouldn't the docs also appear in the
skiboot tree?

-- 
Stewart Smith
OPAL Architect, IBM.

On Thu, Aug 11, 2016 at 02:39:23PM +1000, Stewart Smith wrote:

Forgive me for being absent on the whole discussion here, but is this 
an OPAL specific binding? If so, shouldn't the docs also appear in the
skiboot tree?

Good question. I guess it's not necessarily OPAL-specific, even though 
OPAL may initially be the only implementor of the binding.

Would it be more appropriate to move the file up a directory, directly 
under Documentation/devicetree/bindings/powerpc? I hesitated at that 
because the binding is tied to "ibm,associativity".

-- 
Reza Arbab

On 09/08/16 04:27, Reza Arbab wrote:

When scanning the device tree to initialize the system NUMA topology,
process dt elements with compatible id "ibm,hotplug-aperture" to create
memoryless numa nodes.

These nodes will be filled when hotplug occurs within the associated
address range.

Signed-off-by: Reza Arbab <redacted>
---

Looks good to me

Acked-by: Balbir Singh <bsingharora@gmail.com>

On 09/08/16 04:27, Reza Arbab wrote:

Do we want to do this only for ibm,hotplug-aperture compatible ranges?

I'm OK either ways

Acked-by: Balbir Singh <bsingharora@gmail.com>

On Fri, Aug 12, 2016 at 11:50:43AM +1000, Balbir Singh wrote:

On 09/08/16 04:27, Reza Arbab wrote:

quoted

Remove the check which prevents us from hotplugging into an empty node.

Do we want to do this only for ibm,hotplug-aperture compatible ranges?

We could, but since past discussions and current testing have been 
unable to justify preventing hotplug to a memoryless node in the first 
place, I'm inclined to keep things simple.

If some edge case is discovered, making it conditional as you describe 
will be a good solution.

Thanks for your review! A v2 of this set is pending my investigation of 
Michael's suggestion to get node hotadd working.

-- 
Reza Arbab