[PATCH] Fake NUMA emulation for PowerPC

Here's a dumb simple implementation of fake NUMA nodes for PowerPC. Fake
NUMA nodes can be specified using the following command line option

numa=fake=<node range>

node range is of the format <range1>,<range2>,...<rangeN>

Each of the rangeX parameters is passed using memparse(). I find the patch
useful for fake NUMA emulation on my simple PowerPC machine. I've tested it
on a non-numa box with the following arguments

numa=fake=1G
numa=fake=1G,2G
name=fake=1G,512M,2G
numa=fake=1500M,2800M mem=3500M
numa=fake=1G mem=512M
numa=fake=1G mem=1G

This patch applies on top of 2.6.24-rc4.

All though I've tried my best to handle some of the architecture specific
details of PowerPC, I might have overlooked something obvious, like the usage
of an API or some architecture tweaks. The patch depends on CONFIG_NUMA and
I decided against creating a separate config option for fake NUMA to keep
the code simple.

Comments are as always welcome!

Signed-off-by: Balbir Singh <redacted>
---

 arch/powerpc/mm/numa.c |   55 ++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 50 insertions(+), 5 deletions(-)

diff -puN arch/powerpc/mm/numa.c~ppc-fake-numa-easy arch/powerpc/mm/numa.c

--- linux-2.6.24-rc4-mm1/arch/powerpc/mm/numa.c~ppc-fake-numa-easy	2007-12-07 21:25:55.000000000 +0530
+++ linux-2.6.24-rc4-mm1-balbir/arch/powerpc/mm/numa.c	2007-12-08 02:36:02.000000000 +0530

@@ -24,6 +24,8 @@
 
 static int numa_enabled = 1;
 
+char *cmdline __initdata;
+
 static int numa_debug;
 #define dbg(args...) if (numa_debug) { printk(KERN_INFO args); }
 

@@ -39,6 +41,40 @@ static bootmem_data_t __initdata plat_no
 static int min_common_depth;
 static int n_mem_addr_cells, n_mem_size_cells;
 
+static int __cpuinit fake_numa_create_new_node(unsigned long end_pfn,
+						unsigned int *nid)
+{
+	unsigned long long mem;
+	char *p = cmdline;
+	static unsigned int fake_nid = 0;
+	static unsigned long long curr_boundary = 0;
+
+	*nid = fake_nid;
+	mem = memparse(p, &p);
+	if (!mem)
+		return 0;
+
+	if (mem < curr_boundary)
+		return 0;
+
+	curr_boundary = mem;
+
+	if ((end_pfn << PAGE_SHIFT) > mem) {
+		/*
+		 * Skip commas and spaces
+		 */
+		while (*p == ',' || *p == ' ' || *p == '\t')
+			p++;
+
+		cmdline = p;
+		fake_nid++;
+		*nid = fake_nid;
+		dbg("created new fake_node with id %d\n", fake_nid);
+		return 1;
+	}
+	return 0;
+}
+
 static void __cpuinit map_cpu_to_node(int cpu, int node)
 {
 	numa_cpu_lookup_table[cpu] = node;

@@ -344,12 +380,14 @@ static void __init parse_drconf_memory(s
 			if (nid == 0xffff || nid >= MAX_NUMNODES)
 				nid = default_nid;
 		}
-		node_set_online(nid);
 
 		size = numa_enforce_memory_limit(start, lmb_size);
 		if (!size)
 			continue;
 
+		fake_numa_create_new_node(((start + size) >> PAGE_SHIFT), &nid);
+		node_set_online(nid);
+
 		add_active_range(nid, start >> PAGE_SHIFT,
 				 (start >> PAGE_SHIFT) + (size >> PAGE_SHIFT));
 	}

@@ -429,7 +467,6 @@ new_range:
 		nid = of_node_to_nid_single(memory);
 		if (nid < 0)
 			nid = default_nid;
-		node_set_online(nid);
 
 		if (!(size = numa_enforce_memory_limit(start, size))) {
 			if (--ranges)

@@ -438,6 +475,9 @@ new_range:
 				continue;
 		}
 
+		fake_numa_create_new_node(((start + size) >> PAGE_SHIFT), &nid);
+		node_set_online(nid);
+
 		add_active_range(nid, start >> PAGE_SHIFT,
 				(start >> PAGE_SHIFT) + (size >> PAGE_SHIFT));
 

@@ -461,7 +501,7 @@ static void __init setup_nonnuma(void)
 	unsigned long top_of_ram = lmb_end_of_DRAM();
 	unsigned long total_ram = lmb_phys_mem_size();
 	unsigned long start_pfn, end_pfn;
-	unsigned int i;
+	unsigned int i, nid = 0;
 
 	printk(KERN_DEBUG "Top of RAM: 0x%lx, Total RAM: 0x%lx\n",
 	       top_of_ram, total_ram);

@@ -471,9 +511,11 @@ static void __init setup_nonnuma(void)
 	for (i = 0; i < lmb.memory.cnt; ++i) {
 		start_pfn = lmb.memory.region[i].base >> PAGE_SHIFT;
 		end_pfn = start_pfn + lmb_size_pages(&lmb.memory, i);
-		add_active_range(0, start_pfn, end_pfn);
+
+		fake_numa_create_new_node(end_pfn, &nid);
+		add_active_range(nid, start_pfn, end_pfn);
+		node_set_online(nid);
 	}
-	node_set_online(0);
 }
 
 void __init dump_numa_cpu_topology(void)

@@ -702,6 +744,9 @@ static int __init early_numa(char *p)
 	if (strstr(p, "debug"))
 		numa_debug = 1;
 
+	if (strstr(p, "fake="))
+		cmdline = p + 5;	/* 5 is faster than strlen("fake=") */
+
 	return 0;
 }
 early_param("numa", early_numa);

_

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Hi,

On Sat, Dec 08, 2007 at 02:44:25AM +0530, Balbir Singh wrote:

Comments are as always welcome!

Care to explain what this is useful for? (Not saying it's a stupid idea,
just wondering what the reason for doing it is).

Looks like this should be static.

This doesn't look right.

You check if it contains fake=, not if it starts with it. So if someone
did: "numa=foo,fake=bar", or even "numa=debug,fake=", things wouldn't
work right.


-Olof

Olof Johansson wrote:

Hi,

On Sat, Dec 08, 2007 at 02:44:25AM +0530, Balbir Singh wrote:

quoted

Comments are as always welcome!

Care to explain what this is useful for? (Not saying it's a stupid idea,
just wondering what the reason for doing it is).

In my case, I use it to test parts of my memory controller patches on an
emulated NUMA machine. I plan to use it to test out page migration
across nodes.

quoted

diff -puN arch/powerpc/mm/numa.c~ppc-fake-numa-easy arch/powerpc/mm/numa.c

--- linux-2.6.24-rc4-mm1/arch/powerpc/mm/numa.c~ppc-fake-numa-easy	2007-12-07 21:25:55.000000000 +0530
+++ linux-2.6.24-rc4-mm1-balbir/arch/powerpc/mm/numa.c	2007-12-08 02:36:02.000000000 +0530

@@ -24,6 +24,8 @@
 
 static int numa_enabled = 1;
 
+char *cmdline __initdata;
+

Looks like this should be static.

Yes, good catch!

quoted

@@ -702,6 +744,9 @@ static int __init early_numa(char *p)
 	if (strstr(p, "debug"))
 		numa_debug = 1;
 
+	if (strstr(p, "fake="))
+		cmdline = p + 5;	/* 5 is faster than strlen("fake=") */

This doesn't look right.

You check if it contains fake=, not if it starts with it. So if someone
did: "numa=foo,fake=bar", or even "numa=debug,fake=", things wouldn't
work right.

Yes, you are right. I merely followed the strstr convention already
present, which as you righly point out is wrong. I suspect I need to do
something like

p = strstr(p, "fake=")
if (p)
	cmdline = p + 5;

This would still allow us to do things like

numa=foo,fake=bar but the memparse() utility would fail at fake=bar
								^^^

or even

numa=debug,fake=1G

I suspect that this should be good enough for a command line option.

-Olof

-- 
	Thanks,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

On Dec 7, 2007, at 3:35 PM, Balbir Singh wrote:

Olof Johansson wrote:

quoted

Hi,

On Sat, Dec 08, 2007 at 02:44:25AM +0530, Balbir Singh wrote:

quoted

Comments are as always welcome!

Care to explain what this is useful for? (Not saying it's a stupid  
idea,
just wondering what the reason for doing it is).

In my case, I use it to test parts of my memory controller patches  
on an
emulated NUMA machine. I plan to use it to test out page migration
across nodes.

Can you explain that further.  I'm still not clear on why this is  
useful.

- k

Kumar Gala wrote:

On Dec 7, 2007, at 3:35 PM, Balbir Singh wrote:

quoted

Olof Johansson wrote:

quoted

Hi,

On Sat, Dec 08, 2007 at 02:44:25AM +0530, Balbir Singh wrote:

quoted

Comments are as always welcome!

Care to explain what this is useful for? (Not saying it's a stupid idea,
just wondering what the reason for doing it is).

In my case, I use it to test parts of my memory controller patches on an
emulated NUMA machine. I plan to use it to test out page migration
across nodes.

Can you explain that further.  I'm still not clear on why this is useful.

- k

Sure. In my case I need to emulate NUMA nodes to do some NUMA specific
testing. The memory controller I've written has some interesting data
structures like per node, per zone LRU lists. To be able to test those
features on a non-numa box is a problem, since we get just the default node.

To be able to test the memory controller under NUMA, I use fake NUMA
nodes. x86-64 has a similar feature, the code I have here is the
simplest I could come up with for PowerPC.

I just thought of another very interesting use case, it can be used to
split up the zone's lru lock which is highly contended.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

On Dec 7, 2007, at 4:12 PM, Balbir Singh wrote:

Kumar Gala wrote:

quoted

On Dec 7, 2007, at 3:35 PM, Balbir Singh wrote:

quoted

Olof Johansson wrote:

quoted

Hi,

On Sat, Dec 08, 2007 at 02:44:25AM +0530, Balbir Singh wrote:

quoted

Comments are as always welcome!

Care to explain what this is useful for? (Not saying it's a  
stupid idea,
just wondering what the reason for doing it is).

In my case, I use it to test parts of my memory controller patches  
on an
emulated NUMA machine. I plan to use it to test out page migration
across nodes.

Can you explain that further.  I'm still not clear on why this is  
useful.

- k

Sure. In my case I need to emulate NUMA nodes to do some NUMA specific
testing. The memory controller I've written has some interesting data
structures like per node, per zone LRU lists. To be able to test those
features on a non-numa box is a problem, since we get just the  
default node.

Maybe I'm missing something, what do you mean by memory controller  
you've written?  (I'm use to the term 'memory controller' meaning the  
actual RAM control).

To be able to test the memory controller under NUMA, I use fake NUMA
nodes. x86-64 has a similar feature, the code I have here is the
simplest I could come up with for PowerPC.

I just thought of another very interesting use case, it can be used to
split up the zone's lru lock which is highly contended.

- k

Kumar Gala wrote:

On Dec 7, 2007, at 4:12 PM, Balbir Singh wrote:

quoted

Kumar Gala wrote:

quoted

On Dec 7, 2007, at 3:35 PM, Balbir Singh wrote:

quoted

Olof Johansson wrote:

quoted

Hi,

On Sat, Dec 08, 2007 at 02:44:25AM +0530, Balbir Singh wrote:

quoted

Comments are as always welcome!

Care to explain what this is useful for? (Not saying it's a stupid
idea,
just wondering what the reason for doing it is).

In my case, I use it to test parts of my memory controller patches
on an
emulated NUMA machine. I plan to use it to test out page migration
across nodes.

Can you explain that further.  I'm still not clear on why this is
useful.

- k

Sure. In my case I need to emulate NUMA nodes to do some NUMA specific
testing. The memory controller I've written has some interesting data
structures like per node, per zone LRU lists. To be able to test those
features on a non-numa box is a problem, since we get just the default
node.

Maybe I'm missing something, what do you mean by memory controller
you've written?  (I'm use to the term 'memory controller' meaning the
actual RAM control).

Ah! that explains the disconnect. If you look at the latest -mm tree. We
have a memory controller under control groups, we use it to control how
much memory a group of process can access at a time.

quoted

To be able to test the memory controller under NUMA, I use fake NUMA
nodes. x86-64 has a similar feature, the code I have here is the
simplest I could come up with for PowerPC.

I just thought of another very interesting use case, it can be used to
split up the zone's lru lock which is highly contended.

- k

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

On Sat, 8 Dec 2007, Balbir Singh wrote:

To be able to test the memory controller under NUMA, I use fake NUMA
nodes. x86-64 has a similar feature, the code I have here is the
simplest I could come up with for PowerPC.

Magnus Damm had patches from over a year ago that, I believe, made much of 
the x86_64 fake NUMA code generic so that it could be extended for 
architectures such as i386.  Perhaps he could resurrect those patches if 
there is wider interest in such a tool.

David Rientjes wrote:

On Sat, 8 Dec 2007, Balbir Singh wrote:

quoted

To be able to test the memory controller under NUMA, I use fake NUMA
nodes. x86-64 has a similar feature, the code I have here is the
simplest I could come up with for PowerPC.

Magnus Damm had patches from over a year ago that, I believe, made much of 
the x86_64 fake NUMA code generic so that it could be extended for 
architectures such as i386.  Perhaps he could resurrect those patches if 
there is wider interest in such a tool.

That would be a very interesting patch, but what I have here is the
simplest patch and we could build on it incrementally. The interface is
non-standard but it does amazing things for 59 lines of code change.


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

On Sat 2007-12-08 09:52:06, Balbir Singh wrote:

David Rientjes wrote:

quoted

On Sat, 8 Dec 2007, Balbir Singh wrote:

quoted

To be able to test the memory controller under NUMA, I use fake NUMA
nodes. x86-64 has a similar feature, the code I have here is the
simplest I could come up with for PowerPC.

Magnus Damm had patches from over a year ago that, I believe, made much of 
the x86_64 fake NUMA code generic so that it could be extended for 
architectures such as i386.  Perhaps he could resurrect those patches if 
there is wider interest in such a tool.

That would be a very interesting patch, but what I have here is the
simplest patch and we could build on it incrementally. The interface is
non-standard but it does amazing things for 59 lines of code change.

Well, maybe it is amazing, but having non-standard interface is also
wrong...

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Pavel Machek wrote:

On Sat 2007-12-08 09:52:06, Balbir Singh wrote:

quoted

David Rientjes wrote:

quoted

On Sat, 8 Dec 2007, Balbir Singh wrote:

quoted

To be able to test the memory controller under NUMA, I use fake NUMA
nodes. x86-64 has a similar feature, the code I have here is the
simplest I could come up with for PowerPC.

Magnus Damm had patches from over a year ago that, I believe, made much of 
the x86_64 fake NUMA code generic so that it could be extended for 
architectures such as i386.  Perhaps he could resurrect those patches if 
there is wider interest in such a tool.

That would be a very interesting patch, but what I have here is the
simplest patch and we could build on it incrementally. The interface is
non-standard but it does amazing things for 59 lines of code change.

Well, maybe it is amazing, but having non-standard interface is also
wrong...

I tend to agree with you, but in this case it's mostly debug
infrastructure that is architecture specific.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

On Fri, 7 Dec 2007, Olof Johansson wrote:

quoted

Comments are as always welcome!

Care to explain what this is useful for? (Not saying it's a stupid idea,
just wondering what the reason for doing it is).

Fake NUMA has always been useful for testing NUMA code without having to 
have a wide range of hardware available to you.  It's a clever tool on 
x86_64 intended for kernel developers that simply makes it easier to test 
code and adds an increased level of robustness to the kernel.  I think 
it's a valuable addition.

On Sat, 8 Dec 2007, Balbir Singh wrote:

+	if (strstr(p, "fake="))
+		cmdline = p + 5;	/* 5 is faster than strlen("fake=") */

Really? My gcc is smart enough to replace the `strlen("fake=")' by 5, even
without -O.

With kind regards,
 
Geert Uytterhoeven
Software Architect

Sony Network and Software Technology Center Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
 
Phone:    +32 (0)2 700 8453	
Fax:      +32 (0)2 700 8622	
E-mail:   Geert.Uytterhoeven@sonycom.com	
Internet: http://www.sony-europe.com/
 	
Sony Network and Software Technology Center Europe	
A division of Sony Service Centre (Europe) N.V.	
Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium	
VAT BE 0413.825.160 · RPR Brussels	
Fortis Bank Zaventem · Swift GEBABEBB08A · IBAN BE39001382358619

Geert Uytterhoeven wrote:

On Sat, 8 Dec 2007, Balbir Singh wrote:

quoted

+	if (strstr(p, "fake="))
+		cmdline = p + 5;	/* 5 is faster than strlen("fake=") */

Really? My gcc is smart enough to replace the `strlen("fake=")' by 5, even
without -O.

Thanks for pointing that out, but I am surprised that a compiler would
interpret library routines like strlen.

With kind regards,

Geert Uytterhoeven
Software Architect

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Balbir Singh wrote:

Geert Uytterhoeven wrote:

quoted

On Sat, 8 Dec 2007, Balbir Singh wrote:

quoted

+	if (strstr(p, "fake="))
+		cmdline = p + 5;	/* 5 is faster than strlen("fake=") */

Really? My gcc is smart enough to replace the `strlen("fake=")' by 5, even
without -O.

Thanks for pointing that out, but I am surprised that a compiler would
interpret library routines like strlen.

I just tested it and it turns out that you are right. I'll go hunt to
see where gcc gets its magic powers from.

quoted

With kind regards,

Geert Uytterhoeven
Software Architect

On Friday 07 December 2007, Balbir Singh wrote:

Balbir Singh wrote:

quoted

Geert Uytterhoeven wrote:

quoted

On Sat, 8 Dec 2007, Balbir Singh wrote:

quoted

+=A0=A0=A0if (strstr(p, "fake=3D"))
+=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0cmdline =3D p + 5;=A0=A0=A0=A0=A0=

=A0=A0=A0/* 5 is faster than strlen("fake=3D") */

quoted

quoted

Really? My gcc is smart enough to replace the `strlen("fake=3D")' by 5=

, even

quoted

quoted

without -O.

=20
Thanks for pointing that out, but I am surprised that a compiler would
interpret library routines like strlen.
=20

=20
I just tested it and it turns out that you are right. I'll go hunt to
see where gcc gets its magic powers from.
=20

Even if it wasn't: Why the heck would you want to optimize this? The functi=
on
is run _once_ at boot time and the object code gets thrown away afterwards!

	Arnd <><

Arnd Bergmann wrote:

On Friday 07 December 2007, Balbir Singh wrote:

quoted

Balbir Singh wrote:

quoted

Geert Uytterhoeven wrote:

quoted

On Sat, 8 Dec 2007, Balbir Singh wrote:

quoted

+   if (strstr(p, "fake="))
+           cmdline = p + 5;        /* 5 is faster than strlen("fake=") */

Really? My gcc is smart enough to replace the `strlen("fake=")' by 5, even
without -O.

Thanks for pointing that out, but I am surprised that a compiler would
interpret library routines like strlen.

I just tested it and it turns out that you are right. I'll go hunt to
see where gcc gets its magic powers from.

Even if it wasn't: Why the heck would you want to optimize this? The function
is run _once_ at boot time and the object code gets thrown away afterwards!

	Arnd <><

Cause, I see no downside of doing it. The strlen of fake= is fixed.
But having said that, I am not a purist about the approach, I just want
cmdline to point after "fake="

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

On Friday 07 December 2007, Balbir Singh wrote:

Here's a dumb simple implementation of fake NUMA nodes for PowerPC. Fake
NUMA nodes can be specified using the following command line option

numa=fake=<node range>

node range is of the format <range1>,<range2>,...<rangeN>

Excellent idea! I'd love to have this in RHEL5u1, because that would make
that distro boot on certain machines that have more memory than is supported
without an iommu driver. The problem we have is that when you simply
say mem=1G but all of the first gigabyte is on the first node, you end
up with a memoryless node, which is not supported.

Unfortunately, it comes too late for me now, as all new distros already boot
on Cell machines that need an IOMMU.

	Arnd <><

Arnd Bergmann wrote:

On Friday 07 December 2007, Balbir Singh wrote:

quoted

Here's a dumb simple implementation of fake NUMA nodes for PowerPC. Fake
NUMA nodes can be specified using the following command line option

numa=fake=<node range>

node range is of the format <range1>,<range2>,...<rangeN>

Excellent idea! I'd love to have this in RHEL5u1, because that would make
that distro boot on certain machines that have more memory than is supported
without an iommu driver. The problem we have is that when you simply
say mem=1G but all of the first gigabyte is on the first node, you end
up with a memoryless node, which is not supported.

Unfortunately, it comes too late for me now, as all new distros already boot
on Cell machines that need an IOMMU.

Very interesting use case! I am sure there are others were fake NUMA
nodes can be applied. I just listed one other in another email, apart
from using it for playing around with NUMA like machines.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL