Re: [PATCH v2] Fix fake numa on ppc

Hi David,

On Tue, Sep 01, 2009 at 11:37:05PM -0700, David Rientjes wrote:

On Wed, 2 Sep 2009, Ankita Garg wrote:

quoted

Hi,

Below is a patch to fix a couple of issues with fake numa node creation
on ppc:

1) Presently, fake nodes could be created such that real numa node
boundaries are not respected. So a node could have lmbs that belong to
different real nodes.

On x86_64, we can use numa=off to completely disable NUMA so that all 
memory and all cpus are mapped to a single node 0.  That's an extreme 
example of the above and is totally permissible.

quoted

2) The cpu association is broken. On a JS22 blade for example, which is
a 2-node numa machine, I get the following:

# cat /proc/cmdline
root=/dev/sda6  numa=fake=2G,4G,,6G,8G,10G,12G,14G,16G
# cat /sys/devices/system/node/node0/cpulist
0-3
# cat /sys/devices/system/node/node1/cpulist
4-7
# cat /sys/devices/system/node/node4/cpulist

#

This doesn't show what the true NUMA topology of the machine is, could you 
please post the output of

	$ cat /sys/devices/system/node/node*/cpulist
	$ cat /sys/devices/system/node/node*/distance
	$ ls -d /sys/devices/system/node/node*/cpu[0-8]

from a normal boot without any numa=fake?

Heres the output as requested by you:

# ls /sys/devices/system/node/
has_cpu  has_normal_memory  node0  node1  online  possible
# cat /sys/devices/system/node/node*/cpulist
0-3
4-7
# cat /sys/devices/system/node/node*/distance
10 20
20 10
# ls -d /sys/devices/system/node/node*/cpu[0-8]
/sys/devices/system/node/node0/cpu0  /sys/devices/system/node/node0/cpu3
/sys/devices/system/node/node1/cpu6
/sys/devices/system/node/node0/cpu1  /sys/devices/system/node/node1/cpu4
/sys/devices/system/node/node1/cpu7
/sys/devices/system/node/node0/cpu2  /sys/devices/system/node/node1/cpu5

 

quoted

So, though the cpus 4-7 should have been associated with node4, they
still belong to node1. The patch works by recording a real numa node
boundary and incrementing the fake node count. At the same time, a
mapping is stored from the real numa node to the first fake node that
gets created on it.

If there are multiple fake nodes on a real physical node, all cpus in that 
node should appear in the cpulist for each fake node for which it has 
local distance.

Currently, the behavior of fake numa is not so on x86 as well? Below is
a sample output from a single node x86 system booted with numa=fake=8:

# cat node0/cpulist

# cat node1/cpulist

...
# cat node6/cpulist

# cat node7/cpulist
0-7

Presently, just fixing the cpu association issue with ppc, as explained
in my previous mail.

-- 
Regards,
Ankita Garg (ankita@in.ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs, 
Bangalore, India