From: Peter Zijlstra <hidden> Date: 2011-07-07 11:00:02
On Thu, 2011-07-07 at 15:52 +0530, Mahesh J Salgaonkar wrote:
=20
2.6.39 booted fine on the system and a git bisect shows commit cd4ea6ae -
"sched: Change NODE sched_domain group creation" as the cause.
Weird, there's no locking anywhere around there. The typical problems
with this patch-set were massive explosions due to bad pointers etc..
But not silent hangs.
The code its stuck at:
doesn't contains any locks, its simply looping over all the cpus, and
with that many I can imagine it takes a while, but getting 'stuck' there
is unexpected to say the least.
Surely this isn't the first multi-node P7 to boot a kernel with this
patch? If my git foo is any good it hit -next on 23rd of May.
I guess I'm asking is, do smaller P7 machines boot? And if so, is there
any difference except size?
How many nodes does the thing have anyway, 28? Hmm, that could mean its
the first machine with >16 nodes to boot this, which would make it
trigger the magic ALL_NODES crap.
Let me dig around there.
On Thu, 2011-07-07 at 15:52 +0530, Mahesh J Salgaonkar wrote:
quoted
2.6.39 booted fine on the system and a git bisect shows commit cd4ea6ae -
"sched: Change NODE sched_domain group creation" as the cause.
Weird, there's no locking anywhere around there. The typical problems
with this patch-set were massive explosions due to bad pointers etc..
But not silent hangs.
The code its stuck at:
doesn't contains any locks, its simply looping over all the cpus, and
with that many I can imagine it takes a while, but getting 'stuck' there
is unexpected to say the least.
Surely this isn't the first multi-node P7 to boot a kernel with this
patch? If my git foo is any good it hit -next on 23rd of May.
I guess I'm asking is, do smaller P7 machines boot? And if so, is there
any difference except size?
Yes, the smaller P7 machine that I have with 20 CPUs and 2GB ram boots
fine with 3.0.0-rc.
How many nodes does the thing have anyway, 28? Hmm, that could mean its
the first machine with >16 nodes to boot this, which would make it
trigger the magic ALL_NODES crap.
The P7 machine where kernel fails to boot shows following demsg log w.r.t
node map:
---------------------------
Zone PFN ranges:
DMA 0x00000000 -> 0x01229000
Normal empty
Movable zone start PFN for each node
early_node_map[12] active PFN ranges
0: 0x00000000 -> 0x000fd000
4: 0x000fd000 -> 0x002fb000
5: 0x002fb000 -> 0x004b9000
6: 0x004b9000 -> 0x006b9000
8: 0x006b9000 -> 0x007b5000
12: 0x007b5000 -> 0x008b5000
16: 0x008b5000 -> 0x009b1000
20: 0x009b1000 -> 0x00bb1000
21: 0x00bb1000 -> 0x00db1000
22: 0x00db1000 -> 0x00fb1000
23: 0x00fb1000 -> 0x011b1000
28: 0x011b1000 -> 0x01229000
Could not find start_pfn for node 1
Could not find start_pfn for node 2
Could not find start_pfn for node 3
Could not find start_pfn for node 7
Could not find start_pfn for node 9
Could not find start_pfn for node 10
Could not find start_pfn for node 11
Could not find start_pfn for node 13
Could not find start_pfn for node 14
Could not find start_pfn for node 15
Could not find start_pfn for node 17
Could not find start_pfn for node 18
Could not find start_pfn for node 19
Could not find start_pfn for node 29
Could not find start_pfn for node 30
Could not find start_pfn for node 31
[boot]0015 Setup Done
PERCPU: Embedded 1 pages/cpu @c000000013c00000 s31488 r0 d34048 u65536
Built 28 zonelists in Node order, mobility grouping on. Total pages:
19026032
Policy zone: DMA
Kernel command line: root=/dev/mapper/vg_nish1-lv_root ro
rd_LVM_LV=vg_nish1/lv_root rd_LVM_LV=VolGroup/lv_swap
rd_LVM_LV=vg_nish1/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8
SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=hvc0i memblock=debug
PID hash table entries: 4096 (order: -1, 32768 bytes)
freeing bootmem node 0
freeing bootmem node 4
freeing bootmem node 5
freeing bootmem node 6
freeing bootmem node 8
freeing bootmem node 12
freeing bootmem node 16
freeing bootmem node 20
freeing bootmem node 21
freeing bootmem node 22
freeing bootmem node 23
freeing bootmem node 28
Memory: 1213775296k/1218707456k available (13312k kernel code, 4932160k
reserved, 1600k data, 2727k bss, 4928k init)
---------------------------
Thanks,
-Mahesh.
From: Anton Blanchard <hidden> Date: 2011-07-14 00:34:21
Hi Peter,
Surely this isn't the first multi-node P7 to boot a kernel with this
patch? If my git foo is any good it hit -next on 23rd of May.
I guess I'm asking is, do smaller P7 machines boot? And if so, is
there any difference except size?
How many nodes does the thing have anyway, 28? Hmm, that could mean
its the first machine with >16 nodes to boot this, which would make it
trigger the magic ALL_NODES crap.
We haven't tested a box with more than 16 nodes in quite a while, so it
may be this.
I took a quick look and we are stuck in update_group_power:
do {
power += group->cpu_power;
group = group->next;
} while (group != child->groups);
I looked at the linked list:
child->groups = c000007b2f74ff00
and dumping group as we go:
c000007b2f74ff00 c000007b2f760000 c000007b2fb60000 c000007b2ff60000
at this point we end up in a cycle and never make it back to
child->groups:
c000008b2e68ff00 c000008b2e6a0000 c000008b2eaa0000 c000008b2eea0000
c000009aee77ff00 c000009aee790000 c000009aeeb90000 c000009aeef90000
c00000bafde91800 c00000dafdf81800 c00000fafce81800 c000011afdf71800
c00001226e70ff00 c00001226e720000 c00001226eb20000 c00001226ef20000
c000008b2e68ff00
Still investigating
Anton
From: Anton Blanchard <hidden> Date: 2011-07-14 04:35:24
I took a quick look and we are stuck in update_group_power:
do {
power += group->cpu_power;
group = group->next;
} while (group != child->groups);
I looked at the linked list:
child->groups = c000007b2f74ff00
and dumping group as we go:
c000007b2f74ff00 c000007b2f760000 c000007b2fb60000 c000007b2ff60000
at this point we end up in a cycle and never make it back to
child->groups:
c000008b2e68ff00 c000008b2e6a0000 c000008b2eaa0000 c000008b2eea0000
c000009aee77ff00 c000009aee790000 c000009aeeb90000 c000009aeef90000
c00000bafde91800 c00000dafdf81800 c00000fafce81800 c000011afdf71800
c00001226e70ff00 c00001226e720000 c00001226eb20000 c00001226ef20000
c000008b2e68ff00
It looks like the group ends up in two lists. I added a BUG_ON to
ensure we never link a group twice, and it hits.
I also printed out the cpu spans as we walk through build_sched_groups:
0 1 2 3
0 4 8 12 16 20 24 28
0 32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
0 128 256 384
4 5 6 7
8 9 10 11
12 13 14 15
16 17 18 19
20 21 22 23
24 25 26 27
28 29 30 31
32 33 34 35
32 36 40 44 48 52 56 60
36 37 38 39
40 41 42 43
44 45 46 47
48 49 50 51
52 53 54 55
56 57 58 59
60 61 62 63
64 65 66 67
64 68 72 76 80 84 88 92
68 69 70 71
72 73 74 75
76 77 78 79
80 81 82 83
84 85 86 87
88 89 90 91
92 93 94 95
96 97 98 99
96 100 104 108 112 116 120 124
100 101 102 103
104 105 106 107
108 109 110 111
112 113 114 115
116 117 118 119
120 121 122 123
124 125 126 127
128 129 130 131
128 132 136 140 144 148 152 156
Duplicates start appearing in this span:
128 160 192 224 256 288 320 352 384 416 448 480 512 544 576 608
So it looks like the overlap of the 16 entry spans
(SD_NODES_PER_DOMAIN) is causing our problem.
Anton
Index: linux-2.6-work/kernel/sched.c
===================================================================
Duplicates start appearing in this span:
128 160 192 224 256 288 320 352 384 416 448 480 512 544 576 608
=20
So it looks like the overlap of the 16 entry spans
(SD_NODES_PER_DOMAIN) is causing our problem.
Urgh.. so those spans are generated by sched_domain_node_span(), and it
looks like that simply picks the 15 nearest nodes to the one we've got
without consideration for overlap with previously generated spans.
Now that used to work because it used to simply allocate a new group
instead of using the existing one.
The thing is, we want to track state unique to a group of cpus, so
duplicating that is iffy.
Otoh, making these masks non-overlapping is probably sub-optimal from a
NUMA pov.
Looking at a slightly simpler set-up (4 socket AMD magny-cours):
$ cat /sys/devices/system/node/node*/distance
10 16 16 22 16 22 16 22
16 10 22 16 22 16 22 16
16 22 10 16 16 22 16 22
22 16 16 10 22 16 22 16
16 22 16 22 10 16 16 22
22 16 22 16 16 10 22 16
16 22 16 22 16 22 10 16
22 16 22 16 22 16 16 10
We can translate that into groups like
{0} {0,1,2,4,6} {0-7}
{1} {1,0,3,5,7} {0-7}
...
and we can easily see there's overlap there as well in the NUMA layout
itself.
This seems to suggest we need to separate the unique state from the
sched_group.
Now all I need is a way to not consume gobs of memory.. /me goes prod
From: Anton Blanchard <hidden> Date: 2011-07-15 00:45:58
Hi,
Urgh.. so those spans are generated by sched_domain_node_span(), and
it looks like that simply picks the 15 nearest nodes to the one we've
got without consideration for overlap with previously generated spans.
I do wonder if we need this extra level at all on ppc64. From memory
SGI added it for their massive setups, but our largest setup is 32 nodes
and breaking that down into 16 node chunks seems overkill.
I just realised we were setting NEWIDLE on our node definition and that
was causing large amounts of rebalance work even with
SD_NODES_PER_DOMAIN=16.
After removing it and bumping SD_NODES_PER_DOMAIN to 32, things look
pretty good.
Perhaps we should allow an arch to override SD_NODES_PER_DOMAIN so this
extra level is only used by SGI boxes.
Anton
From: Peter Zijlstra <hidden> Date: 2011-07-15 08:37:53
On Fri, 2011-07-15 at 10:45 +1000, Anton Blanchard wrote:
Hi,
=20
quoted
Urgh.. so those spans are generated by sched_domain_node_span(), and
it looks like that simply picks the 15 nearest nodes to the one we've
got without consideration for overlap with previously generated spans.
=20
I do wonder if we need this extra level at all on ppc64. From memory
SGI added it for their massive setups, but our largest setup is 32 nodes
and breaking that down into 16 node chunks seems overkill.
=20
I just realised we were setting NEWIDLE on our node definition and that
was causing large amounts of rebalance work even with
SD_NODES_PER_DOMAIN=3D16.
=20
After removing it and bumping SD_NODES_PER_DOMAIN to 32, things look
pretty good.
=20
Perhaps we should allow an arch to override SD_NODES_PER_DOMAIN so this
extra level is only used by SGI boxes.
We can certainly remove the whole topology layer that causes this
problem for 3.0 and try to fix up for 3.1 again.
But I was rather hoping to introduce more of those layers in the near
future, I was hoping to create a layer per node_distance() value, such
that the load-balancing is aware of the interconnects.
Now for that I ran into the exact same problem, and at the time didn't
come up with a solution, but I think I now see a way out.
Something like the below ought to avoid the problem.. makes SGI sad
though :-)
---
kernel/sched.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
From: Peter Zijlstra <hidden> Date: 2011-07-18 21:31:37
Anton, could you test the below two patches on that machine?
It should make things boot again, while I don't have a machine nearly
big enough to trigger any of this, I tested the new code paths by
setting FORCE_SD_OVERLAP in /debug/sched_features. Although any review
of the error paths would be much appreciated.
Also, could you send me the node_distance table for that machine? I'm
curious what the interconnects look like on that thing.
From: Anton Blanchard <hidden> Date: 2011-07-19 04:44:56
On Mon, 18 Jul 2011 23:35:56 +0200
Peter Zijlstra [off-list ref] wrote:
Anton, could you test the below two patches on that machine?
It should make things boot again, while I don't have a machine nearly
big enough to trigger any of this, I tested the new code paths by
setting FORCE_SD_OVERLAP in /debug/sched_features. Although any review
of the error paths would be much appreciated.
I get an oops in slub code:
NIP [c000000000197d30] .deactivate_slab+0x1b0/0x200
LR [c000000000199d94] .__slab_alloc+0xb4/0x5a0
[c000000000199d94] .__slab_alloc+0xb4/0x5a0
[c00000000019ac98] .kmem_cache_alloc_node_trace+0xa8/0x260
[c00000000007eb70] .build_sched_domains+0xa60/0xb90
[c000000000a16a98] .sched_init_smp+0xa8/0x228
[c000000000a00274] .kernel_init+0x10c/0x1fc
[c00000000002324c] .kernel_thread+0x54/0x70
I'm guessing it's a result of some nodes not having any local memory.
but a bit surprised I'm not seeing it elsewhere.
Investigating.
Also, could you send me the node_distance table for that machine? I'm
curious what the interconnects look like on that thing.
From: Peter Zijlstra <hidden> Date: 2011-07-19 10:22:14
On Tue, 2011-07-19 at 14:44 +1000, Anton Blanchard wrote:
=20
Our node distances are a bit arbitrary (I make them up based on
information given to us in the device tree). In terms of memory we have
a maximum of three levels. To give some gross estimates, on chip memory
might be 30GB/sec, on node memory 10-15GB/sec and off node memory
5GB/sec.
=20
The only thing we tweak with node distances is to make sure we go into
node reclaim before going off node:
=20
/*
* Before going off node we want the VM to try and reclaim from the local
* node. It does this if the remote distance is larger than RECLAIM_DISTA=
NCE.
* With the default REMOTE_DISTANCE of 20 and the default RECLAIM_DISTANC=
E of
* 20, we never reclaim and go off node straight away.
*
* To fix this we choose a smaller value of RECLAIM_DISTANCE.
*/
#define RECLAIM_DISTANCE 10
40 40 40 40 40 40 40 20 20 20 10 0 0 0 0=20
That looks very strange indeed.. up to node 23 there is the normal
symmetric matrix with all the trace elements on 10 (as we would expect
for local access), and some 4x4 sub-matrix stacked around the trace with
20, suggesting a single hop distance, and the rest on 40 being
out-there.
But row 24-27 and column 28-31 are way weird, how can that ever be?
Aren't the inter-connects symmetric and thus mandating a fully symmetric
matrix? That is, how can traffic from node 23 (row) to node 28 (column)
have inf bandwidth (0) yet traffic from node 28 (row) to node 23
(column) have a multi-hop distance of 40.
So the idea I had to generate numa sched domains from the node distance
( http://marc.info/?l=3Dlinux-kernel&m=3D130218515520540 ), would that stil=
l
work for you? [it does assume a symmetric matrix ]
From: Anton Blanchard <hidden> Date: 2011-07-20 02:03:34
Hi,
That looks very strange indeed.. up to node 23 there is the normal
symmetric matrix with all the trace elements on 10 (as we would expect
for local access), and some 4x4 sub-matrix stacked around the trace
with 20, suggesting a single hop distance, and the rest on 40 being
out-there.
But row 24-27 and column 28-31 are way weird, how can that ever be?
Aren't the inter-connects symmetric and thus mandating a fully
symmetric matrix? That is, how can traffic from node 23 (row) to node
28 (column) have inf bandwidth (0) yet traffic from node 28 (row) to
node 23 (column) have a multi-hop distance of 40.
From: Anton Blanchard <hidden> Date: 2011-07-20 10:14:41
Hi Peter,
That looks very strange indeed.. up to node 23 there is the normal
symmetric matrix with all the trace elements on 10 (as we would expect
for local access), and some 4x4 sub-matrix stacked around the trace
with 20, suggesting a single hop distance, and the rest on 40 being
out-there.
I retested with the latest version of numactl, and get correct results.
I worked out why the patches don't boot, we weren't allocating any
space for the cpumask and ran off the end of the allocation.
Should we also use cpumask_copy instead of open coding it? I added that
too.
Anton
Index: linux-2.6/kernel/sched.c
===================================================================
From: Peter Zijlstra <hidden> Date: 2011-07-20 10:45:48
On Wed, 2011-07-20 at 20:14 +1000, Anton Blanchard wrote:
quoted
That looks very strange indeed.. up to node 23 there is the normal
symmetric matrix with all the trace elements on 10 (as we would expect
for local access), and some 4x4 sub-matrix stacked around the trace
with 20, suggesting a single hop distance, and the rest on 40 being
out-there.
=20
I retested with the latest version of numactl, and get correct results.
One less thing to worry about ;-)
I worked out why the patches don't boot, we weren't allocating any
space for the cpumask and ran off the end of the allocation.
Gah! that's not the first time I made that particular mistake :/
Should we also use cpumask_copy instead of open coding it? I added that
too.
Probably, I looked for cpumask_assign() and on failing to find that used
the direct assignment.
So with that fix the patch makes the machine happy again?
Thanks!
On Wed, Jul 20, 2011 at 5:14 AM, Anton Blanchard [off-list ref] wrote:
quoted
So with that fix the patch makes the machine happy again?
Yes, the machine looks fine with the patches applied. Thanks!
Ok, so what's the situation for 3.0 (I'm waiting for some RCU
resolution now)? Anton's patch may be small, but that's just the tiny
fixup patch to Peter's much scarier one ;)
Linus
From: Peter Zijlstra <hidden> Date: 2011-07-20 14:59:13
On Wed, 2011-07-20 at 07:40 -0700, Linus Torvalds wrote:
On Wed, Jul 20, 2011 at 5:14 AM, Anton Blanchard [off-list ref] wrote:
quoted
quoted
So with that fix the patch makes the machine happy again?
Yes, the machine looks fine with the patches applied. Thanks!
=20
Ok, so what's the situation for 3.0 (I'm waiting for some RCU
resolution now)? Anton's patch may be small, but that's just the tiny
fixup patch to Peter's much scarier one ;)
Right, so we can either merge my scary patches now and have 3.0 boot on
16+ node machines (and risk breaking something), or delay them until
3.0.1 and have 16+ node machines suffer a little.
The alternative quick hack is simply to disable the node domain, but
that'll be detrimental to regular machines in that the top domain used
to have NODE sd_flags will now have ALL_NODE sd_flags which are much
less aggressive.
On Wed, Jul 20, 2011 at 7:58 AM, Peter Zijlstra [off-list ref] wrote:
Right, so we can either merge my scary patches now and have 3.0 boot on
16+ node machines (and risk breaking something), or delay them until
3.0.1 and have 16+ node machines suffer a little.
So how much impact does your scary patch have on machines that don't
have multiple nodes? If it's a "the code isn't even called by normal
machines" kind of setup, I don't think I care a lot.
Linus
On Wed, Jul 20, 2011 at 7:58 AM, Peter Zijlstra [off-list ref] wrote:
quoted
Right, so we can either merge my scary patches now and have 3.0
boot on 16+ node machines (and risk breaking something), or delay
them until 3.0.1 and have 16+ node machines suffer a little.
So how much impact does your scary patch have on machines that
don't have multiple nodes? If it's a "the code isn't even called by
normal machines" kind of setup, I don't think I care a lot.
NUMA systems will trigger the new code - not just 'weird NUMA
systems' - but i still think we could try the patches, the code looks
straightforward and i booted them on NUMA systems and it all seems
fine so far.
Anyway, i'll push the new sched/urgent branch out in a few minutes
and then you'll see the full patches in the commit notifications.
Thanks,
Ingo
From: Peter Zijlstra <hidden> Date: 2011-07-20 16:43:35
On Wed, 2011-07-20 at 09:04 -0700, Linus Torvalds wrote:
On Wed, Jul 20, 2011 at 7:58 AM, Peter Zijlstra [off-list ref] =
wrote:
quoted
Right, so we can either merge my scary patches now and have 3.0 boot on
16+ node machines (and risk breaking something), or delay them until
3.0.1 and have 16+ node machines suffer a little.
=20
So how much impact does your scary patch have on machines that don't
have multiple nodes? If it's a "the code isn't even called by normal
machines" kind of setup, I don't think I care a lot.
Hmm, it does get called, but it looks relatively straight forward to
make it so that it doesn't. Let me try that.
Yes, the below works nicely (on top of the previous two).
Built and boot tested on a single-node and multi-node x86_64.
---
Subject: sched: Avoid creating superfluous domains
From: Peter Zijlstra <redacted>
Date: Wed Jul 20 18:34:30 CEST 2011
When creating sched_domains, stop when we've covered the entire target
span instead of continuing to create domains, only to later find
they're redundant and throw them away again.
This avoids single node systems from touching funny NUMA sched_domain
creation code.
Requested-by: Linus Torvalds [off-list ref]
Signed-off-by: Peter Zijlstra <redacted>
---
kernel/sched.c | 2 ++
1 file changed, 2 insertions(+)
Index: linux-2.6/kernel/sched.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D