From: Sachin Sant <hidden> Date: 2009-09-17 10:51:55
Sachin Sant wrote:
2.6.31-git5(0cb583fd2862f19ea88b02eb307d11c09e51e2f8)
kernel fails to boot on powerpc box.
31-git4(18240904960a39e582ced8ba8ececb10b8c22dd3) boots fine.
The problem seems to have been introduced with
commit ada3fa15057205b7d3f727bba5cd26b5912e350f.
Specifically this patch :
powerpc64: convert to dynamic percpu allocator
If i revert this patch i am able to boot latest git
on a powerpc box.
The kernel is configured with 64K pages.
Have again attached the config file used.
Thanks
-Sachin
The machine hangs just after following messages
Mount CIFS File Systems unused
Loading CPUFreq modules (CPUFreq not supported)
Starting HAL daemon done
NET: Registered protocol family 10
I don't get any other messages, so not sure what the problem
is.
I tried enabling CONFIG_RCU_CPU_STALL_DETECTOR and rebooted the
box to see if i could obtain debug messages. With this config
option enabled i observed the following messages in the dmesg
buffer.
<7>irq: irq 33540 on host null mapped to virtual irq 260
<6>NET: Registered protocol family 10
<3>INFO: RCU detected CPU 0 stall (t=1000 jiffies)
<3>INFO: RCU detected CPU 0 stall (t=4000 jiffies)
<3>BUG: soft lockup - CPU#0 stuck for 61s! [modprobe:1934]
<4>Modules linked in: ipv6(+) fuse loop dm_mod sr_mod cdrom sg sd_mod
crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
<4>NIP: c00000000004657c LR: c000000000189dbc CTR: 0000000000000040
<4>REGS: c0000000deff7630 TRAP: 0901 Not tainted (2.6.31-git5)
<4>MSR: 8000000000009032 <EE,ME,IR,DR> CR: 44224420 XER: 20000004
<4>TASK = c0000000dc1c6b00[1934] 'modprobe' THREAD: c0000000deff4000
CPU: 0
<4>GPR00: 0000000000000040 c0000000deff78b0 c000000000c16580
d00007fffff00000
<4>GPR04: 0000000000000000 0000000000000000 d00007fffff00000
d00007fffff00000
<4>GPR08: 0000000000000000 c000000001220180 c000000000cc5b60
00000000000003c0
<4>GPR12: 0000000088224428 c000000000ce2600
<4>NIP [c00000000004657c] .memset+0x60/0xfc
<4>LR [c000000000189dbc] .pcpu_alloc+0x724/0x910
<4>Call Trace:
<4>[c0000000deff78b0] [c000000000189d4c] .pcpu_alloc+0x6b4/0x910
(unreliable)
<4>[c0000000deff79e0] [c00000000018a03c] .__alloc_percpu+0x3c/0x58
<4>[c0000000deff7a70] [c000000000684130] .snmp_mib_init+0x64/0xb0
<4>[c0000000deff7b00] [d0000000023eeef0] .ipv6_add_dev+0x1e4/0x3fc [ipv6]
<4>[c0000000deff7bb0] [d00000000241e020] .addrconf_init+0x74/0x1a0 [ipv6]
<4>[c0000000deff7c50] [d00000000241dddc] .inet6_init+0x1c4/0x354 [ipv6]
<4>[c0000000deff7ce0] [c000000000009970] .do_one_initcall+0xb0/0x208
<4>[c0000000deff7d90] [c0000000000e4938] .SyS_init_module+0x12c/0x2ac
<4>[c0000000deff7e30] [c0000000000085dc] syscall_exit+0x0/0x40
<4>Instruction dump:
<4>98860000 38c60001 409e000c b0860000 38c60002 409d000c 90860000
38c60004
<4>78a0d183 78a506a0 7c0903a6 4182002c <f8860000> f8860008 f8860010
f8860018
<3>INFO: RCU detected CPU 0 stall (t=7000 jiffies)
<3>INFO: RCU detected CPU 0 stall (t=10000 jiffies)
<3>INFO: RCU detected CPU 0 stall (t=13000 jiffies)
<3>INFO: RCU detected CPU 0 stall (t=16000 jiffies)
I have attached the complete dmesg.log and .config used.
Thanks
-Sachin
------------------------------------------------------------------------
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-17 11:18:25
On Thu, 2009-09-17 at 16:21 +0530, Sachin Sant wrote:
The problem seems to have been introduced with
commit ada3fa15057205b7d3f727bba5cd26b5912e350f.
Specifically this patch :
powerpc64: convert to dynamic percpu allocator
If i revert this patch i am able to boot latest git
on a powerpc box.
The kernel is configured with 64K pages.
Have again attached the config file used.
Thanks for tracking the commit down. I've tried it today but your config
has everything as modules and that didn't work out very well with my
netboot setup. I'll dig a bit more tomorrow.
It's probably my fault for not properly getting the time to test Tejun
patches... oh well, that's what being too busy does to you.
Cheers,
Ben.
On Thu, 2009-09-17 at 16:21 +0530, Sachin Sant wrote:
quoted
The problem seems to have been introduced with
commit ada3fa15057205b7d3f727bba5cd26b5912e350f.
Specifically this patch :
powerpc64: convert to dynamic percpu allocator
If i revert this patch i am able to boot latest git
on a powerpc box.
The kernel is configured with 64K pages.
Have again attached the config file used.
Thanks for tracking the commit down. I've tried it today but your config
has everything as modules and that didn't work out very well with my
netboot setup. I'll dig a bit more tomorrow.
It's probably my fault for not properly getting the time to test Tejun
patches... oh well, that's what being too busy does to you.
Ah... sorry about that. Sachin, is it possible for you to build the
kernel with debug info and ask gdb where the stalling NIP is in the c
file?
Thanks.
--
tejun
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
Latest git (2.6.31-git9:78f28b7c555359c67c2a0d23f7436e915329421e)
still has this bug.
One workaround i have found for this problem is to disable IPv6.
With IPv6 disabled the machine boots OK. Till a reliable solution
is available for this issue, i will keep IPv6 disabled in my configs.
Thanks
-Sachin
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
Latest git (2.6.31-git9:78f28b7c555359c67c2a0d23f7436e915329421e)
still has this bug.
One workaround i have found for this problem is to disable IPv6.
With IPv6 disabled the machine boots OK. Till a reliable solution
is available for this issue, i will keep IPv6 disabled in my configs.
I'm think it's most likely caused by some code accessing invalid
percpu address. I'm currently writing up access validator. Should be
done in several hours. So, ipv6 it is. I couldn't reproduce your
problem here. I'll give ipv6 a shot.
Thanks.
--
tejun
One workaround i have found for this problem is to disable IPv6.
With IPv6 disabled the machine boots OK. Till a reliable solution
is available for this issue, i will keep IPv6 disabled in my configs.
I'm think it's most likely caused by some code accessing invalid
percpu address. I'm currently writing up access validator. Should be
done in several hours. So, ipv6 it is. I couldn't reproduce your
problem here. I'll give ipv6 a shot.
Can you please apply the attached patch and see whether anything
interesting shows up in the kernel log?
Thanks.
--
tejun
From: Sachin Sant <hidden> Date: 2009-09-24 07:58:28
Tejun Heo wrote:
Can you please apply the attached patch and see whether anything
interesting shows up in the kernel log?
Thanks Tejun for the debug patch. Attached here are the relevant logs.
The only messages related to percpu in the logs are
<6>PERCPU: Embedded 2 pages/cpu @c000000001200000 s100232 r0 d30840 u524288
<7>pcpu-alloc: s100232 r0 d30840 u524288 alloc=1*1048576
<7>pcpu-alloc: [0] 0 1
The captured logs are with latest git.
Thanks
-Sachin
Can you please apply the attached patch and see whether anything
interesting shows up in the kernel log?
Thanks Tejun for the debug patch. Attached here are the relevant logs.
The only messages related to percpu in the logs are
<6>PERCPU: Embedded 2 pages/cpu @c000000001200000 s100232 r0 d30840 u524288
<7>pcpu-alloc: s100232 r0 d30840 u524288 alloc=1*1048576
<7>pcpu-alloc: [0] 0 1
The captured logs are with latest git.
Hmm... that means it wasn't caused by rogue percpu pointer access.
Pleast wait a bit. I'll try to reproduce it.
Thanks.
--
tejun
From: Sachin Sant <hidden> Date: 2009-09-24 13:23:14
Tejun Heo wrote:
Sachin Sant wrote:
quoted
Tejun Heo wrote:
quoted
Can you please apply the attached patch and see whether anything
interesting shows up in the kernel log?
Thanks Tejun for the debug patch. Attached here are the relevant logs.
The only messages related to percpu in the logs are
<6>PERCPU: Embedded 2 pages/cpu @c000000001200000 s100232 r0 d30840 u524288
<7>pcpu-alloc: s100232 r0 d30840 u524288 alloc=1*1048576
<7>pcpu-alloc: [0] 0 1
The captured logs are with latest git.
Hmm... that means it wasn't caused by rogue percpu pointer access.
Pleast wait a bit. I'll try to reproduce it.
I was able to reproduce the hang in a different way. (I still had
IPV6 disabled in my config). I executed the network namespace container
tests from LTP and could reproduce a similar hang. The top three
function calls were the same as with IPV6. Here are the traces
using xmon debugger.
Oops: System Reset, sig: 6 [#4]
SMP NR_CPUS=1024 DEBUG_PAGEALLOC NUMA pSeries
Modules linked in: quota_v2 quota_tree fuse loop dm_mod sg sd_mod crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
NIP: c00000000003c310 LR: c0000000000055d0 CTR: 0000000000000040
REGS: c0000000fc90f340 TRAP: 0100 Tainted: G D (2.6.31-git13-autotest)
MSR: 8000000000081032 <ME,IR,DR> CR: 28004420 XER: 20000001
TASK = c00000002c408890[8753] 'check_netns_ena' THREAD: c0000000fc90c000 CPU: 2
GPR00: 00000fffffffffff c0000000fc90f5c0 c000000000b8c2a8 d00007fffff00000
GPR04: 0000000000000201 0000000000000300 d00007fffff00000 d00007fffff00000
GPR08: 0000000000000000 000007fffff00000 0000000000000000 0000000000000000
GPR12: 8000000000009032 c000000000c82a00 0000000000000001 c0000000fc90f924
GPR16: 0000000000000300 0000000000000001 c0000000fa8e2380 0000000000000000
GPR20: 0000000000010000 0000000000000001 0000000000000000 0000000000000000
GPR24: c0000000fa9c09c8 0000000000000001 0000000000000001 c0000000faef6f60
GPR28: c000000000c6b620 0000000000000000 c000000000af2aa0 c000000000c6d1b0
NIP [c00000000003c310] .hash_page+0x24/0x4bc
LR [c0000000000055d0] .do_hash_page+0x50/0x6c
Call Trace:
[c0000000fc90f5c0] [c0000000000055d0] .do_hash_page+0x50/0x6c (unreliable)
--- Exception: 301 at .memset+0x60/0xfc
LR = .pcpu_alloc+0x718/0x8fc
[c0000000fc90f8b0] [c0000000001700dc] .pcpu_alloc+0x6a8/0x8fc (unreliable)
[c0000000fc90f9d0] [c000000000614648] .snmp_mib_init+0x54/0x9c
[c0000000fc90fa60] [c000000000614764] .ipv4_mib_init_net+0xd4/0x1e0
[c0000000fc90fb10] [c0000000005a839c] .setup_net+0x68/0x124
[c0000000fc90fbb0] [c0000000005a8ad0] .copy_net_ns+0x88/0x130
[c0000000fc90fc40] [c0000000000bd5ac] .create_new_namespaces+0x110/0x1d0
[c0000000fc90fce0] [c0000000000bd874] .unshare_nsproxy_namespaces+0x6c/0xe8
[c0000000fc90fd80] [c000000000091ee8] .SyS_unshare+0x13c/0x318
[c0000000fc90fe30] [c0000000000085b4] syscall_exit+0x0/0x40
Instruction dump:
7c0803a6 ebe1fff8 4e800020 78690100 7c0802a6 f8010010 3800ffff fa01ff80
7cb02b78 78000500 fa21ff88 fb61ffd8 <7c912378> fa41ff90 7c7b1b78 fa61ff98
As you can see the call trace is same as far as top three function calls
are concerned [snmp_mib_init(), pcpu_alloc() and memset()].
The snmp_mib_init() function is :
int snmp_mib_init(void *ptr[2], size_t mibsize)
{
BUG_ON(ptr == NULL);
ptr[0] = __alloc_percpu(mibsize, __alignof__(unsigned long long));
if (!ptr[0])
goto err0;
ptr[1] = __alloc_percpu(mibsize, __alignof__(unsigned long long));
if (!ptr[1])
goto err1;
return 0;
.....
May be this might help..
Thanks
-Sachin
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-24 21:06:18
On Thu, 2009-09-24 at 18:53 +0530, Sachin Sant wrote:
quoted hunk
Tejun Heo wrote:
quoted
Sachin Sant wrote:
quoted
Tejun Heo wrote:
quoted
Can you please apply the attached patch and see whether anything
interesting shows up in the kernel log?
Thanks Tejun for the debug patch. Attached here are the relevant logs.
The only messages related to percpu in the logs are
<6>PERCPU: Embedded 2 pages/cpu @c000000001200000 s100232 r0 d30840 u524288
<7>pcpu-alloc: s100232 r0 d30840 u524288 alloc=1*1048576
<7>pcpu-alloc: [0] 0 1
The captured logs are with latest git.
Hmm... that means it wasn't caused by rogue percpu pointer access.
Pleast wait a bit. I'll try to reproduce it.
I was able to reproduce the hang in a different way. (I still had
IPV6 disabled in my config). I executed the network namespace container
tests from LTP and could reproduce a similar hang. The top three
function calls were the same as with IPV6. Here are the traces
using xmon debugger.
Oops: System Reset, sig: 6 [#4]
SMP NR_CPUS=1024 DEBUG_PAGEALLOC NUMA pSeries
Modules linked in: quota_v2 quota_tree fuse loop dm_mod sg sd_mod crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
NIP: c00000000003c310 LR: c0000000000055d0 CTR: 0000000000000040
REGS: c0000000fc90f340 TRAP: 0100 Tainted: G D (2.6.31-git13-autotest)
MSR: 8000000000081032 <ME,IR,DR> CR: 28004420 XER: 200 00001
TASK = c00000002c408890[8753] 'check_netns_ena' THREAD: c0000000fc90c000 CPU: 2
GPR00: 00000fffffffffff c0000000fc90f5c0 c000000000b8c2a8 d00007fffff00000
GPR04: 0000000000000201 0000000000000300 d00007fffff00000 d00007fffff00000
GPR08: 0000000000000000 000007fffff00000 0000000000000000 0000000000000000
GPR12: 8000000000009032 c000000000c82a00 0000000000000001 c0000000fc90f924
GPR16: 0000000000000300 0000000000000001 c0000000fa8e2380 0000000000000000
GPR20: 0000000000010000 0000000000000001 0000000000000000 0000000000000000
GPR24: c0000000fa9c09c8 0000000000000001 0000000000000001 c0000000faef6f60
GPR28: c000000000c6b620 0000000000000000 c000000000af2aa0 c000000000c6d1b0
NIP [c00000000003c310] .hash_page+0x24/0x4bc
LR [c0000000000055d0] .do_hash_page+0x50/0x6c
Call Trace:
[c0000000fc90f5c0] [c0000000000055d0] .do_hash_page+0x50/0x6c (unreliable)
--- Exception: 301 at .memset+0x60/0xfc
LR = .pcpu_alloc+0x718/0x8fc
So it's memsetting something that causes it to hash_page(), ie, faulting
in pages (vmalloc space ?) so far nothing obviously wrong....
[c0000000fc90f8b0] [c0000000001700dc] .pcpu_alloc+0x6a8/0x8fc (unreliable)
[c0000000fc90f9d0] [c000000000614648] .snmp_mib_init+0x54/0x9c
[c0000000fc90fa60] [c000000000614764] .ipv4_mib_init_net+0xd4/0x1e0
[c0000000fc90fb10] [c0000000005a839c] .setup_net+0x68/0x124
[c0000000fc90fbb0] [c0000000005a8ad0] .copy_net_ns+0x88/0x130
[c0000000fc90fc40] [c0000000000bd5ac] .create_new_namespaces+0x110/0x1d0
[c0000000fc90fce0] [c0000000000bd874] .unshare_nsproxy_namespaces+0x6c/0xe8
[c0000000fc90fd80] [c000000000091ee8] .SyS_unshare+0x13c/0x318
[c0000000fc90fe30] [c0000000000085b4] syscall_exit+0x0/0x40
Instruction dump:
7c0803a6 ebe1fff8 4e800020 78690100 7c0802a6 f8010010 3800ffff fa01ff80
7cb02b78 78000500 fa21ff88 fb61ffd8 <7c912378> fa41ff90 7c7b1b78 fa61ff98
As you can see the call trace is same as far as top three function calls
are concerned [snmp_mib_init(), pcpu_alloc() and memset()].
The snmp_mib_init() function is :
int snmp_mib_init(void *ptr[2], size_t mibsize)
{
BUG_ON(ptr == NULL);
ptr[0] = __alloc_percpu(mibsize, __alignof__(unsigned long long));
if (!ptr[0])
goto err0;
ptr[1] = __alloc_percpu(mibsize, __alignof__(unsigned long long));
if (!ptr[1])
goto err1;
return 0;
.....
May be this might help..
Thanks
-Sachin
--- Exception: 301 at .memset+0x60/0xfc
LR = .pcpu_alloc+0x718/0x8fc
So it's memsetting something that causes it to hash_page(), ie, faulting
in pages (vmalloc space ?) so far nothing obviously wrong....
It's probably memset() call near the end of pcpu_populate_chunk()
where percpu allocator clears the allocated areas before returning to
user. I don't think the first chunk is causing the problem as they're
all in the linear mapped area. From the second chunk on, they're on
vmalloc area and very near to the top of it, so that might be exposing
a hidden problem in paging code? BTW, for some reason, the problem is
not reproducible on my powerstation.
Sachin, can you please apply the attached patch on top of the current
linus tree, reproduce the hang and report full kernel log? Let's see
which address is causing the problem.
Thanks.
--
tejun
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-25 03:41:15
On Fri, 2009-09-25 at 12:22 +0900, Tejun Heo wrote:
Benjamin Herrenschmidt wrote:
quoted
quoted
--- Exception: 301 at .memset+0x60/0xfc
LR = .pcpu_alloc+0x718/0x8fc
So it's memsetting something that causes it to hash_page(), ie, faulting
in pages (vmalloc space ?) so far nothing obviously wrong....
It's probably memset() call near the end of pcpu_populate_chunk()
where percpu allocator clears the allocated areas before returning to
user. I don't think the first chunk is causing the problem as they're
all in the linear mapped area. From the second chunk on, they're on
vmalloc area and very near to the top of it, so that might be exposing
a hidden problem in paging code? BTW, for some reason, the problem is
not reproducible on my powerstation.
That's indeed a possibility, though it would be strange...
Definitely worth looking at your logs, and I'll check with Sachin
about getting on the machine after that if I need to dig more.
Cheers,
Ben.
From: Sachin Sant <hidden> Date: 2009-09-25 07:15:55
Tejun Heo wrote:
Benjamin Herrenschmidt wrote:
quoted
quoted
--- Exception: 301 at .memset+0x60/0xfc
LR = .pcpu_alloc+0x718/0x8fc
So it's memsetting something that causes it to hash_page(), ie, faulting
in pages (vmalloc space ?) so far nothing obviously wrong....
It's probably memset() call near the end of pcpu_populate_chunk()
where percpu allocator clears the allocated areas before returning to
user. I don't think the first chunk is causing the problem as they're
all in the linear mapped area. From the second chunk on, they're on
vmalloc area and very near to the top of it, so that might be exposing
a hidden problem in paging code? BTW, for some reason, the problem is
not reproducible on my powerstation.
Sachin, can you please apply the attached patch on top of the current
linus tree, reproduce the hang and report full kernel log? Let's see
which address is causing the problem.
Here is the dmesg log captured with the debug patch.
Some of the debug messages related to PERCPU
<6>PERCPU: Embedded 2 pages/cpu @c000000001100000 s97160 r0 d33912 u524288
<6>pcpu-alloc: s97160 r0 d33912 u524288 alloc=1*1048576
<6>pcpu-alloc: [0] 0 1
<4>PERCPU: initialized 19 slots [c000000001120200,c000000001120330)
<4>PERCPU: chunk 0 relocating -1 -> 13 c000000001120380 <c000000001120380:c000000001120380>
<4>PERCPU: relocated <c0000000011202d0:c0000000011202d0>
<4>PERCPU: chunk 0 relocating 13 -> 12 c000000001120380 <c0000000011202d0:c0000000011202d0>
<4>PERCPU: relocated <c0000000011202c0:c0000000011202c0>
<4>PERCPU: chunk 0 relocating 12 -> 11 c000000001120380 <c0000000011202c0:c0000000011202c0>
<4>PERCPU: relocated <c0000000011202b0:c0000000011202b0>
<6>ehea: eth0: Physical port up
<6>ehea: External switch port is backup port
<7>irq: irq 33540 on host null mapped to virtual irq 260
<6>NET: Registered protocol family 10
<4>PERCPU: chunk 0 relocating 11 -> 10 c000000001120380 <c0000000011202b0:c0000000011202b0>
<4>PERCPU: relocated <c0000000011202a0:c0000000011202a0>
<4>PERCPU: chunk 0 relocating 10 -> 9 c000000001120380 <c0000000011202a0:c0000000011202a0>
<4>PERCPU: relocated <c000000001120290:c000000001120290>
<4>PERCPU: chunk 1 relocating -1 -> 18 c0000000db70fb00 <c0000000db70fb00:c0000000db70fb00>
<4>PERCPU: relocated <c000000001120320:c000000001120320>
<4>PERCPU: chunk 1 relocating 18 -> 16 c0000000db70fb00 <c000000001120320:c000000001120320>
<4>PERCPU: relocated <c000000001120300:c000000001120300>
<4>PERCPU: chunk 1, alloc pages [0,1)
<4>PERCPU: chunk 1, map pages [0,1)
<4>PERCPU: map 0xd00007fffff00000, 1 pages 53544
<4>PERCPU: map 0xd00007fffff80000, 1 pages 53545
<4>PERCPU: chunk 1, will clear 4096b/unit d00007fffff00000 d00007fffff80000
<3>INFO: RCU detected CPU 0 stall (t=1000 jiffies)
Thanks
-Sachin
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
This supports my hypothesis. This is the first area being allocated
from a dynamic chunk and cleared. PFN 53544 and 53545 have been
allocated and successfully mapped to 0xd00007fffff00000 and
0xd00007fffff80000 using map_kernel_range_noflush() but when those
addresses are actually accessed, we end up with infinite faults. The
fault handler probably thinks that the fault has been handled
correctly but, when the control is returned, the processor faults
again. Benjamin, I'm way out of my depth here, can you please help?
Oh, one more simple experiment. Sachin, does the following patch make
any difference?
This supports my hypothesis. This is the first area being allocated
from a dynamic chunk and cleared. PFN 53544 and 53545 have been
allocated and successfully mapped to 0xd00007fffff00000 and
0xd00007fffff80000 using map_kernel_range_noflush() but when those
addresses are actually accessed, we end up with infinite faults. The
fault handler probably thinks that the fault has been handled
correctly but, when the control is returned, the processor faults
again. Benjamin, I'm way out of my depth here, can you please help?
Oh, one more simple experiment. Sachin, does the following patch make
any difference?
This supports my hypothesis. This is the first area being allocated
from a dynamic chunk and cleared. PFN 53544 and 53545 have been
allocated and successfully mapped to 0xd00007fffff00000 and
0xd00007fffff80000 using map_kernel_range_noflush() but when those
addresses are actually accessed, we end up with infinite faults. The
fault handler probably thinks that the fault has been handled
correctly but, when the control is returned, the processor faults
again. Benjamin, I'm way out of my depth here, can you please help?
Oh, one more simple experiment. Sachin, does the following patch make
any difference?
With this patch applied the machine boots OK :-)
Have attached the boot log. Note that this boot log is
from a different machine, but the reported problem can be
recreate on this machine as well.
Thanks
-Sachin
@@ -2056,7 +2056,8 @@ static unsigned long pvm_determine_end(struct vmap_area **pnext,structvmap_area**pprev,unsignedlongalign){-constunsignedlongvmalloc_end=VMALLOC_END&~(align-1);+constunsignedlongvmalloc_start=ALIGN(VMALLOC_START,align);+constunsignedlongvmalloc_end=vmalloc_start+(512<<20);unsignedlongaddr;if(*pnext)
@@ -2102,7 +2103,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,size_talign,gfp_tgfp_mask){constunsignedlongvmalloc_start=ALIGN(VMALLOC_START,align);-constunsignedlongvmalloc_end=VMALLOC_END&~(align-1);+constunsignedlongvmalloc_end=vmalloc_start+(512<<20);structvmap_area**vas,*prev,*next;structvm_struct**vms;intarea,area2,last_area,term_area;
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
This supports my hypothesis. This is the first area being allocated
from a dynamic chunk and cleared. PFN 53544 and 53545 have been
allocated and successfully mapped to 0xd00007fffff00000 and
0xd00007fffff80000 using map_kernel_range_noflush() but when those
addresses are actually accessed, we end up with infinite faults. The
fault handler probably thinks that the fault has been handled
correctly but, when the control is returned, the processor faults
again. Benjamin, I'm way out of my depth here, can you please help?
Definitely looks like a powerpc mm problem. I'll have a look on monday.
Cheers,
Ben.
quoted hunk
Oh, one more simple experiment. Sachin, does the following patch make
any difference?
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-25 09:49:38
On Fri, 2009-09-25 at 18:01 +0900, Tejun Heo wrote:
quoted
With this patch applied the machine boots OK :-)
Ah... so, the problem really is too high address. If you've got some
time, it might be interesting to find out how far high is safe.
Might give me a clue about what the problem is but I think I'll just
cook up a test case that forcibly vmap something high up and see how it
goes from there. It could be a very old bug that nobody ever noticed
because our vmalloc space on 64-bit is so huge :-)
Cheers,
Ben.
From: Sachin Sant <hidden> Date: 2009-10-05 06:54:58
Benjamin Herrenschmidt wrote:
On Fri, 2009-09-25 at 18:01 +0900, Tejun Heo wrote:
quoted
quoted
With this patch applied the machine boots OK :-)
Ah... so, the problem really is too high address. If you've got some
time, it might be interesting to find out how far high is safe.
Might give me a clue about what the problem is but I think I'll just
cook up a test case that forcibly vmap something high up and see how it
goes from there. It could be a very old bug that nobody ever noticed
because our vmalloc space on 64-bit is so huge :-)
I still have this problem with 2.6.32-rc3.
Here is the relevant information
0:mon> t
[link register ] c0000000001a7f78 .pcpu_alloc+0x798/0xa04
[c0000000033e37f0] c0000000001a7f08 .pcpu_alloc+0x728/0xa04 (unreliable)
[c0000000033e3920] c0000000001a8278 .__alloc_percpu+0x3c/0x58
[c0000000033e39b0] c0000000005d1ad0 .snmp_mib_init+0x64/0xb0
[c0000000033e3a40] c0000000005d1c00 .ipv4_mib_init_net+0xe4/0x1f8
[c0000000033e3b00] c00000000055b608 .setup_net+0x78/0x138
[c0000000033e3ba0] c00000000055be38 .copy_net_ns+0x9c/0x148
[c0000000033e3c30] c0000000000d06d8 .create_new_namespaces+0x120/0x1e4
[c0000000033e3ce0] c0000000000d09e0 .unshare_nsproxy_namespaces+0x7c/0xfc
[c0000000033e3d80] c00000000009dd74 .SyS_unshare+0x148/0x33c
[c0000000033e3e30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff8b0ab978
SP (fffe633fe30) is in userspace
0:mon> e
cpu 0x0: Vector: 501 (Hardware Interrupt) at [c0000000033e3570]
pc: c00000000004bdc0: .memset+0x60/0xfc
lr: c0000000001a7f78: .pcpu_alloc+0x798/0xa04
sp: c0000000033e37f0
msr: 8000000000009032
current = 0xc000000003270860
paca = 0xc0000000010c2600
pid = 3442, comm = two_children_ns
0:mon> r
R00 = 0000000000000040 R07 = d00007fffff00000
R01 = c0000000033e37f0 R08 = 0000000000000000
R02 = c000000000fe7c78 R09 = c000000001700180
R03 = d00007fffff00000 R10 = c000000001095aa0
R04 = 0000000000000000 R11 = 00000000000003c0
R05 = 0000000000000000 R12 = 0000000048004428
R06 = d00007fffff00000 R13 = c0000000010c2600
pc = c00000000004bdc0 .memset+0x60/0xfc
lr = c0000000001a7f78 .pcpu_alloc+0x798/0xa04
msr = 8000000000009032 cr = 44004420
ctr = 0000000000000040 xer = 0000000020000020 trap = 501
0:mon> di $.memset
c00000000004bd60 7c0300d0 neg r0,r3
c00000000004bd64 5084442e rlwimi r4,r4,8,16,23
c00000000004bd68 70000007 andi. r0,r0,7
c00000000004bd6c 5084801e rlwimi r4,r4,16,0,15
c00000000004bd70 7c850040 cmplw cr1,r5,r0
c00000000004bd74 7884000e rldimi r4,r4,32,0
c00000000004bd78 7c101120 mtocrf 1,r0
c00000000004bd7c 7c661b78 mr r6,r3
c00000000004bd80 418400ac blt cr1,c00000000004be2c # .memset+0xcc/0xfc
c00000000004bd84 41e2002c beq+ c00000000004bdb0 # .memset+0x50/0xfc
c00000000004bd88 7ca02850 subf r5,r0,r5
c00000000004bd8c 409f000c bns cr7,c00000000004bd98 # .memset+0x38/0xfc
c00000000004bd90 98860000 stb r4,0(r6)
c00000000004bd94 38c60001 addi r6,r6,1
c00000000004bd98 409e000c bne cr7,c00000000004bda4 # .memset+0x44/0xfc
c00000000004bd9c b0860000 sth r4,0(r6)
0:mon>
c00000000004bda0 38c60002 addi r6,r6,2
c00000000004bda4 409d000c ble cr7,c00000000004bdb0 # .memset+0x50/0xfc
c00000000004bda8 90860000 stw r4,0(r6)
c00000000004bdac 38c60004 addi r6,r6,4
c00000000004bdb0 78a0d183 rldicl. r0,r5,58,6
c00000000004bdb4 78a506a0 clrldi r5,r5,58
c00000000004bdb8 7c0903a6 mtctr r0
c00000000004bdbc 4182002c beq c00000000004bde8 # .memset+0x88/0xfc
c00000000004bdc0 f8860000 std r4,0(r6)
At this point R06 contains d00007fffff00000.
Have attached the xmon log.
Thanks
-Sachin
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------