Hi
I get the following kernel core while a user program I have is dumping
core.
Any DIeas at what to look for? (this is runnign 2.6.24, arch/powerpc on
a 8280)
When runnign the program on 2.6.18 arch/ppc, the program gets a sig 11
and dumps core.
On 2.6.24, I ghet the kernel oops, and then the program hangs sround
forever and is unkillable.
Unable to handle kernel paging request for data at address 0x48024000
Faulting instruction address: 0xc000ef88
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT Innovative Systems ApMax
Modules linked in: drv_wd(P) drv_scc devcom drv_pcir tipc drv_ss7
drv_auxcpu drv_leds(P) drv_ethsw proc_sysinfo(P) i2c_8266(P)
NIP: c000ef88 LR: c0012180 CTR: 00000080
REGS: eebc9b70 TRAP: 0300 Tainted: P (2.6.24)
MSR: 00009032 <EE,ME,IR,DR> CR: 24004442 XER: 00000000
DAR: 48024000, DSISR: 20000000
TASK =3D eebac3c0[3131] 'armd' THREAD: eebc8000
GPR00: ee9b7d00 eebc9c20 eebac3c0 48024000 00000080 399a4181 48024000
00000000
GPR08: 399a4181 ee9b7d00 00000000 c2000000 44004422 10100f38 ee82fc00
bfffffff
GPR16: ef377060 00000030 ee9b7d00 00000000 eebc9cdc 00000011 eebc9cd8
eeb96480
GPR24: ee9b7d00 399a4181 48024000 eeb9a370 eeb9a370 399a4181 48024000
c2733480
NIP [c000ef88] __flush_dcache_icache+0x14/0x40
LR [c0012180] update_mmu_cache+0x74/0x114
Call Trace:
[eebc9c20] [eebc8000] 0xeebc8000 (unreliable)
[eebc9c40] [c005d060] handle_mm_fault+0x630/0xbc0
[eebc9c80] [c005d9e4] get_user_pages+0x3f4/0x4fc
[eebc9cd0] [c00aa7c4] elf_core_dump+0x9a4/0xc5c
[eebc9d60] [c00779e4] do_coredump+0x6e0/0x748
[eebc9e50] [c002a5b0] get_signal_to_deliver+0x40c/0x45c
[eebc9e80] [c0008ce8] do_signal+0x50/0x294
[eebc9f40] [c000fb98] do_user_signal+0x74/0xc4
--- Exception: 300 at 0x10044efc
LR =3D 0x10044ec0
Yes. I have a binary that will do this every time it is started (on this
particular system),=20
only takes about 10 seconds before it dumps.
I was going to test HEAD of powerpc.git to see if it is still there.
I cannot test any earlier versions as our board port was done on 2.6.24.
Our older kernel port is 2.6.18 on arch/ppc, and it works just fine.
One potential clue:
Unable to handle kernel paging request for data at address 0x48024000
this adddress is beyond our physical memory. We have 1GB of mem=20
(CONFIG_HIGH_MEM enabled) so 0x3fff_ffff is the last valid address.
0x4000_0000 to 0x7fff_ffff are unused, 0x8000_0000 to 0x9fff_ffff is
used by PCI.
Yes. I have a binary that will do this every time it is started (on
this
particular system),
only takes about 10 seconds before it dumps.
I was going to test HEAD of powerpc.git to see if it is still there.
I cannot test any earlier versions as our board port was done on
2.6.24.
Our older kernel port is 2.6.18 on arch/ppc, and it works just fine.
One potential clue:
quoted
Unable to handle kernel paging request for data at address 0x48024000
this adddress is beyond our physical memory. We have 1GB of mem
(CONFIG_HIGH_MEM enabled) so 0x3fff_ffff is the last valid address.
0x4000_0000 to 0x7fff_ffff are unused, 0x8000_0000 to 0x9fff_ffff is
used by PCI.
Can you git-bisect to narrow this down further.
- k
Not easilly, as the board port to arch/powerpc was done on 2.6.24-rc7
and up.
Is there an somewhat esy way in git to apply the differences from master
branch to our board branch to a branch created by bisect?
And I don't even know where this started to happen.
Would trying arch/ppc help any? I have our arch/ppc port in a
semiworking state for kernels up to 2.6.23
Not easilly, as the board port to arch/powerpc was done on 2.6.24-rc7
and up.
Is there an somewhat esy way in git to apply the differences from master
branch to our board branch to a branch created by bisect?
And I don't even know where this started to happen.
Would trying arch/ppc help any? I have our arch/ppc port in a
semiworking state for kernels up to 2.6.23
Well, we know this happens on other 32-bit powerpc machines (pmac at
least)... perhaps someone could arrange to bisect on a machine that
works with older powerpc kernels (assuming they have a good repro
case).
From: Scott Wood <hidden> Date: 2008-01-31 20:16:46
On Thu, Jan 31, 2008 at 11:40:04AM -0600, Rune Torgersen wrote:
Unable to handle kernel paging request for data at address 0x48024000
Faulting instruction address: 0xc000f0a0
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT Innovative Systems ApMax
Hmm, this doesn't look like a valid DSISR, so I'm guessing this was a TLB
miss that got redirected to DataAccess (or is there something that causes
DSRISR[2] to be set on 8280? I didn't see anything in the manual...).
However, SRR1 in that case seems to indicate a store, which dcbst shouldn't
generate (except on 8xx, according to the comment in update_mmu_cache).
Do you have a simple test case that we could try to reproduce? I tried a
simple core dump on an 8280, and it worked.
Failing that, I'd add code to the page fault handler to dump what is (or
isn't) supposed to be mapped at the faulting address, and something to track
which (if any) TLB miss exception it came through.
-Scott
Do you have a simple test case that we could try to
reproduce? I tried a
simple core dump on an 8280, and it worked.
I do not have a testcase, except a app for our board that does this
reliably after about 10 seconds.
Failing that, I'd add code to the page fault handler to dump what is
(or isn't) supposed to be mapped at the faulting address, and
something to track which (if any) TLB miss exception it came through.
On Thu, Jan 31, 2008 at 11:40:04AM -0600, Rune Torgersen wrote:
quoted
Unable to handle kernel paging request for data at address 0x48024000
Faulting instruction address: 0xc000f0a0
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT Innovative Systems ApMax
I doubt the modules are the problem; there was a practically identical
report from someone with an untainted 2.6.24-rc kernel a few weeks ago
(see my first reply to Rune).
Hmm, this doesn't look like a valid DSISR, so I'm guessing this was a TLB
miss that got redirected to DataAccess (or is there something that causes
DSRISR[2] to be set on 8280? I didn't see anything in the manual...).
However, SRR1 in that case seems to indicate a store, which dcbst shouldn't
generate (except on 8xx, according to the comment in update_mmu_cache).
Do you have a simple test case that we could try to reproduce? I tried a
simple core dump on an 8280, and it worked.
Is the crashing program multithreaded? The first report had firefox
triggering the oops.
From: Scott Wood <hidden> Date: 2008-01-31 20:55:15
Nathan Lynch wrote:
I doubt the modules are the problem; there was a practically identical
report from someone with an untainted 2.6.24-rc kernel a few weeks ago
(see my first reply to Rune).
I didn't think they were; I was just trying to eliminate the low hanging
fruit and get a simpler testcase. :-)
quoted
Do you have a simple test case that we could try to reproduce? I tried a
simple core dump on an 8280, and it worked.
Is the crashing program multithreaded? The first report had firefox
triggering the oops.
OK, I've got a test program that triggers it now. I'll see if I can
figure out what's going on.
-Scott
From: Scott Wood <hidden> Date: 2008-01-31 21:58:47
Scott Wood wrote:
Nathan Lynch wrote:
quoted
Is the crashing program multithreaded? The first report had firefox
triggering the oops.
OK, I've got a test program that triggers it now. I'll see if I can
figure out what's going on.
The problem seems to be that update_mmu_cache() is called on a guard
page with no access rights.
Changing update_mmu_cache() to always call flush_dcache_icache_page()
fixes it, though a better performing fix would probably be to add an
exception table entry for the dcbst.
-Scott
Is the crashing program multithreaded? The first report had firefox
triggering the oops.
=20
OK, I've got a test program that triggers it now. I'll see if I can
figure out what's going on.
=20
The problem seems to be that update_mmu_cache() is called on a guard
page with no access rights.=20
=20
Changing update_mmu_cache() to always call flush_dcache_icache_page()
fixes it, though a better performing fix would probably be to add an
exception table entry for the dcbst.
Hi
I get the following kernel core while a user program I have is dumping
core.
Any DIeas at what to look for? (this is runnign 2.6.24, arch/powerpc on
a 8280)
When runnign the program on 2.6.18 arch/ppc, the program gets a sig 11
and dumps core.
On 2.6.24, I ghet the kernel oops, and then the program hangs sround
forever and is unkillable.
From: Scott Wood <hidden> Date: 2008-02-01 17:39:16
On Thu, Jan 31, 2008 at 10:15:27AM -0600, Nathan Lynch wrote:
Rune Torgersen wrote:
quoted
Hi
I get the following kernel core while a user program I have is dumping
core.
Any DIeas at what to look for? (this is runnign 2.6.24, arch/powerpc on
a 8280)
When runnign the program on 2.6.18 arch/ppc, the program gets a sig 11
and dumps core.
On 2.6.24, I ghet the kernel oops, and then the program hangs sround
forever and is unkillable.
Yes, this program does it reliably:
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
void *threadfn(void *arg)
{
fprintf(stderr, "threadfn\n");
fflush(stderr);
sleep(1);
*(char *)0=0;
return NULL;
}
int main(void)
{
pthread_t thread[4];
int i;
for (i = 0; i < 4; i++)
pthread_create(&thread[0], NULL, threadfn, NULL);
for (;;);
}
From: Clemens Koller <hidden> Date: 2008-02-02 12:05:16
Scott Wood schrieb:
On Thu, Jan 31, 2008 at 10:15:27AM -0600, Nathan Lynch wrote:
quoted
Rune Torgersen wrote:
quoted
I get the following kernel core while a user program I have is dumping
core.
Any DIeas at what to look for? (this is runnign 2.6.24, arch/powerpc on
a 8280)
When runnign the program on 2.6.18 arch/ppc, the program gets a sig 11
and dumps core.
On 2.6.24, I ghet the kernel oops, and then the program hangs sround
forever and is unkillable.
Yes, this program does it reliably:
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
void *threadfn(void *arg)
{
fprintf(stderr, "threadfn\n");
fflush(stderr);
sleep(1);
*(char *)0=0;
return NULL;
}
int main(void)
{
pthread_t thread[4];
int i;
for (i = 0; i < 4; i++)
pthread_create(&thread[0], NULL, threadfn, NULL);
for (;;);
}
Ack!
This is a MPC8540ADS arch/powerpc compatible environment here:
Feb 2 12:59:17 fox_1 kernel: Unable to handle kernel paging request for data at address 0x4802f000
Feb 2 12:59:17 fox_1 kernel: Faulting instruction address: 0xc000d5b8
Feb 2 12:59:17 fox_1 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Feb 2 12:59:17 fox_1 kernel: MPC85xx ADS
Feb 2 12:59:17 fox_1 kernel: Modules linked in:
Feb 2 12:59:17 fox_1 kernel: NIP: c000d5b8 LR: c0010fb8 CTR: 00000080
Feb 2 12:59:17 fox_1 kernel: REGS: c24abb20 TRAP: 0300 Not tainted (2.6.24)
Feb 2 12:59:17 fox_1 kernel: MSR: 00029000 <EE,ME> CR: 22882222 XER: 00000000
Feb 2 12:59:17 fox_1 kernel: DEAR: 4802f000, ESR: 00000000
Feb 2 12:59:17 fox_1 kernel: TASK = cf894d20[942] 'oops' THREAD: c24aa000
Feb 2 12:59:17 fox_1 kernel: GPR00: c22c7680 c24abbd0 cf894d20 4802f000 00000080 000f8b60 4802f000 ffffffff
Feb 2 12:59:17 fox_1 kernel: GPR08: 00000000 c22c7680 000008d1 00000000 22882222 10018a64 00000006 c035a300
Feb 2 12:59:17 fox_1 kernel: GPR16: 00024000 c0380000 c24aa000 c24abc9c c24abc98 c2570480 c22c7680 c0380000
Feb 2 12:59:17 fox_1 kernel: GPR24: c0390420 cf09d000 c0497b60 c5b63948 4802f000 c24aa000 000000bc c0497b60
Feb 2 12:59:17 fox_1 kernel: NIP [c000d5b8] __flush_dcache_icache+0x14/0x40
Feb 2 12:59:17 fox_1 kernel: LR [c0010fb8] update_mmu_cache+0x94/0x98
Feb 2 12:59:17 fox_1 kernel: Call Trace:
Feb 2 12:59:17 fox_1 kernel: [c24abbd0] [c24aa000] 0xc24aa000 (unreliable)
Feb 2 12:59:17 fox_1 kernel: [c24abbe0] [c005d978] handle_mm_fault+0x374/0x6a4
Feb 2 12:59:17 fox_1 kernel: [c24abc30] [c005ddd0] get_user_pages+0x128/0x384
Feb 2 12:59:17 fox_1 kernel: [c24abc90] [c00a80d8] elf_core_dump+0xab8/0xb74
Feb 2 12:59:17 fox_1 kernel: [c24abd30] [c007718c] do_coredump+0x730/0x758
Feb 2 12:59:17 fox_1 kernel: [c24abe30] [c002eeb0] get_signal_to_deliver+0x244/0x3c4
Feb 2 12:59:17 fox_1 kernel: [c24abe80] [c000782c] do_signal+0x48/0x264
Feb 2 12:59:17 fox_1 kernel: [c24abf40] [c000e4ac] do_user_signal+0x74/0xc4
Feb 2 12:59:17 fox_1 kernel: Instruction dump:
Feb 2 12:59:17 fox_1 kernel: 4d820020 7c8903a6 7c001bac 38630020 4200fff8 7c0004ac 4e800020 60000000
Feb 2 12:59:17 fox_1 kernel: 54630026 38800080 7c8903a6 7c661b78 <7c00186c> 38630020 4200fff8 7c0004ac
Feb 2 12:59:17 fox_1 kernel: ---[ end trace a1d91e665173315a ]---
Feb 2 12:59:17 fox_1 kernel: note: oops[942] exited with preempt_count 1
It does not oops when the core dump is disabled.
Regards,
Clemens
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2008-02-03 07:35:46
On Thu, 2008-01-31 at 16:10 -0600, Rune Torgersen wrote:
Scott Wood wrote:
quoted
Scott Wood wrote:
quoted
Nathan Lynch wrote:
quoted
Is the crashing program multithreaded? The first report had firefox
triggering the oops.
OK, I've got a test program that triggers it now. I'll see if I can
figure out what's going on.
The problem seems to be that update_mmu_cache() is called on a guard
page with no access rights.
Changing update_mmu_cache() to always call flush_dcache_icache_page()
fixes it, though a better performing fix would probably be to add an
exception table entry for the dcbst.
I can confirm that this seems to fix it.
Might be better to avoid the flush when the page isn't readable ?
Ben.
From: Scott Wood <hidden> Date: 2008-02-04 18:25:15
Benjamin Herrenschmidt wrote:
On Thu, 2008-01-31 at 16:10 -0600, Rune Torgersen wrote:
quoted
Scott Wood wrote:
quoted
Changing update_mmu_cache() to always call flush_dcache_icache_page()
fixes it, though a better performing fix would probably be to add an
exception table entry for the dcbst.
I can confirm that this seems to fix it.
Might be better to avoid the flush when the page isn't readable ?
Sure, that'd work. I was trying to avoid a tablewalk to determine that,
not noticing the pte argument staring me in the face. :-P
-Scott