From: Torsten Kaiser <hidden> Date: 2007-11-21 19:33:55
On Nov 21, 2007 10:29 AM, Andrew Morton [off-list ref] wrote:
On Wed, 21 Nov 2007 14:52:26 +0530 Kamalesh Babulal [off-list ref] wrote:
quoted
Andrew Morton wrote:
quoted
On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal [off-list ref] wrote:
quoted
ACPI: Core revision 20070126
..MP-BIOS bug: 8254 timer not connected to IO-APIC
Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
I seen an identical error.
quoted
quoted
ACPI or x86 breakage, I guess.
Did 'noapic' work?
Passing noapic works,
OK.
Not for me. I get a similar oops, but then the kernel panics
quoted
but the kernel oops's
[ 97.161103] Unable to handle kernel NULL pointer dereference at 0000000000000009 RIP:
[ 97.193973] [<ffffffff802341df>] cpu_to_allnodes_group+0x69/0x7c
[snip]
urgh, mess. Enabling frame pointers might help here.
From: Torsten Kaiser <hidden> Date: 2007-11-21 19:48:52
On Nov 21, 2007 8:22 PM, Len Brown [off-list ref] wrote:
On Wednesday 21 November 2007 01:18, Andrew Morton wrote:
quoted
On Wed, 21 Nov 2007 11:41:23 +0530 Kamalesh Babulal [off-list ref] wrote:
quoted
quoted
SMP alternatives: switching to UP code
ACPI: Core revision 20070126
..MP-BIOS bug: 8254 timer not connected to IO-APIC
did previous kernels print this too?
Not since my last BIOS upgrade.
This is from what dmesg's I still had laying around:
2.6.22-rc6-mm1: No
2.6.23-rc1-mm1, 2.6.23-rc2-mm1, 2.6.23-rc3-mm1: Yes
2.6.23-rc3-mm1 after BIOS upgrade: No
2.6.23-rc4-mm1...2.6.24-rc2-mm1: No
quoted
quoted
Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
ACPI or x86 breakage, I guess.
If you suspect ACPI breakage, then try "acpi=off" or "acpi=noirq".
ACPI doesn't look guilty.
acpi=noirq:
[ 39.905884] Freeing SMP alternatives: 28k freed
[ 39.910674] ACPI: Core revision 20070126
[ 39.916542] ACPI: setting ELCR to 0e20 (from 0c20)
[ 39.921855] ExtINT not setup in hardware but reported by MP table
[ 39.928244] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[ 39.934586] Kernel panic - not syncing: IO-APIC + timer doesn't
work! Try using the 'noapic' kernel parameter
[ 39.934587]
acpi=off:
[ 0.000000] Freeing SMP alternatives: 28k freed
[ 0.000000] ExtINT not setup in hardware but reported by MP table
[ 0.000000] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[ 0.000000] Kernel panic - not syncing: IO-APIC + timer doesn't
work! Try using the 'noapic' kernel parameter
[ 0.000000]
Torsten
..MP-BIOS bug: 8254 timer not connected to IO-APIC
Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
ACPI or x86 breakage, I guess.
Did 'noapic' work?
No! The box freezes somewhere after "Freeing unused kernel memory"...
Bisection points to git-x86.patch, though.
git-bisect start
# good: [f05092637dc0d9a3f2249c9b283b973e6e96b7d2] Linux 2.6.24-rc3
git-bisect good f05092637dc0d9a3f2249c9b283b973e6e96b7d2
# bad: [46c8c396d2c87b786a7fac615c289f85a18e53ce] w1-build-fix
git-bisect bad 46c8c396d2c87b786a7fac615c289f85a18e53ce
# bad: [4e22f4852c48e1eddfe04299e78c0456164abe86] frv-move-dma-macros-to-scatterlisth-for-consistency
git-bisect bad 4e22f4852c48e1eddfe04299e78c0456164abe86
# bad: [4e22f4852c48e1eddfe04299e78c0456164abe86] frv-move-dma-macros-to-scatterlisth-for-consistency
git-bisect bad 4e22f4852c48e1eddfe04299e78c0456164abe86
# good: [d5135f31313af2be37d8ccb71e2a42f8e221d8c4] ide-mm-ide-disk-extend-timeout-for-pio-out-commands
git-bisect good d5135f31313af2be37d8ccb71e2a42f8e221d8c4
# good: [6be815e83f506f4c39a46cf59014e29a95c5e6c4] iommu-sg-merging-call-blk_queue_segment_boundary-in-__scsi_alloc_queue
git-bisect good 6be815e83f506f4c39a46cf59014e29a95c5e6c4
# good: [6be815e83f506f4c39a46cf59014e29a95c5e6c4] iommu-sg-merging-call-blk_queue_segment_boundary-in-__scsi_alloc_queue
git-bisect good 6be815e83f506f4c39a46cf59014e29a95c5e6c4
# bad: [c792db6d06114a85e33a27c89e9e979f11b951c4] slub-fix-coding-style-violations
git-bisect bad c792db6d06114a85e33a27c89e9e979f11b951c4
# bad: [c792db6d06114a85e33a27c89e9e979f11b951c4] slub-fix-coding-style-violations
git-bisect bad c792db6d06114a85e33a27c89e9e979f11b951c4
# bad: [76f3939b76ff557f73720b57a16716196f04e407] x86_64-make-sparsemem-vmemmap-the-default-memory-model-v2
git-bisect bad 76f3939b76ff557f73720b57a16716196f04e407
# good: [b8ba611566d8799a979b190d4bb14305ca64ee0e] sis-fb-driver-_ioctl32_conversion-functions-do-not-exist-in-recent-kernels
git-bisect good b8ba611566d8799a979b190d4bb14305ca64ee0e
# good: [e34995928859308d2abef1709332e2b12d36db2f] git-ipwireless_cs
git-bisect good e34995928859308d2abef1709332e2b12d36db2f
# bad: [f520abbbe11bc8253714bcd34aaaf19bdf82189e] git-x86-identify_cpu-fix
git-bisect bad f520abbbe11bc8253714bcd34aaaf19bdf82189e
I honestly tried fresh #mm from x86 tree -- the one which ends at commit
70be766db1105c0fc9aed8e954d0c343c1eda067 "x86: Add the RDC machine
specific reboot fixup". FWIW, commit "x86: validate against ACPI motherboard
resources" is innocent. After "x86: make stack size configurable" damn thing
wouldn't build and applying fixets from -mm doesn't help at 3AM.
Again, it's late here, I'll recheck today.
From: Andrew Morton <akpm@linux-foundation.org> Date: 2007-11-26 20:35:35
On Mon, 26 Nov 2007 14:39:43 -0500
Rik van Riel [off-list ref] wrote:
On Tue, 20 Nov 2007 22:18:39 -0800
Andrew Morton [off-list ref] wrote:
quoted
quoted
..MP-BIOS bug: 8254 timer not connected to IO-APIC
Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
ACPI or x86 breakage, I guess.
Did 'noapic' work?
I got the same bug as above, 'noapic' gets past that point
We still don't know what caused this, afaik.
and right to the
next oops. I'm posting it here because this one is different from the others
in the thread, yet looks vaguely related:
Unable to handle kernel NULL pointer dereference at 0000000000000021 RIP:
[<ffffffff8108382a>] refresh_zone_stat_thresholds+0x6d/0x90
PGD 0
Oops: 0002 [1] SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.24-rc3-mm1 #2
RIP: 0010:[<ffffffff8108382a>] [<ffffffff8108382a>] refresh_zone_stat_thresholds+0x6d/0x90
RSP: 0000:ffff81007fb59ec0 EFLAGS: 00010293
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffffffff8146fb38 RDI: 0000000000000001
RBP: ffff81000000c000 R08: 0000000000000000 R09: 0000000000000000
R10: ffff81007fb59e60 R11: 0000000000000028 R12: ffffffff814d4558
R13: 0000000000000000 R14: ffffffff814b62c0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffffffff813d9000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000021 CR3: 0000000000201000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo FFFF81007FB58000, task FFFF81007FB56000)
Stack: 0000000000000000 0000000000000000 0000000000000000 ffffffff814a3839
0000000000000000 ffffffff8148e626 ffff81007fb56000 ffffffff8126d36a
0000000000000000 ffffffffffffffff ffffffff8105786b 0000000000000000
Call Trace:
[<ffffffff814a3839>] setup_vmstat+0x6/0x40
[<ffffffff8148e626>] kernel_init+0x169/0x2d8
[<ffffffff8126d36a>] trace_hardirqs_on_thunk+0x35/0x3a
[<ffffffff8105786b>] trace_hardirqs_on+0x115/0x138
[<ffffffff8100ce48>] child_rip+0xa/0x12
[<ffffffff8100c55f>] restore_args+0x0/0x30
[<ffffffff8148e4bd>] kernel_init+0x0/0x2d8
[<ffffffff8100ce3e>] child_rip+0x0/0x12
INFO: lockdep is turned off.
hm. This smells like a startup ordering problem, but everything which
refresh_zone_stat_thresholds() should be set up by the time we run
initcalls. Maybe the zone lists are bad?
On Mon, 26 Nov 2007 14:39:43 -0500
Rik van Riel [off-list ref] wrote:
quoted
On Tue, 20 Nov 2007 22:18:39 -0800
Andrew Morton [off-list ref] wrote:
quoted
quoted
..MP-BIOS bug: 8254 timer not connected to IO-APIC
Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
ACPI or x86 breakage, I guess.
Did 'noapic' work?
I got the same bug as above, 'noapic' gets past that point
We still don't know what caused this, afaik.
yes. Is it a regression? If yes, could someone try to bisect it so that
we can fix it? If it's caused by x86.git then the 'mm' branch of the x86
git tree can be used for bisection:
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
it's supposed to build and boot fine at every bisection point. The
bisection run can be cut significantly by narrowing the bisection to the
arch/x86 changes only:
git-bisect start arch/x86 include/asm-x86/
(and if it finds a nonsensical commit, i.e. the breakage is not caused
by the x86 commits, save the "git-bisect log" output into a file,
restart the git bisection and use "git-bisect replay" to insert all the
test points into a fuller bisection run - this saves quite some time.)
Ingo
hm. This smells like a startup ordering problem, but everything which
refresh_zone_stat_thresholds() should be set up by the time we run
initcalls. Maybe the zone lists are bad?
Or the CPU array. Look at the oops Kamalesh got a few mails upthread...
I guess I'll have to start a bisect - can't port the VM code to a kernel
that doesn't boot...
--
All Rights Reversed
From: Christoph Lameter <hidden> Date: 2007-11-26 20:56:19
On Mon, 26 Nov 2007, Andrew Morton wrote:
hm. This smells like a startup ordering problem, but everything which
refresh_zone_stat_thresholds() should be set up by the time we run
initcalls. Maybe the zone lists are bad?
refresh_zone_stat_thresholds goes through each zone and updates
the stat threshold for every per cpu structure in each zone.
So this could be a processor marked online where the pcp structures have
not been allocated or a zone NULL pointer.
On Mon, 26 Nov 2007 14:39:43 -0500
Rik van Riel [off-list ref] wrote:
quoted
On Tue, 20 Nov 2007 22:18:39 -0800
Andrew Morton [off-list ref] wrote:
quoted
quoted
..MP-BIOS bug: 8254 timer not connected to IO-APIC
Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
ACPI or x86 breakage, I guess.
Did 'noapic' work?
I got the same bug as above, 'noapic' gets past that point
We still don't know what caused this, afaik.
yes. Is it a regression? If yes, could someone try to bisect it so that
we can fix it? If it's caused by x86.git then the 'mm' branch of the x86
git tree can be used for bisection:
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
I did, but it's hard, if you don't know the BAD point. HEAD boots fine and 'x86:
randomize brk' too (the top of git-x86.patch). Andrew, how do you pull it, git
#mm doesn't fit to the ids from the patch.
Maybe if you can emit a broken-out with the fresh pull to test?
regards,
--
Jiri Slaby (jirislaby@gmail.com)
Faculty of Informatics, Masaryk University
From: Andrew Morton <akpm@linux-foundation.org> Date: 2007-11-26 22:19:07
On Mon, 26 Nov 2007 23:08:33 +0100
Jiri Slaby [off-list ref] wrote:
On 11/26/2007 09:45 PM, Ingo Molnar wrote:
quoted
* Andrew Morton [off-list ref] wrote:
quoted
On Mon, 26 Nov 2007 14:39:43 -0500
Rik van Riel [off-list ref] wrote:
quoted
On Tue, 20 Nov 2007 22:18:39 -0800
Andrew Morton [off-list ref] wrote:
quoted
quoted
..MP-BIOS bug: 8254 timer not connected to IO-APIC
Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 'noapic' kernel parameter
ACPI or x86 breakage, I guess.
Did 'noapic' work?
I got the same bug as above, 'noapic' gets past that point
We still don't know what caused this, afaik.
yes. Is it a regression? If yes, could someone try to bisect it so that
we can fix it? If it's caused by x86.git then the 'mm' branch of the x86
git tree can be used for bisection:
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
I did, but it's hard, if you don't know the BAD point. HEAD boots fine and 'x86:
randomize brk' too (the top of git-x86.patch).
So the bug wasn't in git-x86 in 2.6.24-rc3-mm1.
But it might be in there now, as some patches got moved over.
Or it could be git-acpi. Or lots of other things.
Andrew, how do you pull it, git
#mm doesn't fit to the ids from the patch.
Yes it did :). And it worked. Both in qemu and on my desktop...
qemu output at:
http://www.fi.muni.cz/~xslaby/sklad/qemu-output.txt
thanks,
--
Jiri Slaby (jirislaby@gmail.com)
Faculty of Informatics, Masaryk University
Yes it did :). And it worked. Both in qemu and on my desktop...
boggle. Let's slap 2.6.25 on it and take the rest of the year off.
No worries, the mmotm compiling issue seems to have been fixed:
CC [M] drivers/scsi/libsas/sas_ata.o
drivers/scsi/libsas/sas_ata.c:39: error: field ‘rphy’ has incomplete type
drivers/scsi/libsas/sas_ata.c: In function ‘sas_discover_sata’:
drivers/scsi/libsas/sas_ata.c:773: error: implicit declaration of function ‘ata_sas_rphy_alloc’
drivers/scsi/libsas/sas_ata.c:775: error: dereferencing pointer to incomplete type
drivers/scsi/libsas/sas_ata.c:775: warning: assignment makes pointer from integer without a cast
drivers/scsi/libsas/sas_ata.c:781: error: dereferencing pointer to incomplete type
drivers/scsi/libsas/sas_ata.c:782: error: dereferencing pointer to incomplete type
drivers/scsi/libsas/sas_ata.c:784: warning: type defaults to ‘int’ in declaration of ‘__mptr’
drivers/scsi/libsas/sas_ata.c:784: warning: initialization from incompatible pointer type
drivers/scsi/libsas/sas_ata.c:791: error: implicit declaration of function ‘ata_sas_rphy_add’
drivers/scsi/libsas/sas_ata.c:807: error: implicit declaration of function ‘ata_sas_rphy_delete’
drivers/scsi/libsas/sas_ata.c:809: error: implicit declaration of function ‘ata_sas_rphy_free’
make[3]: *** [drivers/scsi/libsas/sas_ata.o] Error 1
make[2]: *** [drivers/scsi/libsas] Error 2
make[1]: *** [drivers/scsi] Error 2
make: *** [drivers] Error 2
So much for continuing the bisect with that tree, to find the
cause of the second bug :)
Guess I'll extract an x86 tree changeset first, to place into
the 2.6.23-rc3-mm1 broken out tree and work from there...
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan