From: David Woodhouse <dwmw2@infradead.org> Date: 2007-02-26 00:00:32
On Mon, 2006-12-11 at 20:59 +0000, Linux Kernel Mailing List wrote:
Gitweb: http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=8d610dd52dd1da696e199e4b4545f33a2a5de5c6
Commit: 8d610dd52dd1da696e199e4b4545f33a2a5de5c6
Parent: 8993780a6e44fb4e7ed34e33458506a775356c6e
Author: Linus Torvalds [off-list ref]
AuthorDate: Mon Dec 11 12:12:04 2006 -0800
Committer: Linus Torvalds [off-list ref]
CommitDate: Mon Dec 11 12:12:04 2006 -0800
Make sure we populate the initroot filesystem late enough
We should not initialize rootfs before all the core initializers have
run. So do it as a separate stage just before starting the regular
driver initializers.
Signed-off-by: Linus Torvalds [off-list ref]
This seems to be what's triggering the apparent memory corruption we've
been seeing recently -- in the case of the Fedora kernel it manifests
itself as a BUG() in cache_alloc_refill() when the pmac ide driver
initialises.
Another report was at http://lkml.org/lkml/2006/12/17/4
We've been seeing it on a Mac Mini too, and I managed to reproduce it on
my shinybook this evening by booting with 'mem=512M'.
One side-effect of this patch is to move the call to free_initrd() much
later in the init sequence, potentially after other memory management
code is assuming it's already been freed.
--
dwmw2
One side-effect of this patch is to move the call to free_initrd() much
later in the init sequence, potentially after other memory management
code is assuming it's already been freed.
Hmm. No, I don't think that should be a problem. free_initmem() only
happens at the very, after do_basic_setup() has been run, which includes
all the initcall stuff.
However, it's an interesting observation. How sure are you that it's this
commit that triggers it. You say "This seems to be what's triggering ..",
I'm wondering how firm that is..
Linus
From: David Woodhouse <dwmw2@infradead.org> Date: 2007-02-26 00:44:37
On Sun, 2007-02-25 at 16:24 -0800, Linus Torvalds wrote:
Hmm. No, I don't think that should be a problem. free_initmem() only
happens at the very, after do_basic_setup() has been run, which includes
all the initcall stuff.
However, it's an interesting observation. How sure are you that it's this
commit that triggers it. You say "This seems to be what's triggering ..",
I'm wondering how firm that is..
I found it with git-bisect. The Fedora kernel has been broken on this
particular 512MiB Mac Mini for a while, and now I've reverted the patch
it seems to be fine again. So I'm fairly sure. I'll be surer in a few
minutes once the full RPM build has finished with the patch reverted.
Of course, it could easily be an entirely separate bug which by some
bizarre coincidence is just triggered by this.
--
dwmw2
From: David Woodhouse <dwmw2@infradead.org> Date: 2007-02-26 01:17:13
On Sun, 2007-02-25 at 16:24 -0800, Linus Torvalds wrote:
Hmm. No, I don't think that should be a problem. free_initmem() only
happens at the very, after do_basic_setup() has been run, which
includes all the initcall stuff.
I'm inclined to agree that it _shouldn't_ be a problem. Nevertheless,
even this hack seems sufficient to 'fix' it:
I'm inclined to agree that it _shouldn't_ be a problem. Nevertheless,
even this hack seems sufficient to 'fix' it:
Ok. Clearly something is using that memory. That said, I *suspect* that
the commit that you bisected to is just showing the problem indirectly.
The ordering shouldn't make any difference, but it can obviously make a
huge difference in various allocation patterns etc, thus just showing a
pre-existing problem more clearly..
Can you try adding something like
memset(start, 0xf0, end - start);
to before the return? That might give a better idea of exactly what is
using it after it's free'd, hopefully by having the user trigger some more
spectacular oops..
It is, of course, also entirely possible that the rootfs unpacking change
really *was* buggy, and I am just missing something totally obvious. The
memset() might still make it more obvious, though. Maybe.
.. so adding the "memset()" here would be what I'm suggesting ..
+ return;
.. and you might as well leave the return there, so that nobody else comes
along and re-uses the memory. That should just improve on the chances of
the memset() hopefully catching the problem..
Linus "I don't see anything wrong" Torvalds
From: David Woodhouse <dwmw2@infradead.org> Date: 2007-02-26 04:00:37
On Sun, 2007-02-25 at 19:45 -0800, Linus Torvalds wrote:
Ok. Clearly something is using that memory. That said, I *suspect* that
the commit that you bisected to is just showing the problem indirectly.
The ordering shouldn't make any difference, but it can obviously make a
huge difference in various allocation patterns etc, thus just showing a
pre-existing problem more clearly..
Indeed.
Can you try adding something like
memset(start, 0xf0, end - start);
Yeah, I did that before giving up on it for the day and going in search
of dinner. It changes the failure mode to a BUG() in
cache_free_debugcheck(), at line 2876 of mm/slab.c
It smells like the pages weren't actually reserved in the first place
and we were blithely allocating them. The only problem with that theory
is that the initrd doesn't seem to be getting corrupted -- and if we
were handing out its pages like that then surely _something_ would have
scribbled on it before we tried to read it.
When I head back in tomorrow morning I'll instrument free_initrd_mem()
to check that the PageReserved bit was actually set on each page, before
clearing it. And I'll make the page allocation routines check whether
they're giving out pages between initrd_start and initrd_end, etc.
--
dwmw2
Can you try adding something like
memset(start, 0xf0, end - start);
Yeah, I did that before giving up on it for the day and going in search
of dinner. It changes the failure mode to a BUG() in
cache_free_debugcheck(), at line 2876 of mm/slab.c
Ok, that's just strange.
One obvious thing to do would be to remove all the "__initdata" entries in
mm/slab.c.. But I'd also like to see the full backtrace for the BUG_ON(),
in case that gives any clues at all.
It smells like the pages weren't actually reserved in the first place
and we were blithely allocating them. The only problem with that theory
is that the initrd doesn't seem to be getting corrupted -- and if we
were handing out its pages like that then surely _something_ would have
scribbled on it before we tried to read it.
Yeah, I don't think it's necessarily initrd itself, I'd be more inclined
to think that the reason you see this change with the initrd unpacking is
simply that it does a lot of allocations for the initrd files, so I think
it is only indirectly involved - just because it ends up being a slab
user.
When I head back in tomorrow morning I'll instrument free_initrd_mem()
to check that the PageReserved bit was actually set on each page, before
clearing it. And I'll make the page allocation routines check whether
they're giving out pages between initrd_start and initrd_end, etc.
From: William Lee Irwin III <hidden> Date: 2007-02-26 07:00:07
On Sun, Feb 25, 2007 at 11:01:06PM -0500, David Woodhouse wrote:
Yeah, I did that before giving up on it for the day and going in search
of dinner. It changes the failure mode to a BUG() in
cache_free_debugcheck(), at line 2876 of mm/slab.c
It smells like the pages weren't actually reserved in the first place
and we were blithely allocating them. The only problem with that theory
is that the initrd doesn't seem to be getting corrupted -- and if we
were handing out its pages like that then surely _something_ would have
scribbled on it before we tried to read it.
When I head back in tomorrow morning I'll instrument free_initrd_mem()
to check that the PageReserved bit was actually set on each page, before
clearing it. And I'll make the page allocation routines check whether
they're giving out pages between initrd_start and initrd_end, etc.
Another few things to try would be inserting checks in page_alloc.c for
pages in that specific range before some flag set in free_initrd_mem()
is set, and (conflicting with that, though easily reconciled) unmapping
initrd memory in free_initrd_mem() instead of freeing it.
-- wli
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2007-02-26 15:54:55
On Sun, 2007-02-25 at 20:17 -0500, David Woodhouse wrote:
On Sun, 2007-02-25 at 16:24 -0800, Linus Torvalds wrote:
quoted
Hmm. No, I don't think that should be a problem. free_initmem() only
happens at the very, after do_basic_setup() has been run, which
includes all the initcall stuff.
I'm inclined to agree that it _shouldn't_ be a problem. Nevertheless,
even this hack seems sufficient to 'fix' it:
Could be a powerpc specific bug in initrd handling... I'm still
traveling so I can't really look at it right now, but I wouldn't be
surprised if some of that code did indeed bitrot.
Ben.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2007-02-26 15:55:24
On Sun, 2007-02-25 at 23:01 -0500, David Woodhouse wrote:
Yeah, I did that before giving up on it for the day and going in search
of dinner. It changes the failure mode to a BUG() in
cache_free_debugcheck(), at line 2876 of mm/slab.c
It smells like the pages weren't actually reserved in the first place
and we were blithely allocating them. The only problem with that theory
is that the initrd doesn't seem to be getting corrupted -- and if we
were handing out its pages like that then surely _something_ would have
scribbled on it before we tried to read it.
When I head back in tomorrow morning I'll instrument free_initrd_mem()
to check that the PageReserved bit was actually set on each page, before
clearing it. And I'll make the page allocation routines check whether
they're giving out pages between initrd_start and initrd_end, etc.
And check that we didn't end up stupidly having the initrd share a page
with something else ... (like not aligned end or such thingy).
Ben.
From: David Woodhouse <dwmw2@infradead.org> Date: 2007-02-26 16:24:32
On Sun, 2007-02-25 at 20:13 -0800, Linus Torvalds wrote:
On Sun, 25 Feb 2007, David Woodhouse wrote:
quoted
quoted
Can you try adding something like
memset(start, 0xf0, end - start);
Yeah, I did that before giving up on it for the day and going in search
of dinner. It changes the failure mode to a BUG() in
cache_free_debugcheck(), at line 2876 of mm/slab.c
Ok, that's just strange.
In this case I hadn't left the 'return' in free_initrd_mem(). I was
poisoning the pages and then returning them to the pool as usual.
If I poison the pages and _don't_ return them to the pool, it boots
fine. PageReserved is set on every page in the initrd region; total
page_count() is equal to the number of pages (which doesn't
_necessarily_ mean that page_count() for every page is equal to 1 but
it's a strong hint that that's the case).
Looking in /dev/mem after it boots, I see that my poison is still
present throughout the whole region.
One obvious thing to do would be to remove all the "__initdata" entries in
mm/slab.c..
This is biting us long before we call free_initmem().
But I'd also like to see the full backtrace for the BUG_ON(),
in case that gives any clues at all.
I'll see if I can find a camera.
quoted
It smells like the pages weren't actually reserved in the first place
and we were blithely allocating them. The only problem with that theory
is that the initrd doesn't seem to be getting corrupted -- and if we
were handing out its pages like that then surely _something_ would have
scribbled on it before we tried to read it.
Yeah, I don't think it's necessarily initrd itself, I'd be more inclined
to think that the reason you see this change with the initrd unpacking is
simply that it does a lot of allocations for the initrd files, so I think
it is only indirectly involved - just because it ends up being a slab
user.
Whatever happens, initrd as a 'slab user' is fine. The crashes happen
_later_, when someone else is using the memory which used to belong to
the initrd. In that 'BUG at slab.c:2876' I mentioned above, r3 was
within the initrd region. As I said, I'll try to find a camera.
--
dwmw2
From: Milton Miller <hidden> Date: 2007-02-26 16:45:13
On Feb 27, 2007, at 2:24 AM, David Woodhouse wrote:
On Sun, 2007-02-25 at 20:13 -0800, Linus Torvalds wrote:
quoted
On Sun, 25 Feb 2007, David Woodhouse wrote:
quoted
quoted
Can you try adding something like
memset(start, 0xf0, end - start);
Yeah, I did that before giving up on it for the day and going in
search
of dinner. It changes the failure mode to a BUG() in
cache_free_debugcheck(), at line 2876 of mm/slab.c
Ok, that's just strange.
In this case I hadn't left the 'return' in free_initrd_mem(). I was
poisoning the pages and then returning them to the pool as usual.
If I poison the pages and _don't_ return them to the pool, it boots
fine. PageReserved is set on every page in the initrd region; total
page_count() is equal to the number of pages (which doesn't
_necessarily_ mean that page_count() for every page is equal to 1 but
it's a strong hint that that's the case).
Looking in /dev/mem after it boots, I see that my poison is still
present throughout the whole region.
quoted
One obvious thing to do would be to remove all the "__initdata"
entries in
mm/slab.c..
This is biting us long before we call free_initmem().
quoted
But I'd also like to see the full backtrace for the BUG_ON(),
in case that gives any clues at all.
I'll see if I can find a camera.
quoted
quoted
It smells like the pages weren't actually reserved in the first place
and we were blithely allocating them. The only problem with that
theory
is that the initrd doesn't seem to be getting corrupted -- and if we
were handing out its pages like that then surely _something_ would
have
scribbled on it before we tried to read it.
Yeah, I don't think it's necessarily initrd itself, I'd be more
inclined
to think that the reason you see this change with the initrd
unpacking is
simply that it does a lot of allocations for the initrd files, so I
think
it is only indirectly involved - just because it ends up being a slab
user.
Whatever happens, initrd as a 'slab user' is fine. The crashes happen
_later_, when someone else is using the memory which used to belong to
the initrd. In that 'BUG at slab.c:2876' I mentioned above, r3 was
within the initrd region. As I said, I'll try to find a camera.
Just a thought,
Any chance you are using one of the unusal code paths, like the
bootloader
moving the initrd or using a kernel-crash region?
milton
From: john stultz <hidden> Date: 2007-02-26 19:27:52
On Sun, 2007-02-25 at 19:00 -0500, David Woodhouse wrote:
On Mon, 2006-12-11 at 20:59 +0000, Linux Kernel Mailing List wrote:
quoted
Make sure we populate the initroot filesystem late enough
This seems to be what's triggering the apparent memory corruption we've
been seeing recently -- in the case of the Fedora kernel it manifests
itself as a BUG() in cache_alloc_refill() when the pmac ide driver
initialises.
Another report was at http://lkml.org/lkml/2006/12/17/4
We've been seeing it on a Mac Mini too, and I managed to reproduce it on
my shinybook this evening by booting with 'mem=512M'.
Just for reference (as its not in the thread linked above), this issue
disappeared for me after some config changes (I somehow changed my
selection when I backtracked and then moved forward w/ git bisect).
I've not been able to reproduce it since, but I know others (BCC'ed on
this note) have seen it and might prod them to come forth with details
(and broken .config files)
thanks
-john
From: Kumar Gala <hidden> Date: 2007-02-26 20:52:14
On Feb 26, 2007, at 9:51 AM, Benjamin Herrenschmidt wrote:
On Sun, 2007-02-25 at 20:17 -0500, David Woodhouse wrote:
quoted
On Sun, 2007-02-25 at 16:24 -0800, Linus Torvalds wrote:
quoted
Hmm. No, I don't think that should be a problem. free_initmem() only
happens at the very, after do_basic_setup() has been run, which
includes all the initcall stuff.
I'm inclined to agree that it _shouldn't_ be a problem. Nevertheless,
even this hack seems sufficient to 'fix' it:
Could be a powerpc specific bug in initrd handling... I'm still
traveling so I can't really look at it right now, but I wouldn't be
surprised if some of that code did indeed bitrot.
Ben.
Could there be some issue with initrd getting reserved properly via
prom_init.c. I know we make sure there are memreserve's in the fdt
for initrd on embedded ppc.
- k
From: David Woodhouse <dwmw2@infradead.org> Date: 2007-02-26 20:57:19
On Mon, 2007-02-26 at 10:44 -0600, Milton Miller wrote:
Any chance you are using one of the unusal code paths, like the
bootloader moving the initrd or using a kernel crash region?
I'm doing nothing special. And I'm less sure now about the trigger. I
built a Fedora 7 test 2 install tree with the patch reverted, and
managed to boot and install.... but now when I boot the _same_ machine
with the same CD, it fails.
Now I'm starting to wonder if it's something the firmware sets up to DMA
to a certain region of memory, which makes it non-deterministic. And the
other things we're blaming are only making a difference because they
change the layout of what we have in memory.
--
dwmw2
Now I'm starting to wonder if it's something the firmware sets up to DMA
to a certain region of memory, which makes it non-deterministic. And the
other things we're blaming are only making a difference because they
change the layout of what we have in memory.
USB controller issues? We used to have these really hard-to-debug problems
with the USB controller being active and having had the BIOS set up the
command queues etc. Really subtle. It's why we now have PCI quirks for
shutting up (most) USB controllers very early.
If there is some USB controller that we miss, or that sets up its command
chain to some unexpected area (so that USB is active and corrupting memory
even very early on), that could explain it.
Linus
From: Paul TBBle Hampson <hidden> Date: 2007-02-26 22:52:38
On Mon, Feb 26, 2007 at 11:27:47AM -0800, john stultz wrote:
On Sun, 2007-02-25 at 19:00 -0500, David Woodhouse wrote:
quoted
On Mon, 2006-12-11 at 20:59 +0000, Linux Kernel Mailing List wrote:
quoted
Make sure we populate the initroot filesystem late enough
quoted
This seems to be what's triggering the apparent memory corruption we've
been seeing recently -- in the case of the Fedora kernel it manifests
itself as a BUG() in cache_alloc_refill() when the pmac ide driver
initialises.
Another report was at http://lkml.org/lkml/2006/12/17/4
We've been seeing it on a Mac Mini too, and I managed to reproduce it on
my shinybook this evening by booting with 'mem=512M'.
Just for reference (as its not in the thread linked above), this issue
disappeared for me after some config changes (I somehow changed my
selection when I backtracked and then moved forward w/ git bisect).
I've not been able to reproduce it since, but I know others (BCC'ed on
this note) have seen it and might prod them to come forth with details
(and broken .config files)
In my case, disabling CPU_FREQ_PMAC made the failure go away.
After reverting this patch, CPU_FREQ_PMAC is once again operating
successfully, so far.
--
-----------------------------------------------------------
Paul "TBBle" Hampson, B.Sc, LPI, MCSE
On-hiatus Asian Studies student, ANU
The Boss, Bubblesworth Pty Ltd (ABN: 51 095 284 361)
Paul.Hampson@Pobox.Com
Of course Pacman didn't influence us as kids. If it did,
we'd be running around in darkened rooms, popping pills and
listening to repetitive music.
-- Kristian Wilson, Nintendo, Inc, 1989
License: http://creativecommons.org/licenses/by/2.1/au/
-----------------------------------------------------------
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2007-02-27 06:48:46
USB controller issues? We used to have these really hard-to-debug problems
with the USB controller being active and having had the BIOS set up the
command queues etc. Really subtle. It's why we now have PCI quirks for
shutting up (most) USB controllers very early.
On powermacs or powerbooks, the USB controller is shut down by the
firmware when we call the "quiesce" OF call from prom_init.c, which
happens before the kernel relocates itself to 0 and takes over memory.
Unless we fucked up something in there, I wouldn't expect that to be the
cause.
If there is some USB controller that we miss, or that sets up its command
chain to some unexpected area (so that USB is active and corrupting memory
even very early on), that could explain it.
Did we setup the OHCI controller when the crash happen ? Maybe we broke
something subtle in the USB stack ?
Ben.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2007-02-27 07:04:22
quoted
I've not been able to reproduce it since, but I know others (BCC'ed on
this note) have seen it and might prod them to come forth with details
(and broken .config files)
In my case, disabling CPU_FREQ_PMAC made the failure go away.
After reverting this patch, CPU_FREQ_PMAC is once again operating
successfully, so far.
Hrm.. which cpufreq method is used on both your machines ? If it's the
one involving the PMU, it does involve a full hard reset of the
processor (with appropriate cache flushes etc...), maybe something's
going wrong in that area....
Ben.
I've not been able to reproduce it since, but I know others (BCC'ed
on
this note) have seen it and might prod them to come forth with
details
(and broken .config files)
In my case, disabling CPU_FREQ_PMAC made the failure go away.
After reverting this patch, CPU_FREQ_PMAC is once again operating
successfully, so far.
Hrm.. which cpufreq method is used on both your machines ? If it's the
one involving the PMU, it does involve a full hard reset of the
processor (with appropriate cache flushes etc...), maybe something's
going wrong in that area....
It's most likely a red herring, lots of config changes
make the bug go away on some kernel versions (but not
on others); the problem is very sensitive to changes in
memory layout.
Segher
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2007-02-28 06:49:18
It's most likely a red herring, lots of config changes
make the bug go away on some kernel versions (but not
on others); the problem is very sensitive to changes in
memory layout.
I wouldn't be that sure ... I've had problems in the past with PMU based
cpufreq... looks like flushing all caches and hard-resetting the
processor on the fly when there can be pending DMAs might be a source of
trouble... especially on CPUs that don't have working cache flush HW
assist.
Ben.
From: David Woodhouse <dwmw2@infradead.org> Date: 2007-02-28 10:13:29
On Wed, 2007-02-28 at 07:43 +0100, Benjamin Herrenschmidt wrote:
I wouldn't be that sure ... I've had problems in the past with PMU based
cpufreq... looks like flushing all caches and hard-resetting the
processor on the fly when there can be pending DMAs might be a source of
trouble... especially on CPUs that don't have working cache flush HW
assist.
I've seen it on a PowerMac3,1 (400MHz G4) where we don't have cpufreq.
I've also seen it on the latest 1.5GHz Mac Mini, and on my shinybook.
They all fall over with the latest kernel, although the shinybook only
does so immediately when booted with mem=512M. The shinybook does crash
later with new kernels though; I don't yet know why. It could be the
same thing, or it could be something different. That one seemed to
appear between Fedora's 2.6.19-1.2913 and 2.6.19-1.2914 kernels, where
we did nothing but turned CONFIG_SYSFS_DEPRECATED on.
I don't blame cpufreq. At various times I've been equally convinced that
it was due to CONFIG_KPROBES, and Linus' initrd-moving patch.
--
dwmw2
From: Michael Ellerman <hidden> Date: 2007-03-01 00:31:02
On Wed, 2007-02-28 at 10:13 +0000, David Woodhouse wrote:
On Wed, 2007-02-28 at 07:43 +0100, Benjamin Herrenschmidt wrote:
quoted
I wouldn't be that sure ... I've had problems in the past with PMU based
cpufreq... looks like flushing all caches and hard-resetting the
processor on the fly when there can be pending DMAs might be a source of
trouble... especially on CPUs that don't have working cache flush HW
assist.
I've seen it on a PowerMac3,1 (400MHz G4) where we don't have cpufreq.
I've also seen it on the latest 1.5GHz Mac Mini, and on my shinybook.
They all fall over with the latest kernel, although the shinybook only
does so immediately when booted with mem=512M. The shinybook does crash
later with new kernels though; I don't yet know why. It could be the
same thing, or it could be something different. That one seemed to
appear between Fedora's 2.6.19-1.2913 and 2.6.19-1.2914 kernels, where
we did nothing but turned CONFIG_SYSFS_DEPRECATED on.
I don't blame cpufreq. At various times I've been equally convinced that
it was due to CONFIG_KPROBES, and Linus' initrd-moving patch.
Is there any pattern to the way it dies? Or is it just randomly dieing
somewhere depending on which config options you have enabled?
This is starting to sound reminiscent of a bug I chased for a while last
year on Power5, but didn't find. It was "fixed" on some machines by
disabling CONFIG_KEXEC, and/or other random unrelated CONFIG options.
Unfortunately it magically stopped reproducing so I never caught it :/
cheers
--
Michael Ellerman
OzLabs, IBM Australia Development Lab
wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)
We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
From: Paul TBBle Hampson <hidden> Date: 2007-03-12 23:01:48
On Thu, Mar 01, 2007 at 09:30:56AM +0900, Michael Ellerman wrote:
On Wed, 2007-02-28 at 10:13 +0000, David Woodhouse wrote:
quoted
On Wed, 2007-02-28 at 07:43 +0100, Benjamin Herrenschmidt wrote:
quoted
I wouldn't be that sure ... I've had problems in the past with PMU based
cpufreq... looks like flushing all caches and hard-resetting the
processor on the fly when there can be pending DMAs might be a source of
trouble... especially on CPUs that don't have working cache flush HW
assist.
I've seen it on a PowerMac3,1 (400MHz G4) where we don't have cpufreq.
I've also seen it on the latest 1.5GHz Mac Mini, and on my shinybook.
They all fall over with the latest kernel, although the shinybook only
does so immediately when booted with mem=512M. The shinybook does crash
later with new kernels though; I don't yet know why. It could be the
same thing, or it could be something different. That one seemed to
appear between Fedora's 2.6.19-1.2913 and 2.6.19-1.2914 kernels, where
we did nothing but turned CONFIG_SYSFS_DEPRECATED on.
I don't blame cpufreq. At various times I've been equally convinced that
it was due to CONFIG_KPROBES, and Linus' initrd-moving patch.
Is there any pattern to the way it dies? Or is it just randomly dieing
somewhere depending on which config options you have enabled?
This is starting to sound reminiscent of a bug I chased for a while last
year on Power5, but didn't find. It was "fixed" on some machines by
disabling CONFIG_KEXEC, and/or other random unrelated CONFIG options.
Unfortunately it magically stopped reproducing so I never caught it :/
Hmm. The crash came back after I booted into Mac OS X and back. It was however
a different crash, I believe it was coming from the USB modules (as it would
keep going when it happened, and get another crash, which tended to scroll away
too fast for me to capture) but I believe it was still getting down into the
slab code and actually dying there.
However, reverting the reversion of
8d610dd52dd1da696e199e4b4545f33a2a5de5c6 and instead applying
the following patch:
diff -ru linux-source-2.6.20.orig/arch/powerpc/mm/init_32.c linux-source-2.6.20/arch/powerpc/mm/init_32.c
which if I recall correctly David Woodhouse posted to this thread,
seems to have fixed it.
I dunno if it's relevant, but my initrd.img is 13193315 bytes long,
(ie 99 bytes over 12884k) and the above logs:
"NOT Freeing initrd memory: 12888k freed"
which makes sense...
I of course completely failed to think to check this with the crashing
kernel, if it seems relevant I can roll back to it and get the numbers.
--
-----------------------------------------------------------
Paul "TBBle" Hampson, B.Sc, LPI, MCSE
On-hiatus Asian Studies student, ANU
The Boss, Bubblesworth Pty Ltd (ABN: 51 095 284 361)
Paul.Hampson@Pobox.Com
Of course Pacman didn't influence us as kids. If it did,
we'd be running around in darkened rooms, popping pills and
listening to repetitive music.
-- Kristian Wilson, Nintendo, Inc, 1989
License: http://creativecommons.org/licenses/by/2.1/au/
-----------------------------------------------------------
From: Kumar Gala <hidden> Date: 2007-03-13 03:04:13
On Mar 12, 2007, at 6:01 PM, Paul TBBle Hampson wrote:
quoted hunk
On Thu, Mar 01, 2007 at 09:30:56AM +0900, Michael Ellerman wrote:
quoted
On Wed, 2007-02-28 at 10:13 +0000, David Woodhouse wrote:
quoted
On Wed, 2007-02-28 at 07:43 +0100, Benjamin Herrenschmidt wrote:
quoted
I wouldn't be that sure ... I've had problems in the past with
PMU based
cpufreq... looks like flushing all caches and hard-resetting the
processor on the fly when there can be pending DMAs might be a
source of
trouble... especially on CPUs that don't have working cache
flush HW
assist.
I've seen it on a PowerMac3,1 (400MHz G4) where we don't have
cpufreq.
I've also seen it on the latest 1.5GHz Mac Mini, and on my
shinybook.
They all fall over with the latest kernel, although the shinybook
only
does so immediately when booted with mem=512M. The shinybook does
crash
later with new kernels though; I don't yet know why. It could be the
same thing, or it could be something different. That one seemed to
appear between Fedora's 2.6.19-1.2913 and 2.6.19-1.2914 kernels,
where
we did nothing but turned CONFIG_SYSFS_DEPRECATED on.
I don't blame cpufreq. At various times I've been equally
convinced that
it was due to CONFIG_KPROBES, and Linus' initrd-moving patch.
quoted
Is there any pattern to the way it dies? Or is it just randomly
dieing
somewhere depending on which config options you have enabled?
quoted
This is starting to sound reminiscent of a bug I chased for a
while last
year on Power5, but didn't find. It was "fixed" on some machines by
disabling CONFIG_KEXEC, and/or other random unrelated CONFIG options.
Unfortunately it magically stopped reproducing so I never caught
it :/
Hmm. The crash came back after I booted into Mac OS X and back. It
was however
a different crash, I believe it was coming from the USB modules (as
it would
keep going when it happened, and get another crash, which tended to
scroll away
too fast for me to capture) but I believe it was still getting down
into the
slab code and actually dying there.
However, reverting the reversion of
8d610dd52dd1da696e199e4b4545f33a2a5de5c6 and instead applying
the following patch:
diff -ru linux-source-2.6.20.orig/arch/powerpc/mm/init_32.c linux-
source-2.6.20/arch/powerpc/mm/init_32.c
(end - start) >> 10);
+ return;
for (; start < end; start += PAGE_SIZE) {
ClearPageReserved(virt_to_page(start));
init_page_count(virt_to_page(start));
which if I recall correctly David Woodhouse posted to this thread,
seems to have fixed it.
I dunno if it's relevant, but my initrd.img is 13193315 bytes long,
(ie 99 bytes over 12884k) and the above logs:
"NOT Freeing initrd memory: 12888k freed"
which makes sense...
I of course completely failed to think to check this with the crashing
kernel, if it seems relevant I can roll back to it and get the
numbers.
Have you tried 2.6.20.2, there was a significant bug in get_order()
that was deemed to be causing these issues.
- k
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2007-03-13 07:07:20
Hmm. The crash came back after I booted into Mac OS X and back. It was however
a different crash, I believe it was coming from the USB modules (as it would
keep going when it happened, and get another crash, which tended to scroll away
too fast for me to capture) but I believe it was still getting down into the
slab code and actually dying there.
Have you tried, instead, to apply
38f3323037de22bb0089d08be27be01196e7148b ? (That is revert
39d61db0edb34d60b83c5e0d62d0e906578cc707).
I suspect this is the proper fix...
Ben.
quoted hunk
However, reverting the reversion of
8d610dd52dd1da696e199e4b4545f33a2a5de5c6 and instead applying
the following patch:
diff -ru linux-source-2.6.20.orig/arch/powerpc/mm/init_32.c linux-source-2.6.20/arch/powerpc/mm/init_32.c
which if I recall correctly David Woodhouse posted to this thread,
seems to have fixed it.
I dunno if it's relevant, but my initrd.img is 13193315 bytes long,
(ie 99 bytes over 12884k) and the above logs:
"NOT Freeing initrd memory: 12888k freed"
which makes sense...
I of course completely failed to think to check this with the crashing
kernel, if it seems relevant I can roll back to it and get the numbers.
From: Paul TBBle Hampson <hidden> Date: 2007-03-16 07:20:21
On Tue, Mar 13, 2007 at 08:03:49AM +0100, Benjamin Herrenschmidt wrote:
quoted
Hmm. The crash came back after I booted into Mac OS X and back. It was however
a different crash, I believe it was coming from the USB modules (as it would
keep going when it happened, and get another crash, which tended to scroll away
too fast for me to capture) but I believe it was still getting down into the
slab code and actually dying there.
Have you tried, instead, to apply
38f3323037de22bb0089d08be27be01196e7148b ? (That is revert
39d61db0edb34d60b83c5e0d62d0e906578cc707).
That's working fine at the moment, and has even survived a trip to Mac
OS X and back.
Thankyou.
--
-----------------------------------------------------------
Paul "TBBle" Hampson, B.Sc, LPI, MCSE
On-hiatus Asian Studies student, ANU
The Boss, Bubblesworth Pty Ltd (ABN: 51 095 284 361)
Paul.Hampson@Pobox.Com
Of course Pacman didn't influence us as kids. If it did,
we'd be running around in darkened rooms, popping pills and
listening to repetitive music.
-- Kristian Wilson, Nintendo, Inc, 1989
License: http://creativecommons.org/licenses/by/2.1/au/
-----------------------------------------------------------