Random crashes with 2.6.27-rc3 on PPC

I am seeing random kernel and userland application
crashes on a Powerbook running a 2.6.27-rc3 based kernel (wireless-testing.git).

The crashes did recently appear. It might be the case that they were
introduced with the merge of 2.6.27-rc1 into wireless-testing.
I'm not sure on that one, however. Just a guess. I still need to
do more testing (also on vanilla upstream kernels).

The crashes are completely random and they look like bad hardware.
However I cannot reproduce on 2.6.25.9 (That's a kernel I still had
installed, so I tried that one). So it most likely is _not_ caused
by faulty hardware.

The crashes are hard to reproduce, and happen about every 20 minutes
when compiling a kernel tree. (gcc segfaults). Sometimes the kernel
oopses in random places with pointer dereference faults.

Is this a known issue?
I'm going to bisect this one, but it will take a lot of time, as reproducing
takes about 20 minutes. So that's about an hour for one test round.

The kernel configuration is the following:

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.27-rc3
# Fri Aug 22 18:57:55 2008
#
# CONFIG_PPC64 is not set

#
# Processor support
#
CONFIG_6xx=y
# CONFIG_PPC_85xx is not set
# CONFIG_PPC_8xx is not set
# CONFIG_40x is not set
# CONFIG_44x is not set
# CONFIG_E200 is not set
CONFIG_PPC_FPU=y

2207 more lines

Hi,

Michael Buesch [off-list ref] writes:

I am seeing random kernel and userland application
crashes on a Powerbook running a 2.6.27-rc3 based kernel
(wireless-testing.git). 

The crashes did recently appear. It might be the case that they were
introduced with the merge of 2.6.27-rc1 into wireless-testing.
I'm not sure on that one, however. Just a guess. I still need to
do more testing (also on vanilla upstream kernels).

The crashes are completely random and they look like bad hardware.
However I cannot reproduce on 2.6.25.9 (That's a kernel I still had
installed, so I tried that one). So it most likely is _not_ caused
by faulty hardware.

The crashes are hard to reproduce, and happen about every 20 minutes
when compiling a kernel tree. (gcc segfaults). Sometimes the kernel
oopses in random places with pointer dereference faults.

Exact same thing here on a PB 12" (not a ppc64, but ppc32). It was not
on a wireless-testing.git but on a recent net-2.6 or 2.6.27-rc3 (I do
not remember precisely). I thought it was a thermal issue and did not
spend time on it because of another regression I bisected (sysctl+IPv6)
that version useless for me.

Is this a known issue?

Now it is ;-)

I'm going to bisect this one, but it will take a lot of time, as
reproducing takes about 20 minutes. So that's about an hour for one
test round.

Thanks for doing that.

Cheers,

a+

On Sat, 2008-08-23 at 16:10 +0200, Michael Buesch wrote:

I am seeing random kernel and userland application
crashes on a Powerbook running a 2.6.27-rc3 based kernel (wireless-testing.git).

The crashes did recently appear. It might be the case that they were
introduced with the merge of 2.6.27-rc1 into wireless-testing.
I'm not sure on that one, however. Just a guess. I still need to
do more testing (also on vanilla upstream kernels).

The crashes are completely random and they look like bad hardware.
However I cannot reproduce on 2.6.25.9 (That's a kernel I still had
installed, so I tried that one). So it most likely is _not_ caused
by faulty hardware.

The crashes are hard to reproduce, and happen about every 20 minutes
when compiling a kernel tree. (gcc segfaults). Sometimes the kernel
oopses in random places with pointer dereference faults.

Is this a known issue?
I'm going to bisect this one, but it will take a lot of time, as reproducing
takes about 20 minutes. So that's about an hour for one test round.

The kernel configuration is the following:

Random guess:

CONFIG_FRAME_POINTER=y
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y

Note sure what those together do, check if you have any file compiled
with -fno-omit-frame-pointer and if you do, try to change things so
that you don't ... we found some miscompiles when that is set, exposed
by FTRACE typically (which you don't have enabled) but possibly by other
things.

Ben.

On Sunday 24 August 2008 00:52:49 Benjamin Herrenschmidt wrote:

On Sat, 2008-08-23 at 16:10 +0200, Michael Buesch wrote:

quoted

I am seeing random kernel and userland application
crashes on a Powerbook running a 2.6.27-rc3 based kernel (wireless-testing.git).

The crashes did recently appear. It might be the case that they were
introduced with the merge of 2.6.27-rc1 into wireless-testing.
I'm not sure on that one, however. Just a guess. I still need to
do more testing (also on vanilla upstream kernels).

The crashes are completely random and they look like bad hardware.
However I cannot reproduce on 2.6.25.9 (That's a kernel I still had
installed, so I tried that one). So it most likely is _not_ caused
by faulty hardware.

The crashes are hard to reproduce, and happen about every 20 minutes
when compiling a kernel tree. (gcc segfaults). Sometimes the kernel
oopses in random places with pointer dereference faults.

Is this a known issue?
I'm going to bisect this one, but it will take a lot of time, as reproducing
takes about 20 minutes. So that's about an hour for one test round.

The kernel configuration is the following:

Random guess:

CONFIG_FRAME_POINTER=y
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y

Note sure what those together do, check if you have any file compiled
with -fno-omit-frame-pointer and if you do, try to change things so
that you don't ... we found some miscompiles when that is set, exposed
by FTRACE typically (which you don't have enabled) but possibly by other
things.

Ok, thanks for the suggestion.
I could reproduce the crash with 2.6.26, so this is not a regression between
2.6.26 and 2.6.27-rcX.
I'm currently running longer tests on 2.6.25 again to make sure it really
isn't hardware related.


NO_NO_OMIT is a brain screwer, btw :)

-- 
Greetings Michael.

On Sunday 24 August 2008, Benjamin Herrenschmidt wrote:

Random guess:

CONFIG_FRAME_POINTER=y
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y

Note sure what those together do, check if you have any file compiled
with -fno-omit-frame-pointer and if you do, try to change things so
that you don't ... we found some miscompiles when that is set, exposed
by FTRACE typically (which you don't have enabled) but possibly by other
things.

Thanks for your random guess.
The following workaround seems to fix the crashes on powerpc.
However, this patch is clearly not what we want for other architectures,
as they might need -fno-omit-frame-pointer to function properly.

I reproduced the random crashes of kernel and userspace applications
(without the following patch) on a vanilla 2.6.26 and 2.6.27-rc{1-4}
kernel. I did _not_ try a 2.6.25 kernel with -fno-omit-frame-pointer, so
I don't know if it would also crash then.

I'm currently running more tests on a patched 2.6.27-rc4 kernel, but it
didn't crash, yet. I already did 5 complete kernel tree compilations. It
should have crashed by now, but it didn't :)

The compiler is:
gcc (GCC) 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)


Index: linux-2.6/Makefile
===================================================================

--- linux-2.6.orig/Makefile	2008-08-24 11:49:53.000000000 +0200
+++ linux-2.6/Makefile	2008-08-24 12:16:42.000000000 +0200

@@ -523,13 +523,13 @@ endif
 
 # Force gcc to behave correct even for buggy distributions
 # Arch Makefiles may override this setting
 KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
 
 ifdef CONFIG_FRAME_POINTER
-KBUILD_CFLAGS	+= -fno-omit-frame-pointer -fno-optimize-sibling-calls
+KBUILD_CFLAGS	+= -fno-optimize-sibling-calls
 else
 KBUILD_CFLAGS	+= -fomit-frame-pointer
 endif
 
 ifdef CONFIG_DEBUG_INFO
 KBUILD_CFLAGS	+= -g

Index: linux-2.6/kernel/Makefile
===================================================================

--- linux-2.6.orig/kernel/Makefile	2008-08-24 11:50:23.000000000 +0200
+++ linux-2.6/kernel/Makefile	2008-08-24 12:15:54.000000000 +0200

@@ -92,13 +92,13 @@ obj-$(CONFIG_SMP) += sched_cpupri.o
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <alan@linuxcare.com.au>, the -fno-omit-frame-pointer is
 # needed for x86 only.  Why this used to be enabled for all architectures is beyond
 # me.  I suspect most platforms don't need this, but until we know that for sure
 # I turn this off for IA-64 only.  Andreas Schwab says it's also needed on m68k
 # to get a correct value for the wait-channel (WCHAN in ps). --davidm
-CFLAGS_sched.o := $(PROFILING) -fno-omit-frame-pointer
+CFLAGS_sched.o := $(PROFILING)
 endif
 
 $(obj)/configs.o: $(obj)/config_data.h
 
 # config_data.h contains the same information as ikconfig.h but gzipped.
 # Info from config_data can be extracted from /proc/config*

Thanks for your random guess.
The following workaround seems to fix the crashes on powerpc.
However, this patch is clearly not what we want for other architectures,
as they might need -fno-omit-frame-pointer to function properly.

Well, and -pg requires it, even on powerpc, so that won't work for
ftrace. Any chance you can try the workaround that segher proposed
though ?

http://penguinppc.de/~segher/0001-powerpc-Workaround-for-the-ftrace-problem.patch

His workaround only kicks in with CONFIG_FTRACE, that would have to be
fixed of course. Also, I suspect the bits that have -pg in a flag
"remove" section should have also "fno-omit-frame-pointer" in that
remove section too.

I reproduced the random crashes of kernel and userspace applications
(without the following patch) on a vanilla 2.6.26 and 2.6.27-rc{1-4}
kernel. I did _not_ try a 2.6.25 kernel with -fno-omit-frame-pointer, so
I don't know if it would also crash then.

I'm currently running more tests on a patched 2.6.27-rc4 kernel, but it
didn't crash, yet. I already did 5 complete kernel tree compilations. It
should have crashed by now, but it didn't :)

Thanks !

Ben.

This bug is causing random crashes
(http://bugzilla.kernel.org/show_bug.cgi?id=11414).  -fomit-frame-pointer is
only needed on powerpc when -pg is also supplied.  This patch ensures that
CONFIG_FRAME_POINTER is only selected by ftrace.  When CONFIG_FTRACE is enabled
we also pass -mno-sched-epilog to work around the codegen bug

Patch based on work by:
	Andreas Schwab [off-list ref]
	Segher Boessenkool [off-list ref]

Signed-off-by: Tony Breeds <redacted>
---
 arch/powerpc/Makefile                    |    5 +++++
 arch/powerpc/kernel/Makefile             |    7 ++++---
 arch/powerpc/platforms/powermac/Makefile |    2 +-
 lib/Kconfig.debug                        |    6 +++---
 4 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
index 9155c93..c6be19e 100644
--- a/arch/powerpc/Makefile
+++ b/arch/powerpc/Makefile

@@ -116,6 +116,11 @@ ifeq ($(CONFIG_6xx),y)
 KBUILD_CFLAGS		+= -mcpu=powerpc
 endif
 
+# Work around a gcc code-gen bug with -fno-omit-frame-pointer.
+ifeq ($(CONFIG_FTRACE),y)
+KBUILD_CFLAGS		+= -mno-sched-epilog
+endif
+
 cpu-as-$(CONFIG_4xx)		+= -Wa,-m405
 cpu-as-$(CONFIG_6xx)		+= -Wa,-maltivec
 cpu-as-$(CONFIG_POWER4)		+= -Wa,-maltivec

diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
index 64f5948..946daea 100644
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile

@@ -14,12 +14,13 @@ endif
 
 ifdef CONFIG_FTRACE
 # Do not trace early boot code
-CFLAGS_REMOVE_cputable.o = -pg
-CFLAGS_REMOVE_prom_init.o = -pg
+CFLAGS_REMOVE_cputable.o = -pg -mno-sched-epilog
+CFLAGS_REMOVE_prom_init.o = -pg -mno-sched-epilog
+CFLAGS_REMOVE_btext.o = -pg -mno-sched-epilog
 
 ifdef CONFIG_DYNAMIC_FTRACE
 # dynamic ftrace setup.
-CFLAGS_REMOVE_ftrace.o = -pg
+CFLAGS_REMOVE_ftrace.o = -pg -mno-sched-epilog
 endif
 
 endif

diff --git a/arch/powerpc/platforms/powermac/Makefile b/arch/powerpc/platforms/powermac/Makefile
index 58ecdd7..be60d64 100644
--- a/arch/powerpc/platforms/powermac/Makefile
+++ b/arch/powerpc/platforms/powermac/Makefile

@@ -2,7 +2,7 @@ CFLAGS_bootx_init.o  		+= -fPIC
 
 ifdef CONFIG_FTRACE
 # Do not trace early boot code
-CFLAGS_REMOVE_bootx_init.o = -pg
+CFLAGS_REMOVE_bootx_init.o = -pg -mno-sched-epilog
 endif
 
 obj-y				+= pic.o setup.o time.o feature.o pci.o \

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 800ac84..413eaca 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug

@@ -394,7 +394,7 @@ config LOCKDEP
 	bool
 	depends on DEBUG_KERNEL && TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
 	select STACKTRACE
-	select FRAME_POINTER if !X86 && !MIPS
+	select FRAME_POINTER if !X86 && !MIPS && !PPC
 	select KALLSYMS
 	select KALLSYMS_ALL
 

@@ -676,13 +676,13 @@ config FAULT_INJECTION_STACKTRACE_FILTER
 	depends on FAULT_INJECTION_DEBUG_FS && STACKTRACE_SUPPORT
 	depends on !X86_64
 	select STACKTRACE
-	select FRAME_POINTER
+	select FRAME_POINTER if !PPC
 	help
 	  Provide stacktrace filter for fault-injection capabilities
 
 config LATENCYTOP
 	bool "Latency measuring infrastructure"
-	select FRAME_POINTER if !MIPS
+	select FRAME_POINTER if !MIPS && !PPC
 	select KALLSYMS
 	select KALLSYMS_ALL
 	select STACKTRACE

-- 
1.5.6.5