Re: Fwd: -rt and PowerPC
From: Sven-Thorsten Dietrich <hidden>
Date: 2007-12-28 16:23:36
On Wed, 2007-12-26 at 09:53 +0530, Chirag Jog wrote:
* sven@thebigcorporation.com [off-list ref] [2007-12-24 13:00:30]: Hi,
Sorry for top-posting the last email. Have not learned how to inline on the Blackberry. Comments below.
quoted
Do you have DEBUG and LOCKDEP configured?CONFIG_PREEMPT_DEBUG is enabled but LOCKDEP is not.quoted
-----Original Message----- From: Sripathi Kodi <redacted> We are trying to get a ppc64 box booted with the -rt kernel. Tried the latest 2.6.24-rc5-rt1 kernel. Everything goes well [ kernel boots up etc] until the services start coming up. It takes a lot of time at Starting udev. After which either services take too long to start or we get SEGFAULT.
I have seen this exact behavior with lockdep enabled on earlier x86-SMP Kernels. I saw the issue predominantly on larger SMP machines, but it behaved much like a race. This was observed on 2.6.21 Kernels, I will try and verify whether .23 has the same behavior. We eventually bisected it down to CONFIG_PROVE_LOCKING, but I wasn't totally convinced whethere the root cause was in lockdep, or elsewhere.
quoted
This is easily reproducible. Also if we get a chance to login; simple commands like ls, vi etc take either too long or get SEGFAULT.
quoted
Booting the kernel with maxcpus=1 or turning off SMP, doesn't solve the problem.
Hmmm, so have you COMPILED-out SMP, or just booted with nosmp? It might help narrowing this down to compile a UP Kernel. Thanks Sven
quoted
Here is logs from one of the booting sequences: Linux version 2.6.24-rc5-rt1 (root@llm29) (gcc version 4.1.0 (SUSE Linux)) #1 SMP PREEMPT RT Thu Dec 20 13:41:45 IST 2007 [boot]0012 Setup Arch PPC64 nvram contains 16384 bytes Zone PFN ranges: DMA 0 -> 1024000 Normal 1024000 -> 1024000 Movable zone start PFN for each node early_node_map[1] active PFN ranges 0: 0 -> 1024000 [boot]0015 Setup Done Real-Time Preemption Support (C) 2004-2007 Ingo Molnar Built 1 zonelists in Node order, mobility grouping on. Total pages: 1000000 Policy zone: DMA Kernel command line: root=/dev/hda2 quiet sysrq=1 askmethod Starting udevd Creating devices Waiting for device /dev/hda2 to appear: ok rootfs: major=3 minor=2 devn=770 fsck 1.38 (30-Jun-2005) [/bin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/hda2 /dev/hda2: clean, 249426/1311552 files, 1860112/2622611 blocks fsck succeeded. Mounting root device read-write. Mounting root /dev/hda2 INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting procfs at /proc done Mounting sysfs at /sys done Mounting debugfs at /sys/kernel/debug done Initializing /dev done Mounting devpts at /dev/pts done Activating swap-devices in /etc/fstab... done showconsole: Warning: the ioctl TIOCGDEV is not known by the kernel Starting udevd udevd-event[2463]: run_program: '/bin/sh' abnormal exit udevd-event[2474]: run_program: '/sbin/vol_id' abnormal exit udevd-event[2477]: run_program: '/sbin/vol_id' abnormal exit udevd-event[2455]: run_program: '/sbin/ifup' abnormal exit udevd-event[2477]: run_program: '/lib/udev/mount.sh' abnormal exit
quoted
failed Loading required kernel modules done Activating device mapper... FATAL: Module dm_mod not found. failed Starting MD Raid done Waiting for udev to settle... Scanning for LVM volume groups... Reading all physical volumes. This may take a while... No volume groups found No volume groups found Activating LVM volume groups... No volume groups found done showconsole: Warning: the ioctl TIOCGDEV is not known by the kernel Checking file systems... fsck 1.38 (30-Jun-2005) Checking all file systems. [/sbin/fsck.ext3 (1) -- /home] fsck.ext3 -a /dev/hda5 /dev/hda5: clean, 460612/1836928 files, 2774880/3670844 blocks [/sbin/fsck.ext3 (1) -- /home/mohan/mnt] fsck.ext3 -a /dev/hda3 /dev/hda3: clean, 362719/1311552 files, 1651440/2622611 blocks *** glibc detected *** /bin/sh: free(): invalid pointer: 0x100a5170 *** ======= Backtrace: ========= /lib/power4/libc.so.6[0xfde4be4] /lib/p/lib/apparmor/rc.apparmor.functions: line 87: 2603 Segmentation fault grep -q securityfs /proc/filesystems /lib/apparmor/rc.apparmor.functions: line 355: 2606 Aborted grep -qE "^(subdomain|apparmor)[[:space:]]" /proc/modules /lib/apparmor/rc.apparmor.functions: line 318: 2610 Segmentation fault modinfo -F filename subdomain >/dev/null 2>&1 /lib/apparmor/rc.apparmor.functions: line 318: 2613 Segmentation fault grep -qE "^(subdomain|apparmor)[[:space:]]" /proc/modules FATAL: Module apparmor not found. Loading AppArmor module failed - could not start AppArmorfailed /etc/init.d/boot.cleanup: line 23: 2601 Segmentation fault rm -f /var/lib/rpm/__db* /etc/init.d/boot.cleanup: line 23: 2602 Segmentation fault rm -rf /tmp/screens /tmp/uscreens /var/run/screens /var/run/uscreens 2>/dev/null /etc/init.d/boot.cleanup: line 23: 2605 Segmentation fault rm -f /tmp/.X*lock /var/spool/uucp/LCK* /var/log/sa/sadc.LOCK /fsck_corrected_errors 2>/dev/null /etc/init.d/boot.cleanup: line 23: 2608 Segmentation fault find /tmp/ssh-* -type s -name "*agent*" -maxdepth 1 -print0 2>/dev/null 2609 | xargs -0 -r rm -f /etc/init.d/boot.cleanup: line 23: 2611 Segmentation fault find /var/run /var/lock -type f -print0 2612 | xargs -0 -r rm -f 2>/dev/null /etc/init.d/boot.cleanup: line 23: 2615 Segmentation fault chmod 664 /var/run/utmp /etc/init.d/boot.cleanup: line 23: 2616 Segmentation fault chown root:tty /var/run/utmp /etc/init.d/boot.cleanup: line 23: 2617 Segmentation fault ls /etc/resolv.conf.saved.by.* >&/dev/null /etc/init.d/boot.cleanup: line 75: 2618 Segmentation fault chown root:root $CURDIR /etc/init.d/boot^[[6~.cleanup: line 75: 2619 Segmentation fault chown root:root $CURDIR /etc/init.d/boot.cleanup: line 75: 2620 Segmentation fault chown root:root $CURDIR /etc/init.d/boot.cleanup: line 75: 2621 Segmentation fault chown root:root $CURDIR /etc/init.d/boot.cleanup: line 75: 2622 Segmentation fault chown root:root $CURDIR /etc/init.d/boot.cleanup: line 75: 2623 Segmentation fault c^[[6~hown root:root $CURDIR /etc/init.d/boot.cleanup: line 82: 2624 Segmentation fault chown root:root $CURDIR /etc/init.d/boot.cleanup: line 23: 2625 Segmentation fault rm -f /etc/psdevtab /etc/init.d/boot.cleanup: line 23: 2626 Segmentation fault /bin/ps >/dev/null 2>/dev/null No such file or directory No such file or directory No such file or directory No such file or directory No such file or directory No such file or directory System Boot Control: The system has been set up Failed features: boot.udev boot.localfs boot.clock boot.klog boot.scpm boot.crypto boot.localnet boot.swap boot.udev_retry boot.ldconfig boot.sysctl boot.ipconfig System Boot Control: Running /etc/init.d/boot.local done INIT: Entering runlevel: 3 EXT3-fs error (device hda2): ext3_add_entry: bad entry in directory #971592: rec_len % 4 != 0 - offset=0, inode=9349464, rec_len=54811, name_len=129 blogd: can not open /var/run/blogd.pid: Input/output error Boot logging started on /dev/hvc0(/dev/console) at Thu Dec 20 14:19:34 2007 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Master Resource Control: runlevel 3 has been reached /etc/init.d/rc: line 383: 2669 Illegal instruction killproc -QUIT /sbin/blogd Segmentation fault /bin/bash: error while loading shared libraries: �C�x��H: cannot open shared object file: No such file or directory INIT: Id "cons" respawning too fast: disabled for 5 minutes CPU Info: # cat /proc/cpuinfo processor : 0 cpu : PPC970, altivec supported clock : 1600.000000MHz revision : 2.2 (pvr 0039 0202) processor : 1 cpu : PPC970, altivec supported clock : 1600.000000MHz revision : 2.2 (pvr 0039 0202) timebase : 199836440 platform : pSeries machine : CHRP IBM,8842-21X Also Attached is the config file: -- Cheers, Chirag Jog -- Cheers, Chirag Jog -------------------------------------------------------