From: amit mehta <hidden> Date: 2011-08-17 09:34:44
I've a Linux box which is crashing under a particular test scenario.
To debug this issue I was thinking of using the crash (man 8 crash)
utility (available by default on recent releases of RHEL and SuSE) .
This tool requires the kernel crash dump (vmcore) and the debugging symbols.
On my machine, I don't see any crash under /var/crash, Hence please
let me know the
configuration changes which i need to do to enable my machine to dump
the kernel image
locally (do i need to apply diskdump patches and rebuild the kernel??) whenever
there is a kernel crash. Kdump seem to be another tool for this job
but that requires a lot of
setup. Is there a *shortcut* to enable kernel crash dump facility ?
The debugging symbols can be found here:
http://ftp.riken.go.jp/Linux/slc/slc6X/updates/x86_64/debug/http://ftp.riken.go.jp/Linux/slc/slc6X/updates/i386/debug/
-Amit
From: amit mehta <hidden> Date: 2011-08-17 09:42:22
On Wed, Aug 17, 2011 at 3:04 PM, amit mehta [off-list ref] wrote:
I've a Linux box which is crashing under a particular test scenario.
To debug this issue I was thinking of using the crash (man 8 crash)
utility (available by default on recent releases of RHEL and SuSE) .
This tool requires the kernel crash dump (vmcore) and the debugging symbols.
On my machine, I don't see any crash under /var/crash, Hence please
let me know the
Here
configuration changes which i need to do to enable my machine to dump
the kernel image
Here
It seems my gmail setup is randomly adding newline character when i
draft a mail.
Which i'm sure is annoying to those who'll read my mail on this
mailing list. Can you please
help me resolve this issue.
-Amit
hi Amit,
On Wed, Aug 17, 2011 at 3:04 PM, amit mehta [off-list ref] wrote:
I've a Linux box which is crashing under a particular test scenario.
To debug this issue I was thinking of using the crash (man 8 crash)
utility (available by default on recent releases of RHEL and SuSE) .
This tool requires the kernel crash dump (vmcore) and the debugging
symbols.
On my machine, I don't see any crash under /var/crash, Hence please
let me know the
configuration changes which i need to do to enable my machine to dump
the kernel image
locally (do i need to apply diskdump patches and rebuild the kernel??)
whenever
there is a kernel crash. Kdump seem to be another tool for this job
but that requires a lot of
setup. Is there a *shortcut* to enable kernel crash dump facility ?
The debugging symbols can be found here:
http://ftp.riken.go.jp/Linux/slc/slc6X/updates/x86_64/debug/http://ftp.riken.go.jp/Linux/slc/slc6X/updates/i386/debug/
Thank you Sumeet for the link, I've enabled kdump on my machine by
following the steps mentioned
on the link above(http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-kdump.html)
But it seems that there is some setup problem on my machine as
invoking "crash" is giving me an error.
see the logs below.
<<<snip from /etc/kdump.cfg>>>
#raw /dev/sda5
#ext4 /dev/sda2
#ext4 LABEL=/boot
#ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
#net my.server.com:/export/tmp
#net user at my.server.com
path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31
#core_collector cp --sparse=always
#link_delay 60
#kdump_post /var/crash/scripts/kdump-post.sh
#extra_bins /usr/bin/lftp
#disk_timeout 30
#extra_modules gfs2
#options modulename options
default halt
<<<snip from /etc/kdump.cfg>>>
[root at rhel6 boot]# mkinitrd /boot/initrd-2.6.32-71.el6.x86_64kdump.img
2.6.32-71.el6.x86_64
[root at rhel6 ~]# service kdump restart
Stopping kdump: [ OK ]
Your running kernel is using more than 70% of the amount of[WARNING]u
reserved for kdump, you should consider increasing your crashkernel
reservation
Starting kdump: [ OK ]
[root at rhel6 ~]# service kdump status
Kdump is operational
I hit the crash again and this time there was a vmcore under
/var/crash directory
[root at rhel6 2011-08-17-17:50]# pwd
/var/crash/2011-08-17-17:50
[root at rhel6 2011-08-17-17:50]# du -sh vmcore
43M vmcore
These are the two debug rpms that i've installed for kernel symbols:
[root at rhel6 ~]# rpm -qa|grep -i debug
kernel-debuginfo-common-x86_64-2.6.32-71.7.1.el6.x86_64
kernel-debug-debuginfo-2.6.32-71.7.1.el6.x86_64
my production kernel, from which i've booted:
[root at rhel6 ~]# uname -r
2.6.32-71.el6.x86_64
[root at rhel6 ~]# crash -s
/usr/lib/debug/lib/modules/2.6.32-71.7.1.el6.x86_64.debug/vmlinux
/var/crash/2011-08-17-17\:50/vmcore
crash: invalid kernel virtual address: 7180 type: "possible"
WARNING: cannot read cpu_possible_map < - ----------------------------
crash: seek error: kernel virtual address: ffffffff8208e980 type: "xtime"
-Amit
On Wed, Aug 17, 2011 at 6:16 PM, amit mehta [off-list ref] wrote:
quoted
You need to configure kdump on your RHEL system to enable it to dump
the
quoted
vmcore file when the panic happens... Kdump configuration is explained
below.. There is a crashkernel boot parameter to be passed to kernel to
specify the memory reserved for the crash kernel.
Thank you Sumeet for the link, I've enabled kdump on my machine by
following the steps mentioned
on the link above(
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-kdump.html
)
But it seems that there is some setup problem on my machine as
invoking "crash" is giving me an error.
see the logs below.
<<<snip from /etc/kdump.cfg>>>
#raw /dev/sda5
#ext4 /dev/sda2
#ext4 LABEL=/boot
#ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
#net my.server.com:/export/tmp
#net user at my.server.com
path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31
#core_collector cp --sparse=always
#link_delay 60
#kdump_post /var/crash/scripts/kdump-post.sh
#extra_bins /usr/bin/lftp
#disk_timeout 30
#extra_modules gfs2
#options modulename options
default halt
<<<snip from /etc/kdump.cfg>>>
[root at rhel6 boot]# mkinitrd /boot/initrd-2.6.32-71.el6.x86_64kdump.img
2.6.32-71.el6.x86_64
[root at rhel6 ~]# service kdump restart
Stopping kdump: [ OK ]
Your running kernel is using more than 70% of the amount of[WARNING]u
reserved for kdump, you should consider increasing your crashkernel
reservation
Starting kdump: [ OK ]
[root at rhel6 ~]# service kdump status
Kdump is operational
I hit the crash again and this time there was a vmcore under
/var/crash directory
[root at rhel6 2011-08-17-17:50]# pwd
/var/crash/2011-08-17-17:50
[root at rhel6 2011-08-17-17:50]# du -sh vmcore
43M vmcore
These are the two debug rpms that i've installed for kernel symbols:
[root at rhel6 ~]# rpm -qa|grep -i debug
kernel-debuginfo-common-x86_64-2.6.32-71.7.1.el6.x86_64
kernel-debug-debuginfo-2.6.32-71.7.1.el6.x86_64
my production kernel, from which i've booted:
[root at rhel6 ~]# uname -r
2.6.32-71.el6.x86_64
[root at rhel6 ~]# crash -s
/usr/lib/debug/lib/modules/2.6.32-71.7.1.el6.x86_64.debug/vmlinux
/var/crash/2011-08-17-17\:50/vmcore
crash: invalid kernel virtual address: 7180 type: "possible"
WARNING: cannot read cpu_possible_map < - ----------------------------
crash: seek error: kernel virtual address: ffffffff8208e980 type: "xtime"
Hi Amit,
It seems there is a bugzilla from Red Hat on this issue(680864) in RHEL
6.0 kernel. Please could you try this issue on RHEL 6.1 kernel ?
Also as mentioned in the https://bugzilla.redhat.com/show_bug.cgi?id=680864,
which is similar to your issue, you should be able to see the output of bt
command and log command in the crash sesion. Please have a look at and let
us know what you see.
Regards,
Sumeet
From: Leonid V. Fedorenchik <hidden> Date: 2011-08-18 07:55:29
On Wed, 17 Aug 2011 15:12:22 +0530
amit mehta [off-list ref] wrote:
On Wed, Aug 17, 2011 at 3:04 PM, amit mehta [off-list ref] wrote:
quoted
I've a Linux box which is crashing under a particular test scenario.
To debug this issue I was thinking of using the crash (man 8 crash)
utility (available by default on recent releases of RHEL and SuSE) .
This tool requires the kernel crash dump (vmcore) and the debugging symbols.
On my machine, I don't see any crash under /var/crash, Hence please
let me know the
Here
quoted
configuration changes which i need to do to enable my machine to dump
the kernel image
Here
quoted
It seems my gmail setup is randomly adding newline character when i
draft a mail.
Which i'm sure is annoying to those who'll read my mail on this
mailing list. Can you please
help me resolve this issue.
Read Documentation/email-clients.txt
Hope this helps.
From: amit mehta <hidden> Date: 2011-08-18 08:33:52
? It seems there is a bugzilla from Red Hat on this issue(680864) in RHEL
6.0 kernel. Please could you try this issue on RHEL 6.1 kernel ?
?Also as mentioned in the
https://bugzilla.redhat.com/show_bug.cgi?id=680864, which is similar to your
issue, you should be able to see the output of bt command and log command in
the crash sesion. Please have a look at and let us know what you see.
Regards,
Sumeet
It seems that it was a memory issue(not enough memory) to enable crash dump.
Today, after migrating my machine to RHEL 6.1 and bumping up the memory,
i was able to get the kernel crash dump and could analyze it using crash
utility.
Thanks again for your help.
<<<snip from sample forced crash>>
crash> bt
PID: 19208 TASK: ffff8801540200c0 CPU: 3 COMMAND: "bash"
#0 [ffff88003778d9e0] machine_kexec at ffffffff810310db
#1 [ffff88003778da40] kimage_free_pages at ffffffff810b63b2
#2 [ffff88003778dac8] hvc_write_room at ffffffff8131abc6
#3 [ffff88003778db10] setjmp_pre_handler at ffffffff814dec50
#4 [ffff88003778db40] no_context at ffffffff81040cdb
#5 [ffff88003778db90] __bad_area_nosemaphore at ffffffff81040f65
#6 [ffff88003778dbe0] bad_area_access_error at ffffffff8104108e
#7 [ffff88003778dc10] __do_page_fault at ffffffff810417b3
#8 [ffff88003778dd30] __unregister_kprobe_top at ffffffff814e0c3e
#9 [ffff88003778dd60] oops_end at ffffffff814ddfe5
#10 [ffff88003778dde8] hvc_write_room at ffffffff8131abc6
#11 [ffff88003778de20] destroy_hvc_struct at ffffffff8131ae82
#12 [ffff88003778de70] hvc_hangup at ffffffff8131af3e
#13 [ffff88003778dea0] proc_reg_unlocked_ioctl at ffffffff811d4ffe
#14 [ffff88003778def0] default_llseek at ffffffff81172718
#15 [ffff88003778df00] destroy_watch at ffffffff810d1b62
#16 [ffff88003778df30] sys_pread64 at ffffffff81173151
RIP: 0000003b158d95e0 RSP: 00007fff4e780008 RFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff8100b172 RCX: 00007fff4e780098
RDX: 0000000000000002 RSI: 00007f13cb5d7000 RDI: 0000000000000001
RBP: 00007f13cb5d7000 R8: 000000000000000a R9: 00007f13cb5c8700
R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000002
R13: 0000003b15b8b780 R14: 0000000000000002 R15: 0000003b15b8b780
ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b
<<<snip from sample forced crash>>
-Amit