Kernel Core Dump Capture
The instructions below are based on Novell KB 3374462. Please read this document before proceeding to become familiar with the procedure and what we're trying to achieve.
1. Serial Console Logging
Make sure that serial console logging is enabled as per https://supportforums.cisco.com/docs/DOC-28774.
2. Backup Files
Make sure you backup files that will be modified by the procedures belows in case you need to revert your changes.
# cp /etc/sysconfig/sysctl /etc/sysconfig/sysctl.nokdump # cp /boot/grub/menu.lst /boot/grub/menu.lst.nokdump
# cp /etc/sysconfig/kdump /etc/sysconfig/kdum.nokdump
# cp /etc/fstab /etc/fstab.nokdump <- in case you're using NFS mounts to save the core dumps
3. Enable Magic SysRQ
Edit /etc/sysconfig/sysctl using your favorite editor (e.g. vi).
Change the ENABLE_SYSRQ configuration to
ENABLE_SYSRQ="yes"
The system now either needs to be reboot for the setting above to take effect. Alternatively, you can issue the following command:
# echo 1 > /proc/sys/kernel/sysrq
3. Check Packages
Confirm that kdump, kexec-tools, and makedumpfile are installed.
# rpm -qa | grep kdump
# rpm -qa | grep kexec-tools
# rpm -qa | grep makedumpfile
4. Configure Kdump
Open YaST2
# yast2
Go to System > Kernel Kdump
Start-Up
Under Start-Up change Enable/Disable Kdump to Enable Kdump and the Kdump Memory [MB] to 768MB.
Dump Filtering
Go to Dump Filtering and deselect Free Pages in order to decrease the vmcore file.
The corresponding setting in /etc/sysconfig/kdump is KDUMP_DUMPLEVEL="16".
Dump Target
By default Kdump will dump the files in /var/crash (see figure below). If there's enough space on the local partition to accomodate the core dumps, leave this unchanged.
In our lab we have an NFS share where we save all the core dumps and this NFS share gets mounted automatically during boot time (/etc/fstab).
One thing we noticed when using NFS shares is that if you simply type the mount location in "Directory for Saving Dumps," (e.g. /mnt/kdump), kdump tries to save the files to /root/mnt/kdump which is not the right location. We had to choose Browse and browse to /mnt/kdump.
The corresponding setting in /etc/sysconfig/kdump is KDUMP_SAVEDIR="file:///mnt/kdump".
Press F10 to save the changes to exit out of YaST2. YaST2 will automatically modify the default entry in /boot/grub/menu.lst file to include the crashkernel memory reservation defined earlier. You can verify it by looking at the menu.lst file.
cishanar01:/root # cat /boot/grub/menu.lst
# Modified by YaST2. Last modification on Fri Dec 28 13:33:13 EST 2012
default 0
timeout 8
#gfxmenu (hd0,0)/message
##YaST - activate
serial --unit=0 --speed=115200
terminal --timeout=8 serial console
###Don't change this comment - YaST2 identifier: Original name: linux###
title Serial -- SUSE Linux Enterprise Server 11 SP1 - 2.6.32.12-0.7
kernel (hd1,0)/vmlinuz-2.6.32.12-0.7-default root=/dev/rootvg/lv_root resume=/dev/rootvg/lv_swap splash=silent crashkernel=1536M-:768M showopts console=tty0 console=ttyS0,115200
initrd (hd1,0)/initrd-2.6.32.12-0.7-default
###Don't change this comment - YaST2 identifier: Original name: linux###
title SUSE Linux Enterprise Server 11 SP1 - 2.6.32.12-0.7
kernel (hd1,0)/vmlinuz-2.6.32.12-0.7-default root=/dev/rootvg/lv_root resume=/dev/rootvg/lv_swap splash=silent crashkernel=256M-:128M showopts console=ttyS0,115200
initrd (hd1,0)/initrd-2.6.32.12-0.7-default
###Don't change this comment - YaST2 identifier: Original name: failsafe###
title Failsafe -- SUSE Linux Enterprise Server 11 SP1 - 2.6.32.12-0.7
kernel (hd1,0)/vmlinuz-2.6.32.12-0.7-default root=/dev/rootvg/lv_root showopts ide=nodma apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset x11failsafe console=ttyS0,115200
initrd (hd1,0)/initrd-2.6.32.12-0.7-default
5. Activate Kdump
# chkconfig boot.kdump on
6. Reboot the system
# shutdown -r now
Check that kdump is enabled (on):
# chkconfig boot.kdumpboot.kdump on
7. Reproduce Issue and Collect Dump
Once the system experiences a kernel panic or kernel oops, SysRQ should trigger automatically to boot into the crash kernel using kexec and dump the core. If for some reason you would like to manually trigger a crash dump, you can define a keyboard sequence from UCSM:
Go to UCSM > right-click on the blade > KVM console > User Defined Macro > Manage.
Click on New > Create the "magic SysRq" keyboard combination "Print Screen / SysRq + C".
If this macro is issued, a crash will occur. You can also trigger a crash from the command line:
# sync
# echo c > /proc/sysrq-trigger
8. Kdump Not Working
a) In the lab we found that kdump would not work if the crash kernel memory reservation was set higher than 1024MB. If 768MB does not work for you, try lowering or increasing that value. If changing the value, we suggest doing it via YaST so that menu.lst gets updated automatically.
b) If using an NFS share, the share might not get mounted unless the crash kernel boots in runlevel 3. To achieve this, add the following line to /etc/sysconfig/kdump. The default value is 1.
KDUMP_RUNLEVEL="3"
c) Some kdump documents suggest adding maxcpus=1 to KDUMP_COMMANDLINE_APPEND if the kernel was built with SMP support.
cishanar01:~ # zcat /proc/config.gz | grep -i config_smp
CONFIG_SMP=y
Edit /etc/sysconfig/kdump:
KDUMP_COMMANDLINE_APPEND="maxcpus=1"
Notes
If you make changes to /etc/sysconfig/kdump, you need to rebuild /boot/initrd by running `service boot.kdump restart`.
Further Reading
