Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

SAP HANA - Kernel Core Dump Capture

Kernel Core Dump Capture

The instructions below are based on Novell KB 3374462. Please read this document before proceeding to become familiar with the procedure and what we're trying to achieve.

1. Serial Console Logging

Make sure that serial console logging is enabled as per https://supportforums.cisco.com/docs/DOC-28774.

2. Backup Files

Make sure you backup files that will be modified by the procedures belows in case you need to revert your changes.

# cp /etc/sysconfig/sysctl /etc/sysconfig/sysctl.nokdump # cp /boot/grub/menu.lst /boot/grub/menu.lst.nokdump

# cp /etc/sysconfig/kdump /etc/sysconfig/kdum.nokdump

# cp /etc/fstab /etc/fstab.nokdump <- in case you're using NFS mounts to save the core dumps

3. Enable Magic SysRQ

Edit /etc/sysconfig/sysctl using your favorite editor (e.g. vi).

Change the ENABLE_SYSRQ configuration to

ENABLE_SYSRQ="yes"

The system now either needs to be reboot for the setting above to take effect. Alternatively, you can issue the following command:

# echo 1 > /proc/sys/kernel/sysrq

3. Check Packages

Confirm that kdump, kexec-tools, and makedumpfile are installed.

# rpm -qa | grep kdump

# rpm -qa | grep kexec-tools

# rpm -qa | grep makedumpfile

4. Configure Kdump

Open YaST2

# yast2

Go to System > Kernel Kdump

kdump1.PNG

Start-Up

Under Start-Up change Enable/Disable Kdump to Enable Kdump and the Kdump Memory [MB] to 768MB.

kdump2.PNG

Dump Filtering


Go to Dump Filtering and deselect Free Pages in order to decrease the vmcore file.

The corresponding setting in /etc/sysconfig/kdump is KDUMP_DUMPLEVEL="16".

kdump3.PNG

Dump Target

By default Kdump will dump the files in /var/crash (see figure below). If there's enough space on the local partition to accomodate the core dumps, leave this unchanged.

In our lab we have an NFS share where we save all the core dumps and this NFS share gets mounted automatically during boot time (/etc/fstab).

One thing we noticed when using NFS shares is that if you simply type the mount location in "Directory for Saving Dumps," (e.g. /mnt/kdump), kdump tries to save the files to /root/mnt/kdump which is not the right location. We had to choose Browse and browse to /mnt/kdump.

The corresponding setting in /etc/sysconfig/kdump is KDUMP_SAVEDIR="file:///mnt/kdump".

kdump4.PNG

Press F10 to save the changes to exit out of YaST2. YaST2 will automatically modify the default entry in /boot/grub/menu.lst file to include the crashkernel memory reservation defined earlier. You can verify it by looking at the menu.lst file.

cishanar01:/root # cat /boot/grub/menu.lst

# Modified by YaST2. Last modification on Fri Dec 28 13:33:13 EST 2012

default 0

timeout 8

#gfxmenu (hd0,0)/message

##YaST - activate

serial --unit=0 --speed=115200

terminal --timeout=8 serial console

###Don't change this comment - YaST2 identifier: Original name: linux###

title Serial -- SUSE Linux Enterprise Server 11 SP1 - 2.6.32.12-0.7

    kernel (hd1,0)/vmlinuz-2.6.32.12-0.7-default root=/dev/rootvg/lv_root resume=/dev/rootvg/lv_swap splash=silent crashkernel=1536M-:768M showopts console=tty0 console=ttyS0,115200

    initrd (hd1,0)/initrd-2.6.32.12-0.7-default

###Don't change this comment - YaST2 identifier: Original name: linux###

title SUSE Linux Enterprise Server 11 SP1 - 2.6.32.12-0.7

    kernel (hd1,0)/vmlinuz-2.6.32.12-0.7-default root=/dev/rootvg/lv_root resume=/dev/rootvg/lv_swap splash=silent crashkernel=256M-:128M showopts console=ttyS0,115200

    initrd (hd1,0)/initrd-2.6.32.12-0.7-default

###Don't change this comment - YaST2 identifier: Original name: failsafe###

title Failsafe -- SUSE Linux Enterprise Server 11 SP1 - 2.6.32.12-0.7

    kernel (hd1,0)/vmlinuz-2.6.32.12-0.7-default root=/dev/rootvg/lv_root showopts ide=nodma apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset x11failsafe console=ttyS0,115200

    initrd (hd1,0)/initrd-2.6.32.12-0.7-default

5. Activate Kdump

# chkconfig boot.kdump on

6. Reboot the system

# shutdown -r now

Check that kdump is enabled (on):

# chkconfig boot.kdump

boot.kdump  on

7. Reproduce Issue and Collect Dump

Once the system experiences a kernel panic or kernel oops, SysRQ should trigger automatically to boot into the crash kernel using kexec and dump the core. If for some reason you would like to manually trigger a crash dump, you can define a keyboard sequence from UCSM:

Go to UCSM > right-click on the blade > KVM console > User Defined Macro > Manage.

Click on New > Create the "magic SysRq" keyboard combination "Print Screen / SysRq + C".

kdump5.PNG

If this macro is issued, a crash will occur. You can also trigger a crash from the command line:

# sync

# echo c > /proc/sysrq-trigger

8. Kdump Not Working

a) In the lab we found that kdump would not work if the crash kernel memory reservation was set higher than 1024MB. If 768MB does not work for you, try lowering or increasing that value. If changing the value, we suggest doing it via YaST so that menu.lst gets updated automatically.

b) If using an NFS share, the share might not get mounted unless the  crash kernel boots in runlevel 3. To achieve this, add the following  line to /etc/sysconfig/kdump. The default value is 1.

KDUMP_RUNLEVEL="3"

c) Some kdump documents suggest adding maxcpus=1 to KDUMP_COMMANDLINE_APPEND if the kernel was built with SMP support.

cishanar01:~ # zcat /proc/config.gz | grep -i config_smp

CONFIG_SMP=y

Edit /etc/sysconfig/kdump:

KDUMP_COMMANDLINE_APPEND="maxcpus=1"

Notes

If you make changes to /etc/sysconfig/kdump, you need to rebuild /boot/initrd by running `service boot.kdump restart`.

Further Reading

http://www.dedoimedo.com/computers/kdump.html

http://www.dedoimedo.com/computers/kdump-opensuse.html

2687
Views
5
Helpful
0
Comments