ASA: Coredumpの収集について

Taisuke Nakamura · ‎2016-03-04

はじめに
Crashinfoと Coredumpの違い
Coredumpの有効化
Coredumpの生成と収集
Coredumpの無効化
よくある質問
参考情報

はじめに

本ドキュメントでは、クラッシュ時の原因調査に有効である Coredumpの有効化と収集方法について記述します。 Coredumpは、Crashinfoとは異なる情報の収集が可能です。

TACから取得依頼があった場合、本ドキュメントを参考に Coredump有効化と、以下情報の収集をお願いいたします。

Crashinfo : show crashinfoの出力
Coredumpファイル
シスログ : 発生時～その前 30分程度まで
show tech
想定されるトリガー : 特定の操作や通信が発生時にクラッシュする、など
再現手順 (※可能な限りお願いいたします。問題解決が大変早くなります)

Crashinfoと Coredumpの違い

ASAは異常が発生時に、機器が操作不能に陥る事を避けるためにも、システムを強制的に再起動する事で復旧を試みます。このクラッシュ時に、ASAはCrashinfoの生成を試みます。

Crashinfoには、問題の発生したスレッド名(e.g. telnet/ciや Datapathや SNMPなど)や、クラッシュ直前のプロセスやメモリ情報やログが格納されています。 Crashinfoは、ASA CLI Analyzerや、私達TACで解析する事で、既知不具合に該当するかの確認などができます。しかし、Crashinfoの情報のみでは、クラッシュの原因究明に至らないことがあります。また、新規不具合によるクラッシュの場合はCrashinfoの情報のみでは分析や詳細調査は困難です。

Coredumpを追加で取得いただく事で、さらに詳細な調査が可能となる事があります。 Coredumpには、クラッシュが発生時のメモリ上のASAコードのスナップショットとその関連情報が含まれています。

Coredumpの有効化

クラッシュ時にCoredumpの生成を試みるには、"coredump enable"コマンドで事前の有効化が必要です。 Coredumpを有効化時に Flash内に専用の領域が確保されます。当領域の作成速度の目安は、長くとも数分程度が目安です。 Coredumpを有効化後は、"write memory"コマンドで設定を保存します。

　　　　
以下は各機種のデフォルトで確保される領域の目安です。

ASA5505	60MB	ASA5512	1GB	ASA5585-10	300MB	ASA5506	300MB
ASA5510	60MB	ASA5515	1GB	ASA5585-20	600MB	ASA5508	60MB
ASA5520	60MB	ASA5525	1GB	ASA5585-40	600MB	ASA5516	300MB
ASA5540	100MB	ASA5545	1GB	ASA5585-60	1GB	-	-
ASA5550	200MB	ASA5555	1GB	-	-	-	-

　　　
以下はASA5520 9.1(7)4での、Coredump有効化例です。

ciscoasa# configure terminal
ciscoasa(config)#
ciscoasa(config)# coredump enable

WARNING: Enabling coredump on an ASA5520 platform will delay
the reload of the system in the event of software forced reload.
The exact time depends on the size of the coredump generated.

Proceed with coredump filesystem allocation of 60 MB
on 'disk0:' (Note this may take a while) ? [confirm]

Making coredump file system image!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Coredump file system image created & mounted successfully

/dev/loop0 on /mnt/disk0/coredumpfsys type vfat (rw,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1)

ciscoasa(config)#
ciscoasa(config)# write memory
Building configuration...
Cryptochecksum: 01e666fd 6c9cb47f 5e5e1d29 ba24907d
3913 bytes copied in 3.340 secs (1304 bytes/sec)
[OK]
ciscoasa(config)#

確保された領域と、Coredumpファイル生成有無は "show coredump filesystem"コマンドで確認できます。

ciscoasa(config)# show coredump filesystem
Coredump Filesystem Size is 60 MB
Filesystem type is FAT for disk0

Filesystem 1024-blocks Used Available Use% Mounted on
/dev/loop0 61302 0 61302 0% /mnt/disk0/coredumpfsys

Directory of disk0:/coredumpfsys/
No files in directory
255426560 bytes total (32735232 bytes free)

確保する領域は、"coredump enable filesystem disk0: size <SIZE>"コマンドで任意値に設定できます。以下は100MBの領域を作成時の出力例です。なお、Flashに十分な空きがある場合は、デフォルトサイズの利用をお奨めします。

ciscoasa(config)# coredump enable filesystem disk0: size ?

configure mode commands/options:
 <9-243> Specify coredump maximum filesystem allocation (in MB)
 default Let the system choose the maximum coredump filesystem allocation

ciscoasa(config)# coredump enable filesystem disk0: size 100

WARNING: Enabling coredump on an ASA5520 platform will delay
the reload of the system in the event of software forced reload.
The exact time depends on the size of the coredump generated.

Proceed with coredump filesystem allocation of 100 MB
on 'disk0:' (Note this may take a while) ? [confirm]

Making coredump file system image!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Coredump file system image created & mounted successfully

/dev/loop0 on /mnt/disk0/coredumpfsys type vfat (rw,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1)

ciscoasa(config)#

　　　　
冗長構成の場合、"coredump enable"コマンドは自動的に Standby機に同期され、Standby機でもCoredumpの収集が有効になります。

Coredumpの生成と収集

クラッシュ発生時、ASAは Crashinfoと Coredumpの生成を試みます。

Coredumpは圧縮されFlashに保存されますが、その処理には時間を要します。 50MBの場合数分(*)、100MBで10分程度(*)が目安となります(*実測値。) なお、利用機器のCPU性能や書き込まれる情報量により、生成に要する時間は変動します。

つまり、Coredumpの収集・生成に要した時間の分だけ、クラッシュ後の機器再起動に必要な時間が伸びる、という事になります。

　　　　
以下は、実際にASA5520でクラッシュが発生時のコンソールログです。 Crashinfoの生成がされた後、coredumpのFlashへの書き込み処理が実行されます。 CoredumpのFlash書き込みが完了後、システムはリスタートされます。

以下例の場合、おおよそ 10秒で Crashinfoの保存完了、その後おおよそ 2分弱で Coredumpの保存が完了しシステムの再起動に至っています。生成されたCoredumpファイルのサイズは約40MBです。

ciscoasa(config)#
     --- Crash発生 ---
Thread Name: ci/console <---- まずCrashinfoのDumpが出力
Page fault: Address not mapped
 vector 0x0000000e
 edi 0x00000001
 esi 0x6ee37a3c
 ebp 0x750919f0
 esp 0x750919d8
 ebx 0x00000005
 edx 0x00000000
 ecx 0x00000000
 eax 0x00000001
error code 0x00000006
 eip 0x0805db60
 cs 0x00000073
 eflags 0x00013246
 CR2 0x00000000

Cisco Adaptive Security Appliance Software Version 9.1(7)4

Compiled on Fri 19-Feb-16 13:00 by builders
Hardware: ASA5520
Crashinfo collected on 13:54:03.895 JST Fri Mar 4 2016

Traceback:
0: 0x08063660
1: 0x080636a1
2: 0x08065865
3: 0x08f00b62
4: 0xffffe410
5: 0x08c894e4
6: 0x08c8af81
7: 0x080f57cb
8: 0x080f7176
9: 0x080f8030
10: 0x0806a2ac
       --- snip ---
0x750919ec-0x750919e4: 0x00000000
0x750919e0: 0x00000003
0x750919dc: 0x00000001
0x750919d8: 0x0a7b8115 *
0x750919d4-0x750918f4: 0x00000000

Begin to dump crashinfo to flash.... <---- CrashinfoがFlashに保存

End of console dump.
Do 'show crashinfo' after reboot to retrieve other crash information
Process shutdown finished

Writing coredump file to flash. Please do not reload. <---- Coredumpの生成が開始

Coredump starting....
!!!!!!<4>Clocksource tsc unstable (delta = 87040029 ns)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Writing coredump file to flash. Please do not reload. (elapsed time: 0 Min 30 Secs)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Writing coredump file to flash. Please do not reload. (elapsed time: 1 Min 0 Secs)
!!!!!!!!!!
Writing coredump file to flash. Please do not reload. (elapsed time: 1 Min 30 Secs)
!!!!!!!!!!
Writing coredump file to flash. Please do not reload. (elapsed time: 2 Min 0 Secs)
!!!!!!!!!!!!!!!!!!
Coredump completed    <--- Coredumpの保存処理に2分弱かかったことを示す
Rebooting.....
Restarting system. <---- システムが自動で再起動 
machine restart


Booting system, please wait...

CISCO SYSTEMS
Embedded BIOS Version 1.0(11)4 03/21/08 17:09:54.41

Low Memory: 631 KB
High Memory: 2048 MB
PCI Device Table.
Bus Dev Func VendID DevID Class Irq
 00 00 00 8086 2578 Host Bridge
 00 01 00 8086 2579 PCI-to-PCI Bridge
 00 03 00 8086 257B PCI-to-PCI Bridge
 00 1C 00 8086 25AE PCI-to-PCI Bridge
 00 1D 00 8086 25A9 Serial Bus 11
 00 1D 01 8086 25AA Serial Bus 10
 00 1D 04 8086 25AB System
 00 1D 05 8086 25AC IRQ Controller
 00 1D 07 8086 25AD Serial Bus 9
 00 1E 00 8086 244E PCI-to-PCI Bridge
 00 1F 00 8086 25A1 ISA Bridge
 00 1F 02 8086 25A3 IDE Controller 11
 00 1F 03 8086 25A4 Serial Bus 5
 00 1F 05 8086 25A6 Audio 5
 02 01 00 8086 1075 Ethernet 11
 03 01 00 177D 0003 Encrypt/Decrypt 9
 03 02 00 8086 1079 Ethernet 9
 03 02 01 8086 1079 Ethernet 9
 03 03 00 8086 1079 Ethernet 9
 03 03 01 8086 1079 Ethernet 9
 04 02 00 8086 1209 Ethernet 11
 04 03 00 8086 1209 Ethernet 5

Evaluating BIOS Options ...
Launch BIOS Extension to setup ROMMON

Cisco Systems ROMMON Version (1.0(11)4) #0: Fri Mar 21 17:35:35 PDT 2008

Platform ASA5520

Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.

Launching BootLoader...
Boot configuration file contains 1 entry.

Loading disk0:/asa917-4-k8.bin... Booting...   <---- イメージ読み込み開始。起動開始中
Platform ASA5520

　　　　
生成されたCoredumpは "show coredump filesystem"コマンドで確認できます。

ciscoasa# show coredump filesystem
Coredump Filesystem Size is 60 MB

Filesystem type is FAT for disk0

Filesystem 1024-blocks Used Available Use% Mounted on
/dev/loop0 61302 36748 24554 60% /mnt/disk0/coredumpfsys

Directory of disk0:/coredumpfsys/

520 -rwx 37628299 13:56:22 Mar 04 2016 core_lina.2016Mar04_045417.514.11.gz   <---- THIS

255426560 bytes total (32473088 bytes free)

copyコマンドで Coredumpファイルを任意サーバにアップロードします。収集したCoredumpファイルは TACへ送付をお願いいたします。

ciscoasa# copy disk0:/coredumpfsys/core_lina.2016Mar04_045417.514.11.gz tftp

Source filename [/coredumpfsys/core_lina.2016Mar04_045417.514.11.gz]?

Address or name of remote host []? 10.xx.xx.xx

Destination filename [core_lina.2016Mar04_045417.514.11.gz]?
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Coredumpの無効化

Coredumpの収集が不要になりましたら、"clear configure coredump"コマンドを実行します。 Coredump filesystemとその情報を消去できます。無効化後は、"write memory"コマンドで設定を保存します。

ciscoasa(config)# show coredump filesystem

Coredump Filesystem Size is 60 MB

Filesystem type is FAT for disk0

Filesystem 1024-blocks Used Available Use% Mounted on
/dev/loop0 61302 0 61302 0% /mnt/disk0/coredumpfsys

Directory of disk0:/coredumpfsys/

No files in directory

255426560 bytes total (32735232 bytes free)

ciscoasa(config)#
ciscoasa(config)# clear configure coredump

Proceed with removing the coredump filesystem and it's contents on 'disk0:' [confirm]

Coredump filesystem and it's contents removed successfully
ciscoasa(config)#
ciscoasa(config)# show coredump filesystem

'disk0:' has no coredump filesystem

ciscoasa(config)#
ciscoasa(config)# write memory
Building configuration...
Cryptochecksum: 646f847e 25dc041b 1e367a89 da9e60bc
3871 bytes copied in 3.280 secs (1290 bytes/sec)
[OK]

　　　　
冗長構成の場合、"clear config coredump"コマンドは自動的に Standby機に同期され、Standby機でも Coredumpが無効になります。

よくある質問

Multiple Context Modeを利用時、どのようにCoredumpを収集しますか

システム実行スペース、もしくは Admin コンテキストの何れかから有効化と収集が可能です。

以下はASAバージョン 9.6(3)3の Active/Active構成での、システム実行スペースからの、実際の有効例と取得例です。

ASA/admin/sec/act(config)# changeto system
ASA/sec/act(config)#
ASA/sec/act(config)# coredump enable
WARNING: Enabling coredump on an ASA5515 platform will delay the reload of

the system by up to 30 minutes in the event of software forced reload.
The exact time depends on the size of the coredump generated.

Proceed with coredump filesystem allocation of 1000 MB
on 'disk0:' (Note this may take a while) ? [confirm]
filesys_image created ok: disk0:coredumpfsysimage.bin

Making coredump file system image
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Coredump file system image created & mounted successfully

/dev/loop0 on /mnt/disk0/coredumpfsys type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

ASA/sec/act(config)# 

<意図的にSecondary機のクラッシュを発生させる>

 Group 1 Detected Active mate

 Group 2 Detected Active mate

Waiting for the earlier webvpn instance to terminate...
Previous instance shut down. Starting a new one.
End configuration replication from mate.

ASA/sec/stby>
ASA/sec/stby> en      <---- Secondary機クラッシュ後 再度ログイン
Password:
ASA/sec/stby# show coredump filesystem
Coredump Filesystem Size is 1000 MB
Filesystem type is FAT for disk0

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/loop1 1023728 78240 945488 8% /mnt/disk0/coredumpfsys

Directory of disk0:/coredumpfsys/

282 -rwx 80112742 01:37:28 Jul 05 2017 core_smp.2017Jul05_013656.1926.6.gz <---- ★Coredumpファイル

8238202880 bytes total (1936642048 bytes free)

ASA/sec/stby#
ASA/sec/stby# show clock
01:45:36.987 UTC Wed Jul 5 2017

ASA/sec/stby#
ASA/sec/stby# copy disk0:/coredumpfsys/core_smp.2017Jul05_013656.1926.6.gz $

Source filename [/coredumpfsys/core_smp.2017Jul05_013656.1926.6.gz]?

Address or name of remote host []? xx.xx.xx.xx

Destination filename [core_smp.2017Jul05_013656.1926.6.gz]?
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
80112742 bytes copied in 8.950 secs (10014092 bytes/sec)

ASA/sec/stby#
ASA/sec/stby# show version | in Version
Cisco Adaptive Security Appliance Software Version 9.6(3)3 <system>
Device Manager Version 7.7(1)151
Baseboard Management Controller (revision 0x1) Firmware Version: 2.4

参考情報

Command Reference - coredump enable
http://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/A-H/cmdref1/c4.html#pgfId-2171158

Enabling Coredump on the ASA
https://supportforums.cisco.com/document/59021/enabling-coredump-asa

ASA: crashinfoの取得方法について
https://supportforums.cisco.com/ja/document/12795241

ASA CLI アナライザ - クイックスタートガイド
https://supportforums.cisco.com/ja/document/12880216

ファイアウォールトラブルシューティング
https://supportforums.cisco.com/ja/document/12725841

Firepower System and FTDトラブルシューティング
https://community.cisco.com/t5/-/-/ta-p/3161733