The ASA 5510 here is crashing frequently. We have been unable to find out why exactly this is happening, and the device is not under a service contract. The logging that is in place doesn't give any hints as to why this is happening. Since the last time we had problems, it was working fine for months, then it started crashing again every other day. When this happens, the device simply locks up and needs a power-cycle since console access is not working either.
Is this a know failure mode on this type of device, or with this version of the software? I can supply more detail if required.
ASA > show version
Cisco Adaptive Security Appliance Software Version 8.2(4)
Device Manager Version 6.3(5)
Compiled on Tue 14-Dec-10 12:00 by builders
System image file is "disk0:/asa824-k8.bin"
Config file at boot was "startup-config"
ASA up 20 hours 27 mins
Hardware: ASA5510, 256 MB RAM, CPU Pentium 4 Celeron 1599 MHz
Internal ATA Compact Flash, 64MB
BIOS Flash M50FW080 @ 0xffe00000, 1024KB
Encryption hardware device : Cisco ASA-55x0 on-board accelerator (revision 0x0)
Boot microcode : CN1000-MC-BOOT-2.00
SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03
IPSec microcode : CNlite-MC-IPSECm-MAIN-2.05
0: Ext: Ethernet0/0 : address is 0017.5a88.735c, irq 9
1: Ext: Ethernet0/1 : address is 0017.5a88.735d, irq 9
2: Ext: Ethernet0/2 : address is 0017.5a88.735e, irq 9
3: Ext: Ethernet0/3 : address is 0017.5a88.735f, irq 9
4: Ext: Management0/0 : address is 0017.5a88.7360, irq 11
5: Int: Internal-Data0/0 : address is 0000.0001.0002, irq 11
6: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 5
Licensed features for this platform:
Maximum Physical Interfaces : Unlimited
Maximum VLANs : 100
Inside Hosts : Unlimited
Failover : Active/Active
VPN-DES : Enabled
VPN-3DES-AES : Enabled
Security Contexts : 2
GTP/GPRS : Disabled
SSL VPN Peers : 2
Total VPN Peers : 250
Shared License : Disabled
AnyConnect for Mobile : Disabled
AnyConnect for Cisco VPN Phone : Disabled
AnyConnect Essentials : Disabled
Advanced Endpoint Assessment : Disabled
UC Phone Proxy Sessions : 2
Total UC Proxy Sessions : 2
Botnet Traffic Filter : Disabled
This platform has an ASA 5510 Security Plus license.
Serial Number: JMX1020K0R2
Running Activation Key: 0xcb091579 0xfccd86cb 0x5ce1f108 0xb24cc8fc 0x803dde88
Configuration register is 0x1
Configuration has not been modified since last system restart.
I guess you already missing 2 of the usual options I would probably take since you dont have a service contract. And those would be software upgrade to the lastest version in your current software level (8.2(5)) and/or opening a TAC case.
How often does the device crash?
Has the device written Crash Info to the Flash?
show crashinfo save
Is there anything special with the CPU and Memory utilization?
Is there perhaps some configuration change that was done at the same time when the crashes started? Maybe you have some older backup configurations that you could compare to the current running configurations? For example save both configurations to their own files and compare the difference of these configurations with MS Word or some other software.
I am afraid that I probably wont be able to provide any help with this other than trying to think of place where to look.
I wasnt able to find any bug listed for 8.2(4) software that would cause crashes or I just didnt search with the correct keywords.
The device crashed/hung several times in March/April this year. Then it was fine for months and hung again on the 28th July, 26th August, 27th August, 31st August. In April, our network people cleaned up flash: since there seem to have been some space issues, which appeared to have helped.
The latest crash info on the device was from May 2009, so we may be looking at some kind of lock-up rather than crash.
ASA# dir flash:
Directory of disk0:/
96 -rwx 15261696 10:10:08 Apr 18 2011 asa824-k8.bin
97 -rwx 14812604 11:00:16 Sep 23 2011 asdm-635.bin
99 -rwx 13879296 07:42:42 May 11 2009 asa804-32-k8.bin
10 drwx 4096 03:42:44 Nov 22 2007 crypto_archive
101 drwx 4096 07:43:18 Mar 13 2009 sdesktop
3 drwx 4096 06:04:48 Mar 13 2009 log
11 drwx 4096 16:10:18 Apr 15 2011 coredumpinfo
62881792 bytes total (18407424 bytes free)
ASA# sh crashinfo
Thread Name: aaa (Old pc 0x08069626 ebp 0xd45bd290)
Page fault: Address not mapped
error code 0x00000004
Cisco Adaptive Security Appliance Software Version 8.0(4)
Compiled on Thu 07-Aug-08 20:53 by builders
Crashinfo collected on 14:02:31.703 UTC Wed May 6 2009
ASA# show crashinfo save
crashinfo save enable
ASA# show cpu
CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 2%
ASA# show memory
Free memory: 133770744 bytes (50%)
Used memory: 134664712 bytes (50%)
Total memory: 268435456 bytes (100%)
I have a record of "soft" and "hard" crashes/hangs last year. Soft means the device stopped responding, but recovered by itself, probably within a few hours. Hard means that power-cycle was required. I have about four such events recorded for the Apr-Jun 2012 time frame, and no configuration changes between Oct 2011 and July 2012.
I will ask our NOC whether they can provide me with 8.2(5). Is there any way I can monitor cpu/mem on a more continuous basis, e.g. through snmp/mrtg?
Check if the version of IOS has the minimum requirements, I had a similar problem with version 8.2 (5) and 256MB of RAM.
The ASA consumed all RAM memory, and crashed.
The release notes for 8.2 say thta it uses more base memory and a RAM upgrade to 1GB is recommended if the device has less than 20% RAM free. This is not the case here - I've never seen memory consumption go higher than 50%, and cpu is usually well below 10%. But I have finally put some SNMP monitoring in place and will observe for a few days.
I have had a couple ASAs crash frequently on me in the past an in both cases it was a memory leak and replacing the RAM sorted things out.
Remember to take a backup of your config before doing any hardware changes if you haven't done so already.
The status light at the back is flashing amber. Some research here suggests that this may indicate bad ram. I'll upgrade to 1GB and see how that goes.
It's not a memory leak - used memory remains flat.
The status light at the back is still flashing amber after the DIMM upgrade. Any idea what that means?
After the RAM upgrade, I can really only be certain the problem has gone if it doesn't happen for at least six months as we have usually seen it happening anywhere from once a day to every few months.
An interesting observation: after the RAM upgrade, memory consumption jumped right up from around 135MB to 202MB. I was wondering why that is - does the device use excess RAM to copy data that is normally in firmware or PCI space, like certain desktop and workstation models do?
ASA was placed under maintenance and replaced the next time it crashed. No more problems since, which is a strong indication that the box had developed a fault.