ASR 1004 crashed

[Muhammed AKYUZ]

Hi,

Our ASR crashed today. What should be the problem?

------------------ show version ------------------

Cisco IOS Software, IOS-XE Software (X86_64_LINUX_IOSD-ADVENTERPRISEK9-M), Version 15.1(1)S2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Tue 26-Apr-11 13:53 by mcpre


Cisco IOS-XE software, Copyright (c) 2005-2011 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


ROM: IOS-XE ROMMON

ASR1004-79-ADSL uptime is 1 hour, 30 minutes
Uptime for this control processor is 1 hour, 31 minutes
System returned to ROM by reload at 20:46:21 TEB Tue Dec 6 2011
System restarted at 20:49:25 TEB Tue Dec 6 2011
System image file is "bootflash:asr1000rp2-adventerprisek9.03.02.02.S.151-1.S2.bin"
Last reload reason: Reload Command



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco ASR1004 (RP2) processor with 4279411K/6147K bytes of memory.
8 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
8388608K bytes of physical memory.
1925119K bytes of eUSB flash at bootflash:.
78085207K bytes of SATA hard disk at harddisk:.

Configuration register is 0x2102

[Leo Laohoo]

Software caused reload.  Most likely caused by the constant EIGRP adjacency constantly going up and down.

[Muhammed AKYUZ]

so what is the solution?

[Leo Laohoo]

Fix your EIGRP adjacency first.  Why is the adjacency constantly going up and down?

Before this crash, the ASR was up for about 1 week and 4 days.  So I will have to presume that it must've been crashing.  So fix the underlying issue before we tackle the next possibility.

[Muhammed AKYUZ]

it is the first crash, eigrp should not crash ASR.

[Leo Laohoo]

I'm not discounting an IOS bug, however, seconds before the crashed occur nothing is showing that it's caused by an IOS bug.  Instead, the logs is showing that your EIGRP adjacencies keep going down and up and then the appliance crashed.

If you believe that unstable EIGRP adjacencies are not causing the issue (particularly the logging side) then disable logging (not recommendable).

Otherwise another option is to upgrade your IOS.

[Muhammed AKYUZ]

Eigrp is stable. Before the crash, I think the main dmvpn tunnel was down, when it went up, routers' control plane did not handle the all eigrp requests and crashed. But it should not be crashed.

[maayre]

Now that I can open the file on my PC I can see this;

Dec  6 2011 18:48:18.173 : %ASR1000_INFRA-4-NO_PUNT_KEEPALIVE:  Keepalive not received for 300 seconds

Dec  6 2011 18:48:18.173 : %ASR1000_INFRA-2-FATAL_NO_PUNT_KEEPALIVE:  Keepalive not received for 300 seconds resetting

The reason for reset is that the RP lost contact with the ESP (forwarding engine) and reloaded the system.

I think your version is current enough that what the ESP was doing at the time was also captured (enhancement added due to this CSCtl92129) but you will still need TAC to look into it further.

[networkguy13111]

It would be rediculous route flaps crash Cisco core routers. We have quite a few crashes on our main gateway ASR1006 with dual RP1 which cause P1 network outage for us.

Our network staff start to lose confidence on the ASR, if there are choice please don't use it as the core router.

--
Best Regards

[Leo Laohoo]

I've got a few ASR 1002 and I've never had a crash.

Granted the IOS is always (and I mean ALWAYS) gets upgraded.

Our routers are used for internet gateway with 1 Gbps connection. 

[Leo Laohoo]

Fei Li,

If your fleet of ASR keeps crashing, what have you done?  Did you open a TAC Case and got the crashinfo checked?

[maayre]

It would be unlikely that the EIGRP flaps themselves caused a crash.

If you log a TAC case they can decode the crash and at least tell you what the ASR was doing at the time when it decided it had enough and would crash. From there they may be able to confirm some memory/CPU resource issue relating to EIGRP (if it is related at all) or some other reason.

This crash means that the system took itself down, most of the time you will need to decode to see why.

PS sorry if there is an attachment I'm on the iPhone app and can't check it