Router Down Troubleshooting

[ray_stone]

Hello Experts:

Can somebody confirm the troubleshooting steps in order to find out the root cause of the router reboot. I would see all the steps to grab the information to know the actual root cause of the failure. Thanks...                 

[Leo Laohoo]

Can somebody confirm the troubleshooting steps in order to find out the root cause of the router reboot. I would see all the steps to grab the information to know the actual root cause of the failure. Thanks...

Can you please post the output to the following commands:

1.  sh version;

2.  dir

[Tagir Temirgaliyev]

Hi

make external syslog and so you'll find your answer

[Ivan Shirshin]

Hi Ray,

This is what you can do:

1. Check "show version" for last reload or "return to rom" reason line.

2. Check on SYSLOG server for the messages router managed to send before goign down.

3. Check connected file systems for the crashinfo file (typically flash or bootflash) and research its contents. You can read the file by "more ..." command or by ftp'ing it to your server/PC first.

Kind Regards,
Ivan

**Please grade this post if you find it useful.

[ray_stone]

This is what i get:

Please explain the root cause of the issue by analysing the logs stated below for your reference.

router#sh ver
Cisco IOS Software, 2800 Software (C2800NM-SPSERVICESK9-M), Version 12.4(25a), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 22-May-09 22:00 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)

router uptime is 2 hours, 54 minutes
System returned to ROM by power-on
System image file is "flash:c2800nm-spservicesk9-mz.124-25a"

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 2811 (revision 53.51) with 393216K/131072K bytes of memory.
Processor board ID FGL151110X2
2 FastEthernet interfaces
2 Serial(sync/async) interfaces
DRAM configuration is 64 bits wide with parity enabled.
239K bytes of non-volatile configuration memory.
126976K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

router#
router#sh log
Syslog logging: enabled (11 messages dropped, 1 messages rate-limited,
                0 flushes, 0 overruns, xml disabled, filtering disabled)
    Console logging: disabled
    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging: level debugging, 27 messages logged, xml disabled,
                    filtering disabled
    Logging Exception size (4096 bytes)
    Count and timestamp logging messages: disabled

No active filter modules.

    Trap logging: level notifications, 29 message lines logged
        Logging to x.x.x.x (global) (udp port 514, audit disabled,  link up), 29 message lines logged, xml disabled,
               filtering disabled
--More--                          
Log Buffer (8192 bytes):

*Jun  3 192.168.1.1: %LINEPROTO-5-UPDOWN: Line protocol on Interface VoIP-Null0, changed state to up
*Jun  3 192.168.1.1: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Jun  3 192.168.1.1: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up
*Jun  3 192.168.1.1: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
*Jun  3 192.168.1.1: %LINK-3-UPDOWN: Interface Serial0/0/1, changed state to down
*Jun  3 10.0.0.1: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
*Jun  3 10.0.0.1: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Jun  3 10.0.0.1: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to down
*Jun  3 10.0.0.1: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/1, changed state to down
*Jun  3 x.x.x.x:  RSA key size needs to be atleast 768 bits for ssh version 2
*Jun  3 x.x.x.x: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1111, changed state to up
*Jun  3 x.x.x.x: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback11, changed state to up
*Jun  3 x.x.x.x: %LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down
*Jun  3 x.x.x.x: %LINK-5-CHANGED: Interface Serial0/0/1, changed state to administratively down
*Jun  3 x.x.x.x: %SYS-5-CONFIG_I: Configured from memory by console
*Jun  3 x.x.x.x: %SYS-5-RESTART: System restarted --
Cisco IOS Software, 2800 Software (C2800NM-SPSERVICESK9-M), Version 12.4(25a), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 22-May-09 22:00 by prod_rel_team
*Jun  3 x.x.x.x: %SNMP-5-COLDSTART: SNMP agent on host router is undergoing a cold start
*Jun  3 x.x.x.x: %SSH-5-ENABLED: SSH 1.5 has been enabled
*Jun  3 x.x.x.x: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up
*Jun  3 x.x.x.x: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host x.x.x.x started - CLI initiated
*Jun  3 x.x.x.x: %LINEPROTO-5-UPDOWN: Line protocol on Interface , changed state to up
*Jun  3 x.x.x.x: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to up
*Jun  3 x.x.x.x: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up
*Jun  3 x.x.x.x: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
*Jun  3 x.x.x.x: %BGP-5-ADJCHANGE: neighbor 54.142.83.2 Up
*Jun  3 x.x.x.x: %WCCP-5-SERVICEFOUND: Service 61 acquired on WCCP client x.x.x.x
*Jun  3 x.x.x.x: %WCCP-5-SERVICEFOUND: Service 62 acquired on WCCP client x.x.x.x
router#
router#
router#exit

[Leo Laohoo]

System returned to ROM by power-on

The router was either rebooted or someone turned the power ON to the router.

Can you please post the output to the command "dir"?

[ray_stone]

while executing the command, I am getting this error:

router#dir

Command authorization failed.

router#dir
Command authorization failed.

[Leo Laohoo]

router#dir
Command authorization failed.

Your username does not have the right access level to invoke this command.

[ray_stone]

Today there is another alert recieved of router down... checked the router via sh ver where it was not rebooted. check the logs stated below:

Anything happened wrong... I changed the public IP with private IP's.

Jun  4 15:39:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1521, changed state to up

Jun  4 15:40:25: %BGP-5-ADJCHANGE: neighbor 192.168.1.1 Up

Jun  4 15:43:00: %BGP-5-ADJCHANGE: neighbor 192.168.1.1 Down BGP Notification sent

Jun  4 15:43:00: %BGP-3-NOTIFICATION: sent to neighbor 192.168.1.1 4/0 (hold time expired) 0 bytes

Jun  4 15:44:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1521, changed state to down

Jun  4 15:52:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1521, changed state to up

Jun  4 15:52:31: %BGP-5-ADJCHANGE: neighbor 192.168.1.1 Up

Jun  4 17:03:47: %BGP-3-NOTIFICATION: received from neighbor 192.168.10.10 4/0 (hold time expired) 0 bytes

Jun  4 17:03:47: %BGP-5-ADJCHANGE: neighbor 192.168.10.10 Down BGP Notification received

Jun  4 17:04:20: %BGP-5-ADJCHANGE: neighbor 192.168.10.10 Up

Jun  4 17:04:36: %BGP-5-ADJCHANGE: neighbor 192.168.10.10 Down Peer closed the session

Jun  4 17:05:02: %BGP-5-ADJCHANGE: neighbor 192.168.10.10 Up

Jun  4 18:18:06: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.20.20,dstadr=192.168.30.30,size=1456,handle=0x581B

Jun  4 18:18:06: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=2027 local=192.168.30.30 remote=192.168.20.20 spi=45422A6E seqno=0000066B

Jun  5 00:57:23: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.20.20,dstadr=192.168.30.30,size=120,handle=0x5809

Jun  5 00:57:23: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=2009 local=192.168.30.30 remote=192.168.20.20 spi=E3497154 seqno=000015B1 Jun  4 15:39:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1521, changed state to up
Jun  4 15:40:25: %BGP-5-ADJCHANGE: neighbor 192.168.1.1 Up
Jun  4 15:43:00: %BGP-5-ADJCHANGE: neighbor 192.168.1.1 Down BGP Notification sent
Jun  4 15:43:00: %BGP-3-NOTIFICATION: sent to neighbor 192.168.1.1 4/0 (hold time expired) 0 bytes
Jun  4 15:44:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1521, changed state to down
Jun  4 15:52:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1521, changed state to up
Jun  4 15:52:31: %BGP-5-ADJCHANGE: neighbor 192.168.1.1 Up
Jun  4 17:03:47: %BGP-3-NOTIFICATION: received from neighbor 192.168.10.10 4/0 (hold time expired) 0 bytes
Jun  4 17:03:47: %BGP-5-ADJCHANGE: neighbor 192.168.10.10 Down BGP Notification received
Jun  4 17:04:20: %BGP-5-ADJCHANGE: neighbor 192.168.10.10 Up
Jun  4 17:04:36: %BGP-5-ADJCHANGE: neighbor 192.168.10.10 Down Peer closed the session
Jun  4 17:05:02: %BGP-5-ADJCHANGE: neighbor 192.168.10.10 Up
Jun  4 18:18:06: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.20.20,dstadr=192.168.30.30,size=1456,handle=0x581B
Jun  4 18:18:06: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=2027 local=192.168.30.30 remote=192.168.20.20 spi=45422A6E seqno=0000066B
Jun  5 00:57:23: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.20.20,dstadr=192.168.30.30,size=120,handle=0x5809
Jun  5 00:57:23: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=2009 local=192.168.30.30 remote=192.168.20.20 spi=E3497154 seqno=000015B1

[pjmonline]

If the router is on a ups I would say the power supply is on its way out.

Sent from Cisco Technical Support iPhone App

[ray_stone]

I would need to know the troubleshooting steps on router to determine the root cause of the network slowness as one of the user reporting that the network was very slow but after 4hrs its working fine w/o problem.

Now, I need to investigate what was happened & it caused the network slowness. Thanks.

[Leo Laohoo · Accepted solution]

Now, I need to investigate what was happened & it caused the network slowness. Thanks.

Need more information than this.

1.  WHAT is slow?  Application?  Login?  Sending/receving emails?  Internet?

2.  Is it slow across the board or slow with one application and the rest are fine?

3.  Everyone experiencing this problem or ALL are experiencing this problem?

4.  Describe your network?

[ray_stone]

Well, we have around 10 sites connected via MPLS and network slowness issue was experienced by entire users for internet traffic and internal remote traffic which go via MPLS but it has been resolved.

I know that by monotoring the interface using monitoring tools we may see the bandwidth spike which could be a possible reason of this problem but can we check anything like interface details or something to know the same if suppose we don;t use any monitoring tools and what steps would you recommend if we face this issue live... Thanks.

[Leo Laohoo]

How big is your MPLS WAN link?  Is the site being serviced by a 2801 router?