2811 freeze

Unanswered Question
Feb 9th, 2009
User Badges:

Hi, we have a year or 2 old 2811 router as our uplink to the isp. Well 2 times in the past 2 months, this thing seems to fail. All of a sudden there's no traffic passing but it's powered and light up, one day was a heavy traffic day(the inauguration) and a week later it happened on a light traffic day. The first time we contacted our ISP and after about an hour we just rebooted the router and everything was fine. The second time, we tried to console in, and it wouldn't allow it, nothing came up, we disallow vty since its connected to the internet. We rebooted and its been fine since.


So, should I just replace the router? How can I check to see if errors are happening if we can't console in? Is there anything I can check at all?


Thanks in advance

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
Joseph W. Doherty Mon, 02/09/2009 - 08:20
User Badges:
  • Super Bronze, 10000 points or more

Can't say this will cure your problem, but if you have maintenance, you might try upgrading to a later patch version. Either 12.4.3j or perhaps 12.4.23.

Leo Laohoo Mon, 02/09/2009 - 13:06
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

I agree with the previous post. "we tried to console in, and it wouldn't allow it" sounds like you're hitting a bug (CSCej77191). Consider upgrading the IOS.


Enable syslog and you can see the error messages as it go along.


Without the error messages, I can only find the following bug CSCej77191:


877/871 hangs - console prints %SYS-2-NOTQ: unqueue didnt find 0

Symptoms: Accessing some web pages results in the router appearing to hang.

No IP traffic goes to or from the router. None of the lights flash. The

console continously prints the following message:


%SYS-2-NOTQ: unqueue didn't find 0 in queue 831A65B4

-Process= "", ipl= 2

-Traceback= 0x807CBCBC 0x8008FBD8 0x806D1EB4 0x806D2020 0x8037C200 0x80135030

0x80129B34 0x8012C344 0x8012ED68 0x8034B0A4 0x800D3014 0x800D3014 0x8034B164

0x807F0654 0x807F0590 0x807ED9D4


The router must be power cycled to recover.


Conditions: This symptom has been observed on Cisco IOS Release 12.4T and

Release 12.4(4)T when Dynamic Multipoint VPNs (DMVPN) are being used.


Workaround: Disable bridging.

rhopkins_rcps Wed, 02/11/2009 - 07:03
User Badges:

Thanks for the great info Joseph and Leo, I will give upgrading a shot.

rhopkins_rcps Sun, 02/15/2009 - 18:41
User Badges:

Just out of curiosity, could this be a cpu or memory utilization issue due to heavy traffic?


Are there any services I could turn off ie http, bootp, password-encrypt, cdp, etc to free up more processor/memory usage?


Thanks again.

Leo Laohoo Sun, 02/15/2009 - 19:14
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Could be anything. Hardware issue, memory, CPU hog. My bet it's a hardware issue.

You have the router running for the last 2 years without rebooting or upgrading the IOS. Then all of the sudden you have to reboot the appliance twice just to get it going.

Joseph W. Doherty Mon, 02/16/2009 - 04:54
User Badges:
  • Super Bronze, 10000 points or more

"Just out of curiosity, could this be a cpu or memory utilization issue due to heavy traffic? "


Yes, although sometimes very hard to detect since when router "freezes" you're unable to see what's happening now. Sometimes monitoring will see the run up to freeze.


Another reason for later code, sometimes it responds better to being stressed.

Sushil Kumar Katre Mon, 02/16/2009 - 20:16
User Badges:
  • Gold, 750 points or more

Hi,


I would like to know if there were any configuration changes recently.


Also let me know if you have static routes if yes, please post the statements.


-> Sushil

rhopkins_rcps Thu, 02/19/2009 - 06:08
User Badges:

Hi, there were no recently changes before the router started slowing. I've attached a recent sh run and cpu output, unfortunately the cpu output is when the router wasn't problematic. I have disabled all the services I could think of.


Also, is there a way I could turn on logging buffered, then have a trigger/threshold to send a message to the log when the cpu or memory reaches a certain point?


Thanks



Joseph W. Doherty Thu, 02/19/2009 - 06:23
User Badges:
  • Super Bronze, 10000 points or more

Something you might change:


ip route 0.0.0.0 0.0.0.0 FastEthernet0/0

ip route 81.175.x.0 255.255.255.0 FastEthernet0/1


Use next hop IP addresses instead of interfaces. This avoids the router ARPing.


I believe it's posssible to use EEM to monitor CPU and generate a log message, but I haven't done so, so I can't provide an example.


Also with regard to disabling services, you might also want to disable console logging.


Also you might issue a show process CPU sorted command to see what's consuming the CPU.

Sushil Kumar Katre Thu, 02/19/2009 - 09:36
User Badges:
  • Gold, 750 points or more

Sir,


I can guarantee you 99% its the issue with your default and static route as pointed above by Joseph.


I have seen at least 4-5 such cases.


Please make the changes as suggested by Joseph and hopefully you'll not run into this issue again.


Good Luck!!!


-> Sushil

rhopkins_rcps Fri, 02/20/2009 - 13:21
User Badges:

I got the outside route no problem, but I'm running into an issue with the inside route.


The next inside hop is the firewall:

outside int - 81.175.57.3/25

dmz 1 int - 81.175.57.129/26

dmz 2 int - 172.16.129.1/17


Right now theres only one ip route to 81.175.57.0 255.255.255.0 fa0/1


Would I change it to:

ip route 81.175.57.4/25 81.175.57.3

ip route 81.175.57.130/26 81.175.57.129


Thanks again, RT

Joseph W. Doherty Fri, 02/20/2009 - 16:45
User Badges:
  • Super Bronze, 10000 points or more

Can't say for sure, since, at least to me, your topology is still a bit unclear (and you don't show interfaces' full ip address and mask in your posted running config).


Normally you would want the network(s) behind the firewall defined in the ip route statemement and just the ip address of the gateway that's on the segment connected to your router. (The router's interface will define the network for the transit segment.)


Perhaps something like:


ip route 81.175.57.129 255.255.255.92 81.175.57.3

ip route 172.16.128.0 255.255.128.0 81.175.57.3

Actions

This Discussion