Cisco 3660 Router Fails Weekly

[mcintoshs]

About three weeks ago we enabled pim-dm on our HP 5308xl core switches to facilitate VBrick video streaming on our network. All VLANs already had IGMP enabled--we simply added the global 'router pim' setting and the 'ip pim' setting on all VLAN's.

Since then we have had two instances where our core Cisco 3660 router will no longer forward packets and will not respond to a ping. Also, it will not accept console connections from my laptop to extract statistics prior to power recycling the router. These two instances have both occurred weekly on the same day late at night when our full backups are running. This router is a hub router for nine PTP T1 connections and is also the default gateway. Power recycling the router fixes the problem--until the next week at the same time. Is it possible our router needs to have some corresponding pim-dm configuration change made? Our current plan is to remove the pim-dm config on our HP core switches prior to the time it has occurred the past two weeks to observe the behavior. Any guidance would be greatly appreciated as we don't have Cisco support on our routers. Here is the sh ip interface output from the fe and a representative serial interface from the core router. Any other info gladly provided on request:

HS#sh ip int

FastEthernet0/0 is up, line protocol is up

Internet address is 10.0.0.1/16

Broadcast address is 255.255.255.255

Address determined by non-volatile memory

MTU is 1500 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.10

Outgoing access list is not set

Inbound access list is not set

Proxy ARP is enabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is disabled

IP Fast switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Probe proxy name replies are disabled

Policy routing is disabled

Network address translation is disabled

WCCP Redirect outbound is disabled

WCCP Redirect exclude is disabled

BGP Policy Mapping is disabled

FastEthernet0/1 is administratively down, line protocol is down

Internet protocol processing disabled

Serial1/0 is up, line protocol is up

Internet address is 192.168.0.2/30

Broadcast address is 255.255.255.255

Address determined by non-volatile memory

MTU is 1500 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.10

Outgoing access list is not set

Inbound access list is not set

Proxy ARP is enabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is enabled

IP Flow switching is disabled

IP Fast switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Probe proxy name replies are disabled

Policy routing is disabled

Network address translation is disabled

WCCP Redirect outbound is disabled

WCCP Redirect exclude is disabled

BGP Policy Mapping is disabled

[paolo bevilacqua]

Hi,

complete freezes are very bad and generally due to defective hardware. To add to the problem, at times one cannot tell if freezes are due to software problems, altough that is less likely, because the router has a wachdog hardware to protect against software infinite loops.

Your only choice is to upgrade to latest maintenance for the train you are running, and perhpas deconfigure things to see if it goes away.

[ccbootcamp]

I've seen router issues happen like this with high memory utilization resulting from high bandwidth utilization. I have also seen this happen because of a bug in the IOS. And yet I have seen this happen because of defective hardware...take your pick and start troubleshooting.

-brad

www.ccbootcamp.com

(please rate the post!)

[paolo bevilacqua]

HW failures are rare but not inexistent, one need to look at a large installed base to see them - I kind of do.

Had one on a 2600XM last month. Unpredictable freezes that happened at time of some traffic only.

TAC had RMA'd a new unit, end of issue.

[wochanda]

Memory leaks are known to cause this type of behavior, especially since the lockup is periodic. Over time, the processor memory gets more and more depleted, and eventually there is nothing left to operate your console or telnet connections.

You have 2 options if this is the case:

1. Monitor 'show proc mem' over the period of a week, identify the process taking up the memory, and search for a bug on this.

2. Upgrade to the latest stable version

[mcintoshs]

Thanks for the response. I've attached the first sh proc mem. Anything look abnormal. This is about one hour after reboot with very little traffic.

[ccbootcamp]

Back to my original response:

"I've seen router issues happen like this with high memory utilization resulting from high bandwidth utilization. I have also seen this happen because of a bug in the IOS. And yet I have seen this happen because of defective hardware...take your pick and start troubleshooting."

1) easy thing to try first, change the IOS.

2) if that doesn't do it, swap modules with a different box (if you can)

3) try new/different DRAM memory in the box

-brad

www.ccbootcamp.com

(please rate the post if this helps!)

[Danilo Dy]

Hi,

Too bad you don't have Cisco support for this router. I've seen this happen not only in routers but also in PIX/ASA, mostly it's a bug and just need to update the IOS. I suggest you capture "show tech-support" output before rebooting the router even though you don't have Cisco support - just in case you purchase it in future.

Regards,

Dandy

[francisco_1]

what i sugegst you do firstly on your core Cisco 3660 router is to limit the amount of multicast traffic the router is receiving on it interfaces. under each interface (receiving) use the " ip multicast rate-limit in/out" to limit the muticast traffic. you can also use an ACL as well.

Just keep in mind that the mcast traffic is UDP and will not backoff when congestion occurs and can cause you router cpu to spike to 100%.

Try to above and let me know.

Francisco.

[francisco_1]

also send me your ios version i will check to see if the problem is bug related.

[mcintoshs]

Thanks very much, Francisco.

The symptoms were the onset of router hardware failure. We replaced the router with a brand new 3845 when it went belly up. Problem solved!