12-01-2007 11:54 AM - edited 03-03-2019 07:45 PM
About three weeks ago we enabled pim-dm on our HP 5308xl core switches to facilitate VBrick video streaming on our network. All VLANs already had IGMP enabled--we simply added the global 'router pim' setting and the 'ip pim' setting on all VLAN's.
Since then we have had two instances where our core Cisco 3660 router will no longer forward packets and will not respond to a ping. Also, it will not accept console connections from my laptop to extract statistics prior to power recycling the router. These two instances have both occurred weekly on the same day late at night when our full backups are running. This router is a hub router for nine PTP T1 connections and is also the default gateway. Power recycling the router fixes the problem--until the next week at the same time. Is it possible our router needs to have some corresponding pim-dm configuration change made? Our current plan is to remove the pim-dm config on our HP core switches prior to the time it has occurred the past two weeks to observe the behavior. Any guidance would be greatly appreciated as we don't have Cisco support on our routers. Here is the sh ip interface output from the fe and a representative serial interface from the core router. Any other info gladly provided on request:
HS#sh ip int
FastEthernet0/0 is up, line protocol is up
Internet address is 10.0.0.1/16
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Multicast reserved groups joined: 224.0.0.10
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP Fast switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
WCCP Redirect outbound is disabled
WCCP Redirect exclude is disabled
BGP Policy Mapping is disabled
FastEthernet0/1 is administratively down, line protocol is down
Internet protocol processing disabled
Serial1/0 is up, line protocol is up
Internet address is 192.168.0.2/30
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Multicast reserved groups joined: 224.0.0.10
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is enabled
IP Flow switching is disabled
IP Fast switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
WCCP Redirect outbound is disabled
WCCP Redirect exclude is disabled
BGP Policy Mapping is disabled
12-01-2007 12:06 PM
Hi,
complete freezes are very bad and generally due to defective hardware. To add to the problem, at times one cannot tell if freezes are due to software problems, altough that is less likely, because the router has a wachdog hardware to protect against software infinite loops.
Your only choice is to upgrade to latest maintenance for the train you are running, and perhpas deconfigure things to see if it goes away.
12-01-2007 12:56 PM
I've seen router issues happen like this with high memory utilization resulting from high bandwidth utilization. I have also seen this happen because of a bug in the IOS. And yet I have seen this happen because of defective hardware...take your pick and start troubleshooting.
-brad
(please rate the post!)
12-01-2007 02:34 PM
HW failures are rare but not inexistent, one need to look at a large installed base to see them - I kind of do.
Had one on a 2600XM last month. Unpredictable freezes that happened at time of some traffic only.
TAC had RMA'd a new unit, end of issue.
12-01-2007 01:48 PM
Memory leaks are known to cause this type of behavior, especially since the lockup is periodic. Over time, the processor memory gets more and more depleted, and eventually there is nothing left to operate your console or telnet connections.
You have 2 options if this is the case:
1. Monitor 'show proc mem' over the period of a week, identify the process taking up the memory, and search for a bug on this.
2. Upgrade to the latest stable version
12-02-2007 08:35 AM
12-02-2007 01:40 PM
Back to my original response:
"I've seen router issues happen like this with high memory utilization resulting from high bandwidth utilization. I have also seen this happen because of a bug in the IOS. And yet I have seen this happen because of defective hardware...take your pick and start troubleshooting."
1) easy thing to try first, change the IOS.
2) if that doesn't do it, swap modules with a different box (if you can)
3) try new/different DRAM memory in the box
-brad
(please rate the post if this helps!)
12-02-2007 06:53 AM
Hi,
Too bad you don't have Cisco support for this router. I've seen this happen not only in routers but also in PIX/ASA, mostly it's a bug and just need to update the IOS. I suggest you capture "show tech-support" output before rebooting the router even though you don't have Cisco support - just in case you purchase it in future.
Regards,
Dandy
12-11-2007 04:54 AM
what i sugegst you do firstly on your core Cisco 3660 router is to limit the amount of multicast traffic the router is receiving on it interfaces. under each interface (receiving) use the " ip multicast rate-limit in/out" to limit the muticast traffic. you can also use an ACL as well.
Just keep in mind that the mcast traffic is UDP and will not backoff when congestion occurs and can cause you router cpu to spike to 100%.
Try to above and let me know.
Francisco.
12-11-2007 05:01 AM
also send me your ios version i will check to see if the problem is bug related.
12-11-2007 06:25 AM
Thanks very much, Francisco.
The symptoms were the onset of router hardware failure. We replaced the router with a brand new 3845 when it went belly up. Problem solved!
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: