6509 All Routing Dropped

[bryan.mcmullan]

I have three sites running EIGRP in a ring topology. Sites 1 and 2 have been up and running for years, we added site 3 recently.

We've run into an issue where every so often the 6509 in site 3 basically drops all routes for about 6 minutes. It un-neighbors and all traffic is dropped during that time. This includes most traffic for directly connected routes (though for some reason, there seems to be 1 or two subnets that can continue to function).

I'm at a loss for what to look for, at Site 1 and 2 I can see the EIGRP events where site 3 neighbors again after the

issue occurs. But everything looks clean within 6509 and I can't seem to find anything wrong.

Any assistance or advice would be much appreciated! Thank you in advance.

[vmiller]

How physically are the sites connected?

What (If anything) is being logged on the routers?

Might want to post some configs

[bryan.mcmullan]

The sites are connected via bundled T's with a copper handoff. 10Mb on one site, 12Mb on the other, with a 50Mb line between sites 1 and 2.

The issue seems to be relegated to Site 3 as site 1 and 2 only show EIGRP neighboring again after the issue occurs at site 3. Both other sites can route between each other and internally without issue.

Attached is a current config with most ports redacted (basic switchport/port channel configs).

What logging would you suggest I look at? Sh ip eigrp events buffer overwrites too quickly and only shows the routes being re-added. There's nothing in the sh log from anytime within the last 2 weeks. Anyplace else you'd suggest I look?

Thanks again!

[bryan.mcmullan]

Here is the eigrp event log as well.

[Giuseppe Larosa]

Hello Bryan,

among possible reasons I see:

external metroethernet issue (but it is unlikely an STP reconvergence should take 50 seconds max not 6 minutes);

the monitor session of vlan1: we have found out that the traffic that should be mirrored is much more then the speed of the destination port cpu usage goes to 100% and device can become unreachable.

Note:

I find your EIGRP config strange:

router eigrp 1

--More-- network 10.5.5.0 0.0.0.255

--More-- network 10.6.6.0 0.0.0.255

--More-- network 10.9.9.0 0.0.0.255

--More-- network 10.50.8.0 0.0.1.255

--More-- network 10.50.20.0 0.0.0.255

--More-- network 10.50.251.0 0.0.0.255

--More-- network 10.50.255.128 0.0.0.3

--More-- network 10.50.0.0 0.0.255.255

--More-- eigrp router-id 10.50.255.2

--More-- no auto-summary

--More-- redistribute static

--More-- nsf

--More-- !

--More-- ip classless

--More-- ip route 0.0.0.0 0.0.0.0 10.50.255.1

--More-- ip route 10.5.5.0 255.255.255.0 10.50.255.1

--More-- ip route 10.6.6.0 255.255.255.0 10.50.255.1

--More-- ip route 10.9.9.0 255.255.255.0 10.50.255.1

--More-- ip route 10.50.8.0 255.255.254.0 10.50.255.1

--More-- ip route 10.50.20.0 255.255.255.0 10.50.255.1

--More-- ip route 10.50.100.0 255.255.254.0 10.50.255.1

--More-- ip route 10.50.251.0 255.255.255.0 10.50.255.1

--More-- !

I don't understand why you have network commands for the same IP subnets that are known as static routes

And who is 10.50.255.1?

About the sh ip eigrp events: oldest events are the last ones and all start with a network that become unreachable.

There is anything in sh log of device that can be related to the issue?

Hope to help

Giuseppe

[bryan.mcmullan]

There are a few routes that are located behind our firewall. 10.50.255.1 is the vip of our FW cluster. It hides a few vlan's behind it for management.

I think I've found the underlying cause. The issue has happened 3 times since last week. When looking at a few ports, there were also 3 if resets on them for just that single blade. All other blades look clean and had no if resets. This blade also houses our site to site links.

Looks like we have a bad blade. Opening a case now to get it replaced.