Ghost Traffic Entering Up/Down Interface

r.repas · ‎07-23-2008

Hi, all.

I seem to have a very unusual problem. I've got a 7206VXR with an NPE-400 and four PA-MC-2T3+ port adapters. It's running 12.2(18)S12.

The symptom I'm seeing is that a configured, but inactive, T1 interface on a T3 will be in an up/down state and apparently begin taking "ghost" traffic inbound (based on the interface stats). That's odd enough, but that same traffic seems to be switched out another T1 interface on a different T3. It causes all manner of problems for the live customer on the destination interface. Putting the offending up/down T1 interface into an administratively down state ceases the "ghost" traffic and corrects the problem.

I've seen this happen three times on this chassis in the last month. The router was reloaded between each instance and a suspect PA was replaced between the second and third instance. Also, it's not always the same interface or even PA slot causing the issue.

I've checked the bug toolkit and the release notes for 12.2S, but came up empty handed. Has anyone seen anything like this before?

Thanks, Robin.

Edison Ortiz · ‎07-24-2008

up/down describes an interface that has a Layer1 connectivity to another device but they don't agree on the encapsulation or the clocking is missing.

There is a still traffic (keepalives and such) between these 2 devices. What you are seeing is normal.

HTH,

__

Edison.

r.repas · ‎07-24-2008

Thanks for the response, but this is certainly not normal.

First, there is no device connected on the other end of the up/down link. Second, the amount of traffic inbound approaches the full T1 bandwidth. That's certainly a lot more than keepalives. Third, this ghost traffic is switched out another T1 interface and interferes with the live traffic on that interface.

There is an ebb and flow to the ghost traffic - it comes and goes. Another data point is that our NMS which performs ping monitoring starts to note duplicate responses from the CPE on the interface that is being DoS'ed by the ghost traffic.

This is most certainly some form of bug or defect. Perhaps an issue with the CEF switching table. I was just hoping someone else has run across it before and could advise. My next course of action is to change out the entire chassis (including PAs, I/O controller and NPE) to insure I'm not fighting a hardware defect.

Robin Repas, CCNP

Network Engineer, Operations

First Communications