This is what my theory was as well.

What do you guys think about disabling keepalives on trunk ports? Any disadvantages?

Absolutely a bad idea! This will fake the interface into being "UP" always, thus causing a black-hole for your traffic.

The keepalives are useful and protect the network from unforeseen topologies. Just as spanning-tree protects you from loops - don't disable that either (in a general sense, there are ethernet WAN situations where it can be done).

-John G

So why then are keepalives disabled by default on uplinks and fibre ports now?

And why does cisco reccomend to disable them in the links I have read. I have done some testing and and when the keepalives are disabled the interface does not stay 'up' always.

Good question, this is a response to a lot of cases being generated and confusion around this issue. I strongly discourage disabling keepalives unless there is a good reason (like having type 2 cabling). In desktop switches (2960, 3560, 3750, etc) the enhancement does disable the keepalives by default on uplinks and fiber interfaces - and at the same time doesn't fake link up. The faking of link up applies to the 6500 series - sorry for the confusion.

I will reiterate though: you need to find the problem with the topology, don't disable the protocols that stop the network from melting down.

Here is an excerpt from the original reasoning behind the default keepalive setting on desktop switches uplink and fiber ports:

Cisco supports Type 2 cabling, which does cause a loop in some situations. Switches are still used in environments where Type 2 cabling is used, so the keepalive / loopback detection mechanism cannot and will not be removed. If the keepalive / loopback detection mechanism is removed, a spanning-tree loop could occur, which is much worse then a port being errdisabled. (entire network going down vs. a single interface being errdisabled)

In addition, there have been many instances where the keepalive / loopback mechanism has prevented a bridging loop before spanning-tree could block the loop. Again, there is an advantage to errdisabling a single port instead of causing a network meltdown. (If you don't think this is useful, you probably don't see the benefit of UDLD Aggressive either, and you don't want me to start talking about how great a feature that is.)

Cisco has been convinced that there is less of a need to enable the keepalive / loopback mechanism on fiber and uplink ports because it is extremely unlikely that customers will use Type 2 cabling on network infrastructure ports. As a result, starting in 12.2SE, the fiber and uplink interfaces will no longer enable the keepalive / loopback mechanism by default.

The problem with the topology was introuduced when a new switch with redundant links was connected. This caused 9 out of 50 switches to go into err-disabled as a result of the loopback error.

From what I have seen disabling keepalives on ethernet trunk ports does not affect the way STP works. Ports are still put into the blocking state when necessary. Disabling the keepalives does not affect BPDUs which is what STP needs to detemine the physical topology and make decisions.

What i'm trying to understand is why this actually happened. Why did some ports go into err-disabled on not everything? How did these keepalives get out before STP blocked the redunadnt port?

We are using Cat6 on all our trunk ports. And our standard is to use the last two ports (Gi0/47 and Gi0/48) for the uplink to another switch (when fibre isn't used) I only want to disable keepalives on those trunk ports, since these are our uplinks. And if cisco is happy to disable keepalives by default on uplinks then i really cant see any issue with manually disabling them on these ports only.

My understanding of keepalives is that the are sent from and to the source interafces MAC address just as a test of the line. So they are supposed to be received by the interface that sent them. I believe that the error occurs when a keepalive from one interface is received on a different interface.

You are correct, the loopback detection will not affect STP. It can act as a second layer of protection, however, when STP is configured incorrectly. For instance, if you connect a switch with BPDU filter and you have portfast enabled without bpduguard.

I believe you see some ports go err-disable and not all of them *because* enough ports went err-disabled to stop the loop before it propagated entirely through the network.

Also, keepalives are at a lower level and cheaper for CPU to process than STP as it is a "dumb" protocol. If CPU is busy, you may have either STP or loopback detection kick in, whichever the CPU can service first. In a bridging loop scenario, there is a lot of chaos and it never happens the same way twice.

Feel free to disable keepalives, but I would not do it if I was not 100% sure what was going on in my network. As a troubleshooting step, it could teach us more about what might have gone wrong though. Are you going to be able to try to reproduce the problem for troubleshooting, or is there a network freeze now? Maybe we can go over the configs for the directly connected new device and existing 3750 stack.

Some cisco switches decided to disable the keepalives to reduce the number of cases it generated by customers that got STP and loopback messages and either one can do the same job in many scenarios. It's a compromise really. I will always say it's prudent to use them because a) they don't hurt you and b) they can alert you to a problem. The only time I would say you should go forward and intentionally disable them long term is if your network is known to loop back frames due to cable type or other L1 constraints.

As far as keepalives - they are sent to/from the same address and are supposed to be received at the time they are sent. If another copy is received, that means there is an L1 loop, or L2 loop. The frame shouldn't go through another switch and back out where it came from unless a loop is present. Keepalives can be seen on other interfaces in a hub network, and they should be ignored by other devices or interfaces as the mac address does not belong to the far end.

Regards,

John

Ah ok, what you said about the keepalives makes sence... The error is triggered when more than one copy of the same keepalived is received.

I have tried to reproduce the issue in a test lab but have been unable to. We don't have a 3750 stack in the test lab, just a few 3560s. We are planning to hook up the offending switch again once the keepalives have been disabled.

Just some info - the 3750 is the root bridge for almost all our VLANs in the campus. It has about 16 other switches hanging off it, but none with redundant links. (soon to change) The whole campus has around 50 switches connected using a mix of Fibre and Cat6. There are loops in the network for redundany which STP is taking care of without any issues.

The offending 3560 was connected to the 3750 first on Gi1/0/18, then on Gi2/0/18.

I have attached the configs from the 3750 and the 3560 if you would like to have a look. I don't think it was a config issue however

Thanks for your help so far!

Hello John,

>> As far as keepalives - they are sent to/from the same address and are supposed to be received at the time they are sent. If another copy is received, that means there is an L1 loop, or L2 loop. The frame shouldn't go through another switch and back out where it came from unless a loop is present. Keepalives can be seen on other interfaces in a hub network,

This makes sense and it is what should happen but I'm afraid that some implementations react also to first copy of keepalive and so the bug.

Best Regards

Giuseppe

HI Giuseppe,

I also agree with you that the bug description from Cisco is confusing. Perhaps it means that the keepalives are looped back to a different port? i.e. A different port to the one that sent the keepalive?

My theory still is that a few of these keepalives got out before STP could block the second port. These keepalives when then received on a few ports that did not expect them, which created the err-disabled state.

We had the same issue a while ago and made the decision to disable the keepalives. Haven't had any problems since.