Good question. I don't know. The symptoms were no, or very very little, traffic was going through the switch. And two, the management console was non responsive. The fix was power down the switch, let traffic fail over to the "B" redundant switch. The "B" switch had no clue the primary switch/route path was not routing traffic.

This wasn't a race condition, as such. Race conditions happen when two threads or semaphores can only proceed when they acquire a resource the other has already secured.

You DID experienced a high-cpu condition, whose cause is yet unknown. More often than not this is caused by interrupt traffic (traffic hitting the CPU for one reason or another). When this happens, try consoling into the switch (Telnet/SSH will be unresponsive). Issue the command 'show process cpu' and see if that sheds some light.

It may not be "race" condition as such, then. An example of a race condition that immediately popped in my head is when you have BGP neighbor IP address discovery that relies on BGP routing itself, which will cause BGP session to come up for a few seconds before being torn down again, then retry, etc. etc. causing BGP to flap. Race condition affecting h/w typically causes the platform to be stuck in "reboot loops".

Now, back to your problem... You said you couldn't console to it, how about telnet? If I'm not mistaken, console actually uses more CPU cycle than Telnet. So, if your problem is related to high CPU utilisation like the previous poster said, console may not work but telnet *might* (*might* being the operative word here!). If you could telnet to it, then try issuing the "show processes cpu" command as suggested before. Failing that, I think you need to log a case with TAC.

Other way around, here, if the high CPU is interrupt-triggered. The influx of packets will oversubscribe the link to the ASIC on the 6k that sits between the supervisor port asic and the RP CPU, and the packets will get dropped there. Very few telnet packets will get through.

We always recommend consoling into the device during a high CPU condition.

Yes, the switch was in a High CPU Mode. An engineer was able to SSH into the the console, reported that responses were very very slow and was kicked out/disconnected. Thereafter no one was able to SSH in.

What are likely causes or common reasons of hich CPU conditions in the switch? What kinds of admin commands/mistakes could cause a high cpu condition ?(switch contains fibre and copper cards)

thanks -M

There can be many things that cause high CPU utilization. Some of the more common culprits that I've come across are:

- Process Switching on one or more of the highly utilized interfaces.

- Excessive Spanning Tree topology changes (may be due to misconfigured Spanning Tree)

- Excessive routing topology changes, may be due to mutual redistribution that causes partial route feedback/loop requiring constant route recalculation

I think your best course of action is to log a case with TAC, supplying them with "show tech-support" output, somehow.

Edit: Do you have backup configs from both primary and secondary switches? If you do, you might want to compare the two and see if there are any differences. Also, is there any difference in hardware?

Thank you very much, this is very helpful. I will act on these recommendations.

If you think of anything else please don't hestitate to post an update.

Not a problem. Happy to help.

If you've solved the issue, can you please post an update so that we can all learn from your experience too? Thx