High CPU % on Catalyst 6509

burleyman · ‎12-03-2008

I just got an alert for high CPU utilization on our Catalyst 6509

CPU utilization for five seconds: 89%; one minute: 91%; five minutes: 90%

These seem to be the two culprits....

PID 5Sec 1Min 5Min Process

16407 72.5% 75.1% 74.7% ios-base

16429 8.5% 9.7% 10.1% udp.proc

What could be causing this? Where should I start? It looks like the IOS-Base is the main issue....why would that be so high?

Any direction would be great.

Thanks,

Mike

ariela · ‎12-03-2008

Hi Mike,

please send us more details.

A show tech could be the best (without passwords or private data), but first of all IOS version and a show logg if applicable.

Andrea

mihanlin · ‎12-03-2008

Hi,

You are running modular IOS on the switch. The main culprit is ios-base, which is basically a group of process that haven't been split individually.

To drill down into what is in ios-base, do:

show processes cpu detailed ios-base. This will tell you the process that's causing the CPU spike.

burleyman · ‎12-04-2008

I have attached the output of the command. Looks like PID 6 is using the most. Is intr state mean that it is interupted?

I also included the show ver as well. We have two 6509's running VSS.

Any help would be great.

Mike

ariela · ‎12-04-2008

Well,

just my first impression is an unknown bug. I've investigated with bug toolkit, and I've found some bugs for SXH train (ex. CSCsq66355) but fixed on SXH3 (your release). First of all, I suggest an IOS upgrade to SXH4, that is the last release of your train.

Another thing to investigate is your log, I see some interfaces flapping there.

Please let me know if I could be more helpful.

Regards

Andrea

burleyman · ‎12-04-2008

I did take care of the flapping interfaces when I saw them. I thought a bug but this spiked and stayed yesterday around noon, and nothing was done to the switch. We normally run between 20 and 30% and have been until the spike yesterday.

I have found the task in the process called ION IP output that looks like it is the main culprit. any one know what this is or does? Also is it something I can kill or restart with out rebooting the switches?

Thanks

Mike

ariela · ‎12-04-2008

Another bug (maybe the same as you indicate) is CSCsr12976, that is strictly related to the use of 'show run' command. But is another bug fixed on SXH3.

Rebooting the entire device could do the trick.

HTH

Andrea

burleyman · ‎12-04-2008

Reboot is very last option. I will check the Bugs you listed. If you think of anything else that would be great. Thanks for all you help so far.

Mike

ariela · ‎12-04-2008

Yes I know, reboot is always the last option ... but in a VSS environment is not as critical as you can imagine.

at the same time I would upgrade your IOS image.

HTH

Andrea

burleyman · ‎12-04-2008

This has been resolved. After checking the buffers and interfaces.I found a machine flooding directed broadcasts and it was overwhelming the switch. I remove the culprit and I am now back to normal. Now I have to find why it is doing that.

Thanks to everyone for your help, you are why this forum is the best on the net!

Mike