High CPU % on Catalyst 6509

Unanswered Question
Dec 3rd, 2008
User Badges:
  • Blue, 1500 points or more

I just got an alert for high CPU utilization on our Catalyst 6509

CPU utilization for five seconds: 89%; one minute: 91%; five minutes: 90%


These seem to be the two culprits....

PID 5Sec 1Min 5Min Process

16407 72.5% 75.1% 74.7% ios-base

16429 8.5% 9.7% 10.1% udp.proc


What could be causing this? Where should I start? It looks like the IOS-Base is the main issue....why would that be so high?


Any direction would be great.


Thanks,

Mike

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4 (3 ratings)
Loading.
ariela Wed, 12/03/2008 - 10:32
User Badges:
  • Silver, 250 points or more

Hi Mike,


please send us more details.

A show tech could be the best (without passwords or private data), but first of all IOS version and a show logg if applicable.


Andrea

mihanlin Wed, 12/03/2008 - 15:05
User Badges:
  • Silver, 250 points or more

Hi,


You are running modular IOS on the switch. The main culprit is ios-base, which is basically a group of process that haven't been split individually.


To drill down into what is in ios-base, do:

show processes cpu detailed ios-base. This will tell you the process that's causing the CPU spike.

burleyman Thu, 12/04/2008 - 04:29
User Badges:
  • Blue, 1500 points or more

I have attached the output of the command. Looks like PID 6 is using the most. Is intr state mean that it is interupted?

I also included the show ver as well. We have two 6509's running VSS.


Any help would be great.


Mike



Attachment: 
ariela Thu, 12/04/2008 - 05:42
User Badges:
  • Silver, 250 points or more

Well,


just my first impression is an unknown bug. I've investigated with bug toolkit, and I've found some bugs for SXH train (ex. CSCsq66355) but fixed on SXH3 (your release). First of all, I suggest an IOS upgrade to SXH4, that is the last release of your train.


Another thing to investigate is your log, I see some interfaces flapping there.


Please let me know if I could be more helpful.

Regards

Andrea

burleyman Thu, 12/04/2008 - 06:00
User Badges:
  • Blue, 1500 points or more

I did take care of the flapping interfaces when I saw them. I thought a bug but this spiked and stayed yesterday around noon, and nothing was done to the switch. We normally run between 20 and 30% and have been until the spike yesterday.


I have found the task in the process called ION IP output that looks like it is the main culprit. any one know what this is or does? Also is it something I can kill or restart with out rebooting the switches?



Thanks

Mike

ariela Thu, 12/04/2008 - 06:17
User Badges:
  • Silver, 250 points or more

Another bug (maybe the same as you indicate) is CSCsr12976, that is strictly related to the use of 'show run' command. But is another bug fixed on SXH3.


Rebooting the entire device could do the trick.


HTH

Andrea

burleyman Thu, 12/04/2008 - 06:28
User Badges:
  • Blue, 1500 points or more

Reboot is very last option. I will check the Bugs you listed. If you think of anything else that would be great. Thanks for all you help so far.


Mike

ariela Thu, 12/04/2008 - 06:33
User Badges:
  • Silver, 250 points or more

Yes I know, reboot is always the last option ... but in a VSS environment is not as critical as you can imagine.


at the same time I would upgrade your IOS image.


HTH

Andrea

burleyman Thu, 12/04/2008 - 11:29
User Badges:
  • Blue, 1500 points or more

This has been resolved. After checking the buffers and interfaces.I found a machine flooding directed broadcasts and it was overwhelming the switch. I remove the culprit and I am now back to normal. Now I have to find why it is doing that.



Thanks to everyone for your help, you are why this forum is the best on the net!


Mike

Actions

This Discussion