cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1958
Views
0
Helpful
5
Replies

Very weird cpu utilization issue

Pim Scheffers
Level 1
Level 1

Ok this is just weird.

I have a stack of 2 3750's (WS-C3750G-24TS-S1U) both running IOS 12.2(46)SE.

During the work day these switches are constantly at 62% CPU utilization. (this information comes from my monitoring system)

Because i wanted to find out what the main reason for this was, i logged on to the switch with ssh and constantly  did a sh proc cpu sort 5sec. to find out what process the switches were working on.

Now for the strange part: When i was logged on to the switch the cpu would drop dramaticly to just 27%. 

When i logged OUT of the switch i saw the cpu increase immediately (again in the monitoring system) 

At first i thought it would have somthing to do with logging. So i turned off all possible logging & Debugging, but the issue remains the same.

Because a picture is worth a thousand words i added a picture from my monitoring system

Does anyone have an explanation or a solution for this behaviour?

Thanks,

Pim

5 Replies 5

Ganesh Hariharan
VIP Alumni
VIP Alumni

Ok this is just weird.

I have a stack of 2 3750's (WS-C3750G-24TS-S1U) both running IOS 12.2(46)SE.

During the work day these switches are constantly at 62% CPU utilization. (this information comes from my monitoring system)

Because i wanted to find out what the main reason for this was, i logged on to the switch with ssh and constantly  did a sh proc cpu sort 5sec. to find out what process the switches were working on.

Now for the strange part: When i was logged on to the switch the cpu would drop dramaticly to just 27%. 

When i logged OUT of the switch i saw the cpu increase immediately (again in the monitoring system) 

At first i thought it would have somthing to do with logging. So i turned off all possible logging & Debugging, but the issue remains the same.

Because a picture is worth a thousand words i added a picture from my monitoring system

Does anyone have an explanation or a solution for this behaviour?

Thanks,

Pim

Hi Pim,

Check out the below link on troubleshootin high CPU on Cisco 3750 switches

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml

Hope to Help !!

Ganesh.H

Remember to rate the helpful post

Hello Ganesh,

Thanks for the reply.

I know the document but it does not explain to me why the switch is 35% busier when i am NOT logged in with a  VTY line.

Pim

Hello Ganesh,

Thanks for the reply.

I know the document but it does not explain to me why the switch is 35% busier when i am NOT logged in with a  VTY line.

Pim

Hi Pim,

In order to troubleshoot the high CPU problem we need to check the following by issueing the commnd in switch at high CPU time

First check which process is taking high cpu by command show process cpu then we can come to some point for further analysis.

Hope to Help !!

Ganesh.H

Remember to rate helpful post

Hello Ganesh,

Well that's just the problem. i tried that (read my original post) but when i log on to the switch with ssh or the console port the cpu drops immediately 35%.

The only way i can see the cpu is at 62% is through my monitoring system.

When is do a sh proc cpu sorted 1min immediately after i login the busiest process does not consume more than 0.17%. (See below)

So my guess would be that the the 35% Extra is at Interrupt level.

That still does not explain why it's not doing that when i login to the switch either with ssh or console port.

sh proc cpu sort 1min
CPU utilization for five seconds: 27%/22%; one minute: 57%; five minutes: 59%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
137    41077777   5652281       7267  0.00%  0.14%  0.15%   0 HQM Stack Proces
267    43759037  18032643       2426  0.00%  0.14%  0.07%   0 SNMP ENGINE
138    37829575  22594904       1674  0.15%  0.13%  0.15%   0 HRPC qos request
192    47496746 300953302        157  0.00%  0.11%  0.17%   0 Spanning Tree
   4    45436147   3602719      12611  0.00%  0.10%  0.11%   0 Check heaps
  95    14105374  44922008        313  0.00%  0.06%  0.05%   0 hpm counter proc
265     9971220  35293026        282  0.00%  0.06%  0.01%   0 IP SNMP

Ok the cpu utilization comes definately from the interrupt level.

I just logged in and immediately did a sh proc cpu sort 5sec twice in a row and see the results:

sh proc cpu sort 5sec
CPU utilization for five seconds: 45%/39%; one minute: 61%; five minutes: 62%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
137    41086091   5653308       7267  0.15%  0.15%  0.15%   0 HQM Stack Proces
  59     67289051516009344          4  0.15%  0.03%  0.00%   0 Fifo Error Detec
138    37838979  22599012       1674  0.15%  0.05%  0.06%   0 HRPC qos request
192    47509352 301009620        157  0.15%  0.38%  0.25%   0 Spanning Tree
130    23577882 120024363        196  0.15%  0.04%  0.04%   0 HL3U bkgrd proce

sh proc cpu sort 5sec
CPU utilization for five seconds: 26%/20%; one minute: 56%; five minutes: 60%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
137    41086107   5653310       7267  0.15%  0.13%  0.15%   0 HQM Stack Proces
192    47509369 301009757        157  0.15%  0.33%  0.25%   0 Spanning Tree
  63     6008588  31366068        191  0.15%  0.03%  0.00%   0 hrpc <- response
  95    14109188  44930185        314  0.15%  0.06%  0.05%   0 hpm counter proc
  40     5023895   5657153        888  0.15%  0.03%  0.00%   0 Compute load avg

Now for the question why does it drop so drasticly when the second i logon to the switch?

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: