Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Community Member

3560 CPU - 88%

hello, we are using a 3560 switch and the cpu utilization is constantly showing 88 %.

This switch is doing routing and leased lines are connected to service providers.

Is there any trouble shooting steps that can be done?

thanks

VIvk

Everyone's tags (1)
16 REPLIES
Bronze

3560 CPU - 88%

Hello Vivek,

By having high CPU it means that from show proc cpu history the large spot of # means that the CPU experienced constant high load with averages up to 88%.

Changing the sdm template to routing may alleviate performance issues if we are getting TCAM problems since the switch is doing routing.
Check tcam with sh platf tcam utilization, sh logs and check if we are getting %PLATFORM_UCAST-4-PREFIX: messages.

Try to gather:

sh proc cpu | exc 0.00
sh contro cpu-int

Check the process(es) causing the switch to process-switching a lot of traffic.

Link
-------

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af3.shtml

Haihua

Community Member

3560 CPU - 88%

hi Haihua, thank you for the help, I had pasted the output of the commands. Could you kindly take a look.

TATAIDC-L3-Sw#sh proc cpu | exc 0.00

CPU utilization for five seconds: 85%/64%; one minute: 87%; five minutes: 87%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  74  2428691633 632791155       3838 13.09% 14.72% 14.82%   0 HLFM address lea

  94    23028294  23684199        972  0.15%  0.12%  0.11%   0 hpm counter proc

107         519        58       8948  1.27%  0.49%  0.10%   1 SSH Process     

131    11454478 680366466         16  0.31%  0.12%  0.07%   0 Hulc LED Process

132   129347046  17550716       7369  0.31%  0.51%  0.51%   0 HL3U bkgrd proce

204    89892689 117800715        763  0.95%  0.42%  0.40%   0 Spanning Tree   

309    41667912  27523089       1513  0.31%  0.28%  0.28%   0 SNMP ENGINE     

TATAIDC-L3-Sw#

TATAIDC-L3-Sw#sh contro cpu-int

ASIC    Rxbiterr   Rxunder    Fwdctfix   Txbuflos   Rxbufloc   Rxbufdrain

-------------------------------------------------------------------------

ASIC0     0          0          0          0          0          0        

ASIC1     0          0          0          0          0          0        

ASIC2     0          0          0          0          0          0        

ASIC3     0          0          0          0          0          0        

ASIC4     0          0          0          0          0          0        

ASIC5     0          0          0          0          0          0        

ASIC6     0          0          0          0          0          0        

HOL Fix Counts

--------------

No Fixes:          0 Added:          0 In Use:          0 Both:          0

CPU Heartbeat Statistics

Tx Success Tx Fail    1st Thr    2nd Thr    Unthr      RetryCtMax

---------- ---------- ---------- ---------- ---------- ----------

471361363          0          0          0          0          1

Rx Delay

         0          1          2          3          4

---------- ---------- ---------- ---------- ----------

471361363          0          0          0          0

AddlDelay AdvanceCnt

---------- ----------

         0          0

Rx Retries by RetryCount

         0          1          2          3          4          5          6

---------- ---------- ---------- ---------- ---------- ---------- ----------

471361363          0          0          0          0          0          0

         7          8          9

---------- ---------- ----------

         0          0          0

--More--                          

AddlRetry

----------

         0

cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray

----------------- ---------- ---------- ---------- ---------- ----------

rpc               1          0          0          0          0        

stp               55630558   0          0          0          0        

ipc               0          0          0          0          0        

routing protocol  531317387  0          0          0          0        

L2 protocol       11709909   0          0          0          0        

remote console    0          0          0          0          0        

sw forwarding     1851430181 0          0          0          0        

host              57685503   0          0          0          0        

broadcast         402389844  0          0          0          0        

cbt-to-spt        0          0          0          0          0        

igmp snooping     44308332   0          0          0          0        

icmp              615701075  0          0          0          0        

logging           0          0          0          0          0        

rpf-fail          0          0          0          0          0        

dstats            0          0          0          0          0        

cpu heartbeat     471361430  0          0          0          0        

cpu-queue         static inuse static added

----------------- ------------ ------------

rpc               0            0          

stp               0            0          

ipc               0            0          

routing protocol  0            0          

L2 protocol       0            0          

remote console    0            0          

sw forwarding     0            0          

host              0            0          

broadcast         0            0          

cbt-to-spt        0            0          

igmp snooping     0            0          

icmp              0            0          

--More--                          

logging           0            0          

rpf-fail          0            0          

dstats            0            0          

cpu heartbeat     0            0          

Supervisor ASIC receive-queue parameters

----------------------------------------

queue 0 maxrecevsize 0 pakhead 0 paktail 0

queue 1 maxrecevsize 7E0 pakhead 525AD98 paktail 5256CA0

queue 2 maxrecevsize 7E0 pakhead 4350598 paktail 5226E94

queue 3 maxrecevsize 7E0 pakhead 446A804 paktail 422BB54

queue 4 maxrecevsize 7E0 pakhead 5263324 paktail 43872E0

queue 5 maxrecevsize 7E0 pakhead 0 paktail 0

queue 6 maxrecevsize 7E0 pakhead 5336D68 paktail 445F248

queue 7 maxrecevsize 7E0 pakhead 441CBA8 paktail 4426A78

queue 8 maxrecevsize 7E0 pakhead 44305AC paktail 442C850

queue 9 maxrecevsize 7E0 pakhead 43C87B0 paktail 43C87B0

queue A maxrecevsize 7E0 pakhead 43BADE4 paktail 43C0F58

queue B maxrecevsize 7E0 pakhead 534AFE8 paktail 534C1F4

queue C maxrecevsize 7E0 pakhead 52AA8A8 paktail 43CF910

queue D maxrecevsize 7E0 pakhead 52A2210 paktail 43C8414

queue E maxrecevsize 0 pakhead 0 paktail 0

queue F maxrecevsize 7E0 pakhead 43B4FA4 paktail 528D2B0

Supervisor ASIC exception status

--------------------------------

Receive overrun    00000000   Transmit overrun 00000000

FrameSignatureErr  00000000   MicInitialize    00000000

BadFrameErr        00000000   LenExceededErr   00000000

BadJumboSegments   00000000

Supervisor ASIC Mic Registers

------------------------------

MicDirectPollInfo                         80000200

MicIndicationsReceived                    00000000

MicInterruptsReceived                    00000000

MicPcsInfo                               0000011F

MicPlbMasterConfiguration          00000000

MicRxFifosAvailable                      00000040

--More--                          

MicRxFifosReady                          0000BFBE

MicTimeOutPeriod:          FrameTOPeriod: 00000EA6          DirectTOPeriod: 00004000

MicTransmFramesCopied                    00000003

MicTxFifosAvailable                      0000007F

MicConfiguration:          Conf flag: 00000110          Interrupt Flag: 0000000A

MicReceiveFifoAssignmen          Queue 0 - 7: 44444444          Queue 8 - 15:44444444

MicReceiveFramesReady:          FrameAvailable: 00000041          frameAvaiMask: 00000000

MicException:   

Exception_flag          00000000

Message-1          00000000

Message-2          00000000

Message-3          00000000

MicIntRxFifo:

ReadPtr                    000001B0          WritePtr          000001B0

WHeadPtr          000001B0          TxFifoDepth          C0000800

MicIntTxFifo:

ReadPtr                    000004B0          WritePtr          000004B0

WHeadPtr          000004B0          TxFifoDepth          C0000800

MicDecodeInfo:

Fifo0:          address:          03FF4000 asic_num:          00000100

Fifo1:          address:          03FF4400 asic_num:          00000101

MicTransmitFifoInfo:

Fifo0:           StartPtrs:          0697E800          ReadPtr:          0697EE10

WritePtrs:          0697EE10          Fifo_Flag:          8A800800

Weights:          001E001E

Fifo1:           StartPtrs:          06827400          ReadPtr:          068275B8

WritePtrs:          068275B8          Fifo_Flag:          89800400

Weights:          000A000A

MicReceiveFifoInfo:

Fifo0:          StartPtr:          0697E760          ReadPtr:          0697E760

WritePtrs:          0697E768          Fifo_Flag:          04000018

writeHeaderPtr:          0697E768

Fifo1:          StartPtr:          06998C00          ReadPtr:          06998F10

WritePtrs:          06998F10          Fifo_Flag:          89800400

writeHeaderPtr:          06998F10

Fifo2:          StartPtr:          06827300          ReadPtr:          06827360

WritePtrs:          06827300          Fifo_Flag:          86000020

writeHeaderPtr:          06827300

--More--                          

Fifo3:          StartPtr:          06B14000          ReadPtr:          06B140A0

WritePtrs:          06B140A0          Fifo_Flag:          89800400

writeHeaderPtr:          06B140A0

Fifo4:          StartPtr:          069E0200          ReadPtr:          069E02A8

WritePtrs:          069E02A8          Fifo_Flag:          88800200

writeHeaderPtr:          069E02A8

Fifo5:          StartPtr:          06827268          ReadPtr:          06827268

WritePtrs:          06827268          Fifo_Flag:          82800008

writeHeaderPtr:          06827268

Fifo6:          StartPtr:          06ACC800          ReadPtr:          06ACC888

WritePtrs:          06ACC888          Fifo_Flag:          89400000

writeHeaderPtr:          06ACC888

Fifo7:          StartPtr:          06A3C000          ReadPtr:          06A3C018

WritePtrs:          06A3C018          Fifo_Flag:          89800400

writeHeaderPtr:          06A3C018

Fifo8:          StartPtr:          06AA6600          ReadPtr:          06AA66E0

WritePtrs:          06AA66E0          Fifo_Flag:          88800200

writeHeaderPtr:          06AA66E0

Fifo9:          StartPtr:          06990CD8          ReadPtr:          06990CD8

WritePtrs:          06990CD8          Fifo_Flag:          82800008

writeHeaderPtr:          06990CD8

Fifo10:          StartPtr:          06A06000          ReadPtr:          06A06168

WritePtrs:          06A06168          Fifo_Flag:          88800200

writeHeaderPtr:          06A06168

Fifo11:          StartPtr:          06AA6500          ReadPtr:          06AA6518

WritePtrs:          06AA6518          Fifo_Flag:          86800080

writeHeaderPtr:          06AA6518

Fifo12:          StartPtr:          06A33000          ReadPtr:          06A33000

WritePtrs:          06A33000          Fifo_Flag:          86800080

writeHeaderPtr:          06A33000

Fifo13:          StartPtr:          069E0100          ReadPtr:          069E0100

WritePtrs:          069E0100          Fifo_Flag:          86800080

writeHeaderPtr:          069E0100

Fifo14:          StartPtr:          00000000          ReadPtr:          00000000

WritePtrs:          00000000          Fifo_Flag:          00800000

writeHeaderPtr:          00000000

Fifo15:          StartPtr:          0697E760          ReadPtr:          0697E778

WritePtrs:          0697E778          Fifo_Flag:          84800020

--More--                           

writeHeaderPtr:          0697E778

===========================================================

Complete Board Id:0x401E

===========================================================

TATAIDC-L3-Sw#    

Bronze

3560 CPU - 88%

Hello Vivek,

After reviewing the information I see that the last issue with the stack now is that the CPU is getting high due to interrupts.

Now the interrupts are packets that need to be processes by the CPU, for example packets destined to one of the IP addresses of the switch, packet with TTL of 1, etc.

We need to verify what is causing the interrupts, in order to do that we need to verify if from the information of the "show ip traffic" the values of the "bad hop count" is increasing which tell us that there are too much packets coming with a TTL of 1.

The high cpu is due to CPU interrupts where traffic on the vlan is being directed to the CPU for processing. It has nothing to do with the processes the CPU is managing. Hence, the explanation of X/Y in the output. One (X) is total CPU utilization and the other is the amount that traffic sent to the CPU (Y) is utilizing.

Total CPU utilization = total CPU used by processes + total CPU used processing incoming packets

Show proc cpu shows X/Y or [Total CPU utilization]/[ total CPU used processing incoming packets]

Specifically, CPU utilization for five seconds: 85%/64%; one minute: 87%; five minutes: 87% shows 64% caused by packets hitting the Cpu and the difference is the amount used by processes (21%)

Sh contr cpu-inte:

cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray

----------------- ---------- ---------- ---------- ---------- ----------

sw forwarding     1851430181 0          0          0          0      

As you can see the sw forwarding queue has the largest difference in retrieved.

I am not going to keep writing because it will take more than 3 pages for a detailed troubleshooting (because high cpu needs to be TS on live) but please go through the next link that may help you to find the RC for high cpu utilization due to interrupts:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1026206

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1034726

Tip:

For sw forwarding:

++ Collect the output of show buffer pool RxQ6 packet (6 is the sw forwarding queu)

++ Check for input queue drops. We can filter the same with a switch#sh int | inc is up|drops

Haihua

Community Member

3560 CPU - 88%

hi Haihua, thank you for taking time to review the logs.

I think the CPU util is going high , because of PBR configuration.

I have couple of things planned, like upgrading IOS etc.

we will initiate this activity soon.

Hall of Fame Super Blue

3560 CPU - 88%

What is the switch's IOS version?

Community Member

3560 CPU - 88%

hi Leo, thank you for helping us out.

please find the details below.

Cisco IOS Software, C3560 Software (C3560-IPSERVICESK9-M), Version 12.2(55)SE1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Thu 02-Dec-10 07:17 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3560 boot loader

BOOTLDR: C3560 Boot Loader (C3560-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

Hall of Fame Super Blue

3560 CPU - 88%

Upgrade to 12.2(55)SE8.  A lot of CPU and memory hogs have been fixed since SE6.

Community Member

3560 CPU - 88%

ok, I had checked and got this version for 3560

c3560-ipservicesk9-mz.150-2.SE5.bin

Would this be sufficient?

thanks

VIvk

Hall of Fame Super Blue

Re: 3560 CPU - 88%

Would this be sufficient?

Upgrade to 12.2(55)SE8.

Your switch is capable of loading this IOS.

Community Member

3560 CPU - 88%

hi Leo, I had checked the IOS, available for 3560, the latest release is 15.0.2-SE5

please see below link for your referance.

http://software.cisco.com/download/release.html?mdfid=279486441&flowid=2534&softwareid=280805680&release=15.0.2-SE5&relind=AVAILABLE&rellifecycle=ED&reltype=latest

if you dont mind, please give me any link to download 12.2(55)SE8 version.

Community Member

Re: 3560 CPU - 88%

The recommended IOS is available from the link you already have. See the attached screenshot for the exact location.

Community Member

3560 CPU - 88%

thanks Everyone, I will initiate this upgrade on one of the spare L3 switch and test it out.

Hall of Fame Super Blue

Re: 3560 CPU - 88%

hi Leo, I had checked the IOS, available for 3560, the latest release is 15.0.2-SE5

Vivek,

Please let me spell this out for you.  You have an issue in regards to your CPU hog with your appliance.   Cisco has been attempting to fix this problem.  The IOS you are using ain't helping.  As a matter of fact, the IOS you are using is one of the versions where CPU and memory hogs occur. 

I have recommended 12.2(55)SE8 because I have tested this.  I have tested this for MONTHS.  And I have been recommending this version to other people.  Of these people, I've so far received one (1) bad comment. 

If you really want to use 15.2(1)E, and later, or even the scarier 15.0(2)SE5, then you do so because you know your network more than I do, however, these versions you have been highlighting are IOS versions I wouldn't even recommend even to my enemies.  Would the versions you've highlighted, 15.0(2)SE5 and/or 15.2(1)E, fix your problem?  I doubt it.  They are THAT BAD. 

if you dont mind, please give me any link to download 12.2(55)SE8 version.

Download 12.2(55)SE8 from here.

Community Member

3560 CPU - 88%

thanks Leo , for the constructive feedback, we will go for SE8 itself.

1358
Views
0
Helpful
16
Replies
CreatePlease to create content