Re: 3560 CPU - 88%

viveknathmangalat33 · ‎12-18-2013

hello, we are using a 3560 switch and the cpu utilization is constantly showing 88 %.

This switch is doing routing and leased lines are connected to service providers.

Is there any trouble shooting steps that can be done?

thanks

VIvk

Haihua Rong · ‎12-18-2013

Hello Vivek,

By having high CPU it means that from show proc cpu history the large spot of # means that the CPU experienced constant high load with averages up to 88%.

Changing the sdm template to routing may alleviate performance issues if we are getting TCAM problems since the switch is doing routing.
Check tcam with sh platf tcam utilization, sh logs and check if we are getting %PLATFORM_UCAST-4-PREFIX: messages.

Try to gather:

sh proc cpu | exc 0.00
sh contro cpu-int

Check the process(es) causing the switch to process-switching a lot of traffic.

Link
-------

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af3.shtml

Haihua

viveknathmangalat33 · ‎12-18-2013

hi Haihua, thank you for the help, I had pasted the output of the commands. Could you kindly take a look.

TATAIDC-L3-Sw#sh proc cpu | exc 0.00

CPU utilization for five seconds: 85%/64%; one minute: 87%; five minutes: 87%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

74 2428691633 632791155 3838 13.09% 14.72% 14.82% 0 HLFM address lea

94 23028294 23684199 972 0.15% 0.12% 0.11% 0 hpm counter proc

107 519 58 8948 1.27% 0.49% 0.10% 1 SSH Process

131 11454478 680366466 16 0.31% 0.12% 0.07% 0 Hulc LED Process

132 129347046 17550716 7369 0.31% 0.51% 0.51% 0 HL3U bkgrd proce

204 89892689 117800715 763 0.95% 0.42% 0.40% 0 Spanning Tree

309 41667912 27523089 1513 0.31% 0.28% 0.28% 0 SNMP ENGINE

TATAIDC-L3-Sw#

TATAIDC-L3-Sw#sh contro cpu-int

ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain

-------------------------------------------------------------------------

ASIC0 0 0 0 0 0 0

ASIC1 0 0 0 0 0 0

ASIC2 0 0 0 0 0 0

ASIC3 0 0 0 0 0 0

ASIC4 0 0 0 0 0 0

ASIC5 0 0 0 0 0 0

ASIC6 0 0 0 0 0 0

HOL Fix Counts

--------------

No Fixes: 0 Added: 0 In Use: 0 Both: 0

CPU Heartbeat Statistics

Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax

---------- ---------- ---------- ---------- ---------- ----------

471361363 0 0 0 0 1

Rx Delay

0 1 2 3 4

---------- ---------- ---------- ---------- ----------

471361363 0 0 0 0

AddlDelay AdvanceCnt

---------- ----------

0 0

Rx Retries by RetryCount

0 1 2 3 4 5 6

---------- ---------- ---------- ---------- ---------- ---------- ----------

471361363 0 0 0 0 0 0

7 8 9

---------- ---------- ----------

0 0 0

--More--

AddlRetry

----------

0

cpu-queue-frames retrieved dropped invalid hol-block stray

----------------- ---------- ---------- ---------- ---------- ----------

rpc 1 0 0 0 0

stp 55630558 0 0 0 0

ipc 0 0 0 0 0

routing protocol 531317387 0 0 0 0

L2 protocol 11709909 0 0 0 0

remote console 0 0 0 0 0

sw forwarding 1851430181 0 0 0 0

host 57685503 0 0 0 0

broadcast 402389844 0 0 0 0

cbt-to-spt 0 0 0 0 0

igmp snooping 44308332 0 0 0 0

icmp 615701075 0 0 0 0

logging 0 0 0 0 0

rpf-fail 0 0 0 0 0

dstats 0 0 0 0 0

cpu heartbeat 471361430 0 0 0 0

cpu-queue static inuse static added

----------------- ------------ ------------

rpc 0 0

stp 0 0

ipc 0 0

routing protocol 0 0

L2 protocol 0 0

remote console 0 0

sw forwarding 0 0

host 0 0

broadcast 0 0

cbt-to-spt 0 0

igmp snooping 0 0

icmp 0 0

--More--

logging 0 0

rpf-fail 0 0

dstats 0 0

cpu heartbeat 0 0

Supervisor ASIC receive-queue parameters

----------------------------------------

queue 0 maxrecevsize 0 pakhead 0 paktail 0

queue 1 maxrecevsize 7E0 pakhead 525AD98 paktail 5256CA0

queue 2 maxrecevsize 7E0 pakhead 4350598 paktail 5226E94

queue 3 maxrecevsize 7E0 pakhead 446A804 paktail 422BB54

queue 4 maxrecevsize 7E0 pakhead 5263324 paktail 43872E0

queue 5 maxrecevsize 7E0 pakhead 0 paktail 0

queue 6 maxrecevsize 7E0 pakhead 5336D68 paktail 445F248

queue 7 maxrecevsize 7E0 pakhead 441CBA8 paktail 4426A78

queue 8 maxrecevsize 7E0 pakhead 44305AC paktail 442C850

queue 9 maxrecevsize 7E0 pakhead 43C87B0 paktail 43C87B0

queue A maxrecevsize 7E0 pakhead 43BADE4 paktail 43C0F58

queue B maxrecevsize 7E0 pakhead 534AFE8 paktail 534C1F4

queue C maxrecevsize 7E0 pakhead 52AA8A8 paktail 43CF910

queue D maxrecevsize 7E0 pakhead 52A2210 paktail 43C8414

queue E maxrecevsize 0 pakhead 0 paktail 0

queue F maxrecevsize 7E0 pakhead 43B4FA4 paktail 528D2B0

Supervisor ASIC exception status

--------------------------------

Receive overrun 00000000 Transmit overrun 00000000

FrameSignatureErr 00000000 MicInitialize 00000000

BadFrameErr 00000000 LenExceededErr 00000000

BadJumboSegments 00000000

Supervisor ASIC Mic Registers

------------------------------

MicDirectPollInfo 80000200

MicIndicationsReceived 00000000

MicInterruptsReceived 00000000

MicPcsInfo 0000011F

MicPlbMasterConfiguration 00000000

MicRxFifosAvailable 00000040

--More--

MicRxFifosReady 0000BFBE

MicTimeOutPeriod: FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000

MicTransmFramesCopied 00000003

MicTxFifosAvailable 0000007F

MicConfiguration: Conf flag: 00000110 Interrupt Flag: 0000000A

MicReceiveFifoAssignmen Queue 0 - 7: 44444444 Queue 8 - 15:44444444

MicReceiveFramesReady: FrameAvailable: 00000041 frameAvaiMask: 00000000

MicException:

Exception_flag 00000000

Message-1 00000000

Message-2 00000000

Message-3 00000000

MicIntRxFifo:

ReadPtr 000001B0 WritePtr 000001B0

WHeadPtr 000001B0 TxFifoDepth C0000800

MicIntTxFifo:

ReadPtr 000004B0 WritePtr 000004B0

WHeadPtr 000004B0 TxFifoDepth C0000800

MicDecodeInfo:

Fifo0: address: 03FF4000 asic_num: 00000100

Fifo1: address: 03FF4400 asic_num: 00000101

MicTransmitFifoInfo:

Fifo0: StartPtrs: 0697E800 ReadPtr: 0697EE10

WritePtrs: 0697EE10 Fifo_Flag: 8A800800

Weights: 001E001E

Fifo1: StartPtrs: 06827400 ReadPtr: 068275B8

WritePtrs: 068275B8 Fifo_Flag: 89800400

Weights: 000A000A

MicReceiveFifoInfo:

Fifo0: StartPtr: 0697E760 ReadPtr: 0697E760

WritePtrs: 0697E768 Fifo_Flag: 04000018

writeHeaderPtr: 0697E768

Fifo1: StartPtr: 06998C00 ReadPtr: 06998F10

WritePtrs: 06998F10 Fifo_Flag: 89800400

writeHeaderPtr: 06998F10

Fifo2: StartPtr: 06827300 ReadPtr: 06827360

WritePtrs: 06827300 Fifo_Flag: 86000020

writeHeaderPtr: 06827300

--More--

Fifo3: StartPtr: 06B14000 ReadPtr: 06B140A0

WritePtrs: 06B140A0 Fifo_Flag: 89800400

writeHeaderPtr: 06B140A0

Fifo4: StartPtr: 069E0200 ReadPtr: 069E02A8

WritePtrs: 069E02A8 Fifo_Flag: 88800200

writeHeaderPtr: 069E02A8

Fifo5: StartPtr: 06827268 ReadPtr: 06827268

WritePtrs: 06827268 Fifo_Flag: 82800008

writeHeaderPtr: 06827268

Fifo6: StartPtr: 06ACC800 ReadPtr: 06ACC888

WritePtrs: 06ACC888 Fifo_Flag: 89400000

writeHeaderPtr: 06ACC888

Fifo7: StartPtr: 06A3C000 ReadPtr: 06A3C018

WritePtrs: 06A3C018 Fifo_Flag: 89800400

writeHeaderPtr: 06A3C018

Fifo8: StartPtr: 06AA6600 ReadPtr: 06AA66E0

WritePtrs: 06AA66E0 Fifo_Flag: 88800200

writeHeaderPtr: 06AA66E0

Fifo9: StartPtr: 06990CD8 ReadPtr: 06990CD8

WritePtrs: 06990CD8 Fifo_Flag: 82800008

writeHeaderPtr: 06990CD8

Fifo10: StartPtr: 06A06000 ReadPtr: 06A06168

WritePtrs: 06A06168 Fifo_Flag: 88800200

writeHeaderPtr: 06A06168

Fifo11: StartPtr: 06AA6500 ReadPtr: 06AA6518

WritePtrs: 06AA6518 Fifo_Flag: 86800080

writeHeaderPtr: 06AA6518

Fifo12: StartPtr: 06A33000 ReadPtr: 06A33000

WritePtrs: 06A33000 Fifo_Flag: 86800080

writeHeaderPtr: 06A33000

Fifo13: StartPtr: 069E0100 ReadPtr: 069E0100

WritePtrs: 069E0100 Fifo_Flag: 86800080

writeHeaderPtr: 069E0100

Fifo14: StartPtr: 00000000 ReadPtr: 00000000

WritePtrs: 00000000 Fifo_Flag: 00800000

writeHeaderPtr: 00000000

Fifo15: StartPtr: 0697E760 ReadPtr: 0697E778

WritePtrs: 0697E778 Fifo_Flag: 84800020

--More--

writeHeaderPtr: 0697E778

===========================================================

Complete Board Id:0x401E

===========================================================

TATAIDC-L3-Sw#

Haihua Rong · ‎12-19-2013

Hello Vivek,

After reviewing the information I see that the last issue with the stack now is that the CPU is getting high due to interrupts.

Now the interrupts are packets that need to be processes by the CPU, for example packets destined to one of the IP addresses of the switch, packet with TTL of 1, etc.

We need to verify what is causing the interrupts, in order to do that we need to verify if from the information of the "show ip traffic" the values of the "bad hop count" is increasing which tell us that there are too much packets coming with a TTL of 1.

The high cpu is due to CPU interrupts where traffic on the vlan is being directed to the CPU for processing. It has nothing to do with the processes the CPU is managing. Hence, the explanation of X/Y in the output. One (X) is total CPU utilization and the other is the amount that traffic sent to the CPU (Y) is utilizing.

Total CPU utilization = total CPU used by processes + total CPU used processing incoming packets

Show proc cpu shows X/Y or [Total CPU utilization]/[ total CPU used processing incoming packets]

Specifically, CPU utilization for five seconds: 85%/64%; one minute: 87%; five minutes: 87% shows 64% caused by packets hitting the Cpu and the difference is the amount used by processes (21%)

Sh contr cpu-inte:

cpu-queue-frames retrieved dropped invalid hol-block stray

----------------- ---------- ---------- ---------- ---------- ----------

sw forwarding 1851430181 0 0 0 0

As you can see the sw forwarding queue has the largest difference in retrieved.

I am not going to keep writing because it will take more than 3 pages for a detailed troubleshooting (because high cpu needs to be TS on live) but please go through the next link that may help you to find the RC for high cpu utilization due to interrupts:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1026206

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1034726

Tip:

For sw forwarding:

++ Collect the output of show buffer pool RxQ6 packet (6 is the sw forwarding queu)

++ Check for input queue drops. We can filter the same with a switch#sh int | inc is up|drops

Haihua

viveknathmangalat33 · ‎12-20-2013

hi Haihua, thank you for taking time to review the logs.

I think the CPU util is going high , because of PBR configuration.

I have couple of things planned, like upgrading IOS etc.

we will initiate this activity soon.

Leo Laohoo · ‎12-18-2013

What is the switch's IOS version?

viveknathmangalat33 · ‎12-18-2013

hi Leo, thank you for helping us out.

please find the details below.

Cisco IOS Software, C3560 Software (C3560-IPSERVICESK9-M), Version 12.2(55)SE1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 02-Dec-10 07:17 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3560 boot loader

BOOTLDR: C3560 Boot Loader (C3560-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

Leo Laohoo · ‎12-18-2013

Upgrade to 12.2(55)SE8. A lot of CPU and memory hogs have been fixed since SE6.

viveknathmangalat33 · ‎12-18-2013

ok, I had checked and got this version for 3560

c3560-ipservicesk9-mz.150-2.SE5.bin

Would this be sufficient?

thanks

VIvk

Leo Laohoo · ‎12-18-2013

Would this be sufficient?

Upgrade to 12.2(55)SE8.

Your switch is capable of loading this IOS.

viveknathmangalat33 · ‎12-19-2013

hi Leo, I had checked the IOS, available for 3560, the latest release is 15.0.2-SE5

please see below link for your referance.

http://software.cisco.com/download/release.html?mdfid=279486441&flowid=2534&softwareid=280805680&release=15.0.2-SE5&relind=AVAILABLE&rellifecycle=ED&reltype=latest

if you dont mind, please give me any link to download 12.2(55)SE8 version.

James Neilson · ‎12-19-2013

The recommended IOS is available from the link you already have. See the attached screenshot for the exact location.

viveknathmangalat33 · ‎12-19-2013

thanks Everyone, I will initiate this upgrade on one of the spare L3 switch and test it out.

Leo Laohoo · ‎12-20-2013

hi Leo, I had checked the IOS, available for 3560, the latest release is 15.0.2-SE5

Vivek,

Please let me spell this out for you. You have an issue in regards to your CPU hog with your appliance. Cisco has been attempting to fix this problem. The IOS you are using ain't helping. As a matter of fact, the IOS you are using is one of the versions where CPU and memory hogs occur.

I have recommended 12.2(55)SE8 because I have tested this. I have tested this for MONTHS. And I have been recommending this version to other people. Of these people, I've so far received one (1) bad comment.

If you really want to use 15.2(1)E, and later, or even the scarier 15.0(2)SE5, then you do so because you know your network more than I do, however, these versions you have been highlighting are IOS versions I wouldn't even recommend even to my enemies. Would the versions you've highlighted, 15.0(2)SE5 and/or 15.2(1)E, fix your problem? I doubt it. They are THAT BAD.

if you dont mind, please give me any link to download 12.2(55)SE8 version.

Download 12.2(55)SE8 from here.

viveknathmangalat33 · ‎12-20-2013

thanks Leo , for the constructive feedback, we will go for SE8 itself.