Hey Josh, Thank you for the

JCedeno86 · ‎04-09-2015

All,

Recently my 4507 has had an issue where the CPU is in the upper 80-90 percent area but yesterday it hit 99 and stayed there for a couple hours until I could narrow things down. During that time the network came to crawl causing applications to time out which of course the business wasn't happy about. The interesting thing is that when sifting through CPU statistics I found a large about of Packets Received at CPU per Input Interface coming from primarily 5 ports. I grabbed there MAC addresses and shut them down. Once shut, the CPU came down to the 40% range and all was good again. When I traced back those MACs I found one common denominator, Windows 8. I've scanned those machines for malware and they came back as clean. I'm not sure what else to do at this point but continue to montior these devices closely as the day starts. If anyone has any suggestions or ideas I'd be more than happy to give them a try.

Aninda Chatterjee · ‎04-09-2015

Hello,

It will be difficult to narrow down the exact cause of the high CPU without troubleshooting it live. Typically, we'd want to see what kind of packets are being punted to the CPU (or see what process is consuming a lot of CPU cycles and so on). There are ways to capture such packets but naturally, this is only useful when the issue actually occurs.

Are these still machines still on the network? If so and if you starting seeing this problem again, I'd recommend following the document below for collecting outputs that would help identify the cause:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html

Regards,

Aninda

Madhukrishnan Gopinathan Nair · ‎04-09-2015

If they re-occur try to capture them with SPAN to see what they are sending?

Also 4500 has built in capture tools to capture also. You could turn the below on to capture.

debug platfrom packet all buffer

show platform cpu packet buffered

Hope this helps and kindly grade the posts which are usful.

Thanks,

Madhu.

JCedeno86 · ‎04-21-2015

Thank you to those who replied. The issue is coming back after I restarted those windows 8 machines 12 days ago to get my network going again. Here's what I can see:

sh proc cpu hist

9999888888888877779999988888999999999988888999999999999999
9999555552222288883333388888333330000077777999999999999999
100 **** ***************
90 ********* ****************************************
80 **********************************************************
70 **********************************************************
60 **********************************************************
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

9999999999999999999999999999999999999999999999999999999999
9999999999999999999997899969999977899999799979999989999999
100 **********************************************************
90 ##################################################*#######
80 ##########################################################
70 ##########################################################
60 ##########################################################
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

9999998878887889999999999787776676767679676767776666667667766667667776
9999932372137137999999999832279939388791829097206987541982156450572886
100 ***** **********
90 #***** ***#####** *
80 #####***********#########** * * ** * **
70 #####**********##########**************************** ******* ********
60 ##########################********************************************
50 #######################################################***############
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

The following has some omitted sections. Also I tried to format best I could

sh platform health
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
GalChassisVp-review 3.00 1.47 10 677 100 500 1 1 1 513:48
S2w-JobEventSchedule 10.00 0.57 10 7 100 500 0 0 0 295:44
Stub-JobEventSchedul 10.00 4.40 10 11 100 500 5 5 3 1700:40
Lj-poll 1.00 0.01 2 1 100 500 0 0 0 10:04
StatValueMan Update 1.00 0.02 1 1 100 500 0 0 0 49:34
Pim-review 0.10 0.03 1 1 100 500 0 0 0 3:18
Ebm-host-review 1.00 0.00 8 4 100 500 0 0 0 28:29
Ebm-host-util-review 1.00 0.00 10 0 100 500 0 0 0 0:04
Ebm-port-review 0.10 0.00 1 1 100 500 0 0 0 0:14
Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:00
EbmHostRedundancyMan 2.00 0.00 20 0 100 500 0 0 0 0:00
Acl-Flattener 1.00 0.00 10 0 100 500 0 0 0 0:00
IrmFibThrottler Thro 2.00 0.00 7 5 100 500 0 0 0 4:45
GalChassisVp Ondeman 2.00 0.00 2 0 100 500 0 0 0 0:00
KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 3:32
KxAclPathMan update 2.00 0.00 10 1 100 500 0 0 0 1:40
KxAclPathMan reprogr 1.00 0.00 2 0 100 500 0 0 0 0:00
TagMan-RecreateMtegR 1.00 0.00 10 6 100 500 0 0 0 0:00
K2CpuMan Review 30.00 68.00 30 110 100 500 102 96 64 15659:01
K2AccelPacketMan: Tx 10.00 1.60 20 1 100 500 1 1 1 691:38

---------

K2PortMan Review 3.00 2.86 15 11 100 500 3 3 2 849:43

---------

K2PacketBufMonitor-P 3.00 2.15 10 1 100 500 2 2 1 859:45

----------

PacketBufRaw 25219.50 100% 25219.50 100%
PacketBufRawJumbo 184.29 100% 184.29 100%

---------

PimPhyports 1023.43 24% 254.83 100%
PimPorts 906.25 31% 283.65 100%
PimModules 160.00 2% 3.75 100%
PimSlots 6.00 2% 0.16 100%
PimChassis 38.12 6% 2.38 100%
PimQuack 0.35 0% 0.00 0%
EbmVlans 5024.00 0% 14.71 100%
EbmVlanGroupEntrys 7679.88 0% 1.64 92%
EbmPorts 328.00 30% 99.61 100%

-----------

CommandTables 48.00 22% 10.59 100%

----------

sh platform cpu packet statistics all

-------------

Packets Received by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp 404542050 253 214 167 157
L2/L3Control 663770373 592 585 465 435
Host Learning 2602651 0 0 0 0
L3 Fwd High 0 0 0 0 0
L3 Fwd Medium 4 0 0 0 0
L3 Fwd Low 14751384 15 2 1 1
L2 Fwd High 1021 0 0 0 0
L2 Fwd Medium 2039 0 0 0 0
L2 Fwd Low 290804364 258 223 173 155
L3 Rx High 4230 0 0 0 0
L3 Rx Low 281866 2 0 0 0
RPF Failure 0 0 0 0 0
ACL fwd(snooping) 0 0 0 0 0
ACL log, unreach 840233 0 0 0 0
ACL sw processing 0 0 0 0 0
MTU Fail/Invalid 0 0 0 0 0

Packets Dropped by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp 0 0 0 0 0
L2/L3Control 545781 0 0 0 0
Host Learning 0 0 0 0 0
L3 Fwd High 0 0 0 0 0
L3 Fwd Medium 0 0 0 0 0
L3 Fwd Low 783 0 0 0 0
L2 Fwd High 0 0 0 0 0
L2 Fwd Medium 0 0 0 0 0
L2 Fwd Low 2712613 0 0 0 0
L3 Rx High 0 0 0 0 0
L3 Rx Low 0 0 0 0 0
RPF Failure 0 0 0 0 0
ACL fwd(snooping) 0 0 0 0 0
ACL log, unreach 0 0 0 0 0
ACL sw processing 0 0 0 0 0
MTU Fail/Invalid 0 0 0 0 0

----------

Packets Received at CPU per Input Interface

Interface Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Gi3/1 0 0 0 0 0
Gi3/2 34076 0 0 0 0
Gi3/3 46901 0 0 0 0
Gi3/4 0 0 0 0 0
Gi3/5 0 0 0 0 0
Gi3/6 24226740 0 0 0 0
Gi3/7 0 0 0 0 0
Gi3/8 123967 0 0 0 0
Gi3/9 87076 1 0 0 0
Gi3/10 345651 0 0 0 0
Gi3/11 92107 0 0 0 0
Gi3/12 174803 0 0 0 0

-------- I would show more but I think you get my drift. Gi3/6 is one of my Windows 8 PCs. But since that was recently restarted here is another batch with two other windows 8 PCs

---------

Gi3/38 50376655 191 158 124 112
Gi3/39 31470 0 0 0 0
Gi3/40 52393 0 0 0 0
Gi3/41 0 0 0 0 0
Gi3/42 183671 0 0 0 0
Gi3/43 87356 0 0 0 0
Gi3/44 2741 0 0 0 0
Gi3/45 159 0 0 0 0
Gi3/46 24 0 0 0 0
Gi3/47 164477 0 0 0 0
Gi3/48 0 0 0 0 0
Gi4/1 89748 0 0 0 0
Gi4/2 51001 0 0 0 0
Gi4/3 103379 0 0 0 0
Gi4/4 0 0 0 0 0
Gi4/5 111986578 237 216 173 160

Let me know if you guys have any ideas. I will keep the CPU running as long as I can if you need any other stats. Thank you in advanced.

Josh

Aninda Chatterjee · ‎04-21-2015

Hey Josh,

Thank you for the outputs. Can you take a CPU capture and paste that as a text file here please?

debug platform packet all receive buffer

show platform cpu packet buffered

Regards,

Aninda

High CPU Utilization (C4507R) caused by Windows 8