04-27-2010 11:07 AM - edited 03-06-2019 10:50 AM
I'm having a problem with extremely high CPU utilization on a 6506-E and have not been able to determine the cause. Any help would be appreciated.
I suspect that the high CPU is due to a very large amount of multicast traffic routed to the device. The traffic is necessary in my environment, and I don't see this issue on any of my other 6500's.
sh ver:
System image file is "sup-bootdisk:s72033-ipservices_wan-mz.122-18.SXF14.bin"
cisco WS-C6506-E (R7000) processor (revision 1.1) with 983008K/65536K bytes of memory.
The CPU isn't that bad at the moment, but was at 90 - 99 percent earlier today, as you can see from the 72 hour view.
sh proc cpu hist:
3333333322222333332222222222222222222222222222223333344444
4444444466666333331111166666888884444455555666665555555555
100
90
80
70
60
50 *****
40 **********
30 ****************** ********** ********************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
4565766467668767583658988557647794565564855598777885654446
5674954939193824349469181962444489965707591413644994200884
100 *
90 *** * * * **
80 * * ** * **** # * *** **
70 * ** * *** * * **** * **# * * *****#*
60 ** *** ******** * **#*##**** ### ***** ** ##**### * *
50 *********#*#*#*#*# **####**** ###***********#######*** ***
40 *###*#**##########*##################################***##
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
99993 299889992 9
9999454964485455555532197699658544465554454445544444444544544444444549
100 **** *** *
90 **** ******* *
80 **** ******* *
70 *#** ******* *
60 *### ******* *
50 #### *****## *
40 #### #***### *
30 ####* #######* *
20 ####* *#######* *
10 ####** ** ** *******#######**** **** * ** * * * #
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
This seems very odd to me, I would have expected to see IP Input as the top CPU hog, but it is always Net Input instead. Anybody know why this would be?
sh proc cpu sorted:
CPU utilization for five seconds: 35%/33%; one minute: 38%; five minutes: 36%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
44 966912722670748050 0 1.35% 1.79% 1.77% 0 Net Input
263 1478109241164581828 126 0.31% 0.26% 0.27% 0 Port manager per
117 13692638842495457791 0 0.23% 0.19% 0.21% 0 IP Input
112 36825176 13804198 2667 0.07% 0.07% 0.07% 0 CDP Protocol
210 8964368 91039912 98 0.07% 0.08% 0.08% 0 PIM Process
289 18370844 209083288 87 0.07% 0.15% 0.15% 0 MLSM Process
6 108011628 8992916 12010 0.00% 0.16% 0.17% 0 Check heaps
7 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager
5 0 1 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req
4 3101884 17464655 177 0.00% 0.00% 0.00% 0 IP SNMP
8 0 2 0 0.00% 0.00% 0.00% 0 Timers
9 2409356 6783332 355 0.00% 0.00% 0.00% 0 ARP Input
13 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage
3 12 123 97 0.00% 0.00% 0.00% 0 RSVP
2 132608 10472696 12 0.00% 0.00% 0.00% 0 Load Meter
16 87124 52350331 1 0.00% 0.00% 0.00% 0 IPC Periodic Tim
12 92 384 239 0.00% 0.00% 0.00% 0 Entity MIB API
18 44319624 71883211 616 0.00% 0.01% 0.00% 0 IPC Seat Manager
10 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
11 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
21 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
1 32 44 727 0.00% 0.00% 0.00% 0 Chunk Manager
23 3022580 29728594 101 0.00% 0.00% 0.00% 0 EnvMon
24 2448 5236387 0 0.00% 0.00% 0.00% 0 Compute SRP rate
25 28672 922957 31 0.00% 0.00% 0.00% 0 EEM ED Syslog
26 36 457 78 0.00% 0.00% 0.00% 0 rf proxy rp agen
14 844 872728 0 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
The multicast traffic to the switch is coming in on vlan 500, and being consumed on vlan 58
interface Vlan500
ip address 192.168.254.29 255.255.255.252
ip nat outside
ip pim sparse-dense-mode
logging event link-status
end
interface Vlan58
ip address 10.20.10.10 255.255.255.0
ip nat inside
ip pim sparse-dense-mode
end
The high cpu is causing tons of drops on Vlan500, although they are only on the SVI interface, and not the physical interfac
show int vlan 601:
Input queue: 0/75/2282636580/2730974 (size/max/drops/flushes); Total output drops: 0
Any suggestions?
04-27-2010 11:55 AM
This output seems to be of the time when input queue drops were not incrementing thus you did not see ip input on top. At the time when process switching occurred, you may have seen ip input consuming more cpu.
natted packets are process switched and can be a cause here...
04-27-2010 12:36 PM
Even during heavy traffic Net Input is the number 1 process, and IP Input is a few down in the list.
04-27-2010 12:48 PM
Do you see any %SYS-3-CPUHOG errors in logs, tracebacks?.. .something like %SYS-3-CPUHOG: Task is running for (2000)msecs, more
than (2000)msecs (5359/4),process = Net Input.
That would provide tracebacks. Normally these are troubleshot at the time of problem if these logs are available, can be researched afterwards...
04-27-2010 01:03 PM
Also check:
Net Input |
|
http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_tech_note09186a00800a65d0.shtml
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: