6506 high CPU problem

Unanswered Question
Apr 27th, 2010

I'm having a problem with extremely high CPU utilization on a 6506-E and have not been able to determine the cause.  Any help would be appreciated.


I suspect that the high CPU is due to a very large amount of multicast traffic routed to the device.  The traffic is necessary in my environment, and I don't see this issue on any of my other 6500's. 


sh ver:

System image file is "sup-bootdisk:s72033-ipservices_wan-mz.122-18.SXF14.bin"

cisco WS-C6506-E (R7000) processor (revision 1.1) with 983008K/65536K bytes of memory.


The CPU isn't that bad at the moment, but was at 90 - 99 percent earlier today, as you can see from the 72 hour view.


sh proc cpu hist:


     3333333322222333332222222222222222222222222222223333344444
     4444444466666333331111166666888884444455555666665555555555
100
90
80
70
60
50                                                       *****
40                                                  **********
30  ******************     **********     ********************
20  **********************************************************
10  **********************************************************
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5

               CPU% per second (last 60 seconds)


     4565766467668767583658988557647794565564855598777885654446
     5674954939193824349469181962444489965707591413644994200884
100                                  *
90                       ***        *       *   *    **
80      *    *  **   *   ****       #       *   ***  **
70    * **   * *** * *   ****  *  **#  *    *   *****#*
60   ** *** ******** * **#*##**** ### ***** **  ##**### *    *
50  *********#*#*#*#*# **####**** ###***********#######*** ***
40  *###*#**##########*##################################***##
30  ##########################################################
20  ##########################################################
10  ##########################################################
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5

               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%


     99993               299889992                                        9
     9999454964485455555532197699658544465554454445544444444544544444444549
100  ****                     ***                                         *
90  ****                 *******                                         *
80  ****                 *******                                         *
70  *#**                 *******                                         *
60  *###                 *******                                         *
50  ####                 *****##                                         *
40  ####                 #***###                                         *
30  ####*                #######*                                        *
20  ####*               *#######*                                        *
10  ####** **  ** *******#######****   ****  *   **        *  *        * #
    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
              0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%



This seems very odd to me, I would have expected to see IP Input as the top CPU hog, but it is always Net Input instead.  Anybody know why this would be?


sh proc cpu sorted:

CPU utilization for five seconds: 35%/33%; one minute: 38%; five minutes: 36%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  44    966912722670748050          0  1.35%  1.79%  1.77%   0 Net Input
263   1478109241164581828        126  0.31%  0.26%  0.27%   0 Port manager per
117  13692638842495457791          0  0.23%  0.19%  0.21%   0 IP Input
112    36825176  13804198       2667  0.07%  0.07%  0.07%   0 CDP Protocol
210     8964368  91039912         98  0.07%  0.08%  0.08%   0 PIM Process
289    18370844 209083288         87  0.07%  0.15%  0.15%   0 MLSM Process
   6   108011628   8992916      12010  0.00%  0.16%  0.17%   0 Check heaps
   7           0         1          0  0.00%  0.00%  0.00%   0 Pool Manager
   5           0         1          0  0.00%  0.00%  0.00%   0 PF Redun ICC Req
   4     3101884  17464655        177  0.00%  0.00%  0.00%   0 IP SNMP
   8           0         2          0  0.00%  0.00%  0.00%   0 Timers
   9     2409356   6783332        355  0.00%  0.00%  0.00%   0 ARP Input
  13           0         1          0  0.00%  0.00%  0.00%   0 IFS Agent Manage
   3          12       123         97  0.00%  0.00%  0.00%   0 RSVP
   2      132608  10472696         12  0.00%  0.00%  0.00%   0 Load Meter
  16       87124  52350331          1  0.00%  0.00%  0.00%   0 IPC Periodic Tim
  12          92       384        239  0.00%  0.00%  0.00%   0 Entity MIB API
  18    44319624  71883211        616  0.00%  0.01%  0.00%   0 IPC Seat Manager
  10           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
  11           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit
  21           0         2          0  0.00%  0.00%  0.00%   0 Dialer event
   1          32        44        727  0.00%  0.00%  0.00%   0 Chunk Manager
  23     3022580  29728594        101  0.00%  0.00%  0.00%   0 EnvMon
  24        2448   5236387          0  0.00%  0.00%  0.00%   0 Compute SRP rate
  25       28672    922957         31  0.00%  0.00%  0.00%   0 EEM ED Syslog
  26          36       457         78  0.00%  0.00%  0.00%   0 rf proxy rp agen
  14         844    872728          0  0.00%  0.00%  0.00%   0 IPC Dynamic Cach



The multicast traffic to the switch is coming in on vlan 500, and being consumed on vlan 58


interface Vlan500

ip address 192.168.254.29 255.255.255.252
ip nat outside
ip pim sparse-dense-mode
logging event link-status
end

interface Vlan58
ip address 10.20.10.10 255.255.255.0
ip nat inside
ip pim sparse-dense-mode

end


The high cpu is causing tons of drops on Vlan500, although they are only on the SVI interface, and not the physical interfac


show int vlan 601:


  Input queue: 0/75/2282636580/2730974 (size/max/drops/flushes); Total output drops: 0



Any suggestions?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
anumishr Tue, 04/27/2010 - 11:55

This output seems to be of the time when input queue drops were not incrementing thus you did not see ip input on top. At the time when process switching occurred, you may have seen ip input consuming more cpu.

natted packets are process switched and can be a cause here...

David Trocki Tue, 04/27/2010 - 12:36

Even during heavy traffic Net Input is the number 1 process, and IP Input is a few down in the list.

anumishr Tue, 04/27/2010 - 12:48

Do you see any %SYS-3-CPUHOG errors in logs, tracebacks?.. .something like %SYS-3-CPUHOG: Task is running for (2000)msecs, more
than (2000)msecs (5359/4),process = Net Input.

That would provide tracebacks. Normally these are troubleshot at the time of problem if these logs are available, can be researched afterwards...

anumishr Tue, 04/27/2010 - 13:03

Also check:

Net Input

  • Handles otherwise unknown packets. This is done at process level so that input queuing comes into play. If you operate at interrupt level, you could very easily lock up the router.

  • Handles some known protocols which you may decide should be offered to bridging. In this case, net_input either sends the packet to NULL, or bridges it.

http://www.cisco.com/en/US/products/sw/iosswrel/ps1828/products_tech_note09186a00800a65d0.shtml

Actions

This Discussion