Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Community Member

Cisco 3640 with NW-16ESW, high CPU

We have a 3640 with a NW-16ESW that has high CPU when routing between VLANs.  We get about 75% CPU utilization when we are only doing about 5000pps with a bandwidth of about 7Mb going through the router.  I'm thinking we should be able to handle > 40K pps with this router.  Why is the router not handling the traffic better?

Most of the traffic is SCTP going between F3/2 Vlan 61 and F3/15 Vlan 1000.

Here's some relevent info:

Router2#show ver 
Cisco IOS Software, 3600 Software (C3640-IK9O3S-M), Version 12.4(25b), RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 12-Aug-09 12:52 by prod_rel_team

ROM: System Bootstrap, Version 11.1(20)AA2, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

Router2 uptime is 12 weeks, 1 day, 21 hours, 28 minutes
System returned to ROM by power-on
System restarted at 15:20:57 CST Sat Feb 27 2010
System image file is "flash:c3640-ik9o3s-mz.124-25b.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 3640 (R4700) processor (revision 0x00) with 121856K/9216K bytes of memory.
Processor board ID 19590546
R4700 CPU at 100MHz, Implementation 33, Rev 1.0
18 FastEthernet interfaces
DRAM configuration is 64 bits wide with parity disabled.
125K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
no service password-encryption
!
hostname Router2
!
ip cef
!
no spanning-tree vlan 1
no spanning-tree vlan 2
no spanning-tree vlan 11
no spanning-tree vlan 20
no spanning-tree vlan 21
no spanning-tree vlan 30
no spanning-tree vlan 50
no spanning-tree vlan 51
no spanning-tree vlan 52
no spanning-tree vlan 60
no spanning-tree vlan 61
no spanning-tree vlan 62
no spanning-tree vlan 70
no spanning-tree vlan 71
no spanning-tree vlan 1000
no spanning-tree vlan 1001
!
interface Loopback0
ip address 192.168.1.2 255.255.255.255
!
interface FastEthernet1/0
ip address 172.22.102.247 255.255.255.0
duplex auto
speed auto
!
interface FastEthernet2/0
ip address 10.1.1.5 255.255.255.252
duplex auto
speed auto
!
interface FastEthernet3/1
switchport access vlan 52
load-interval 30
!
interface FastEthernet3/2
switchport access vlan 61
load-interval 30
!
interface FastEthernet3/3
switchport access vlan 62
load-interval 30
no cdp enable
!
interface FastEthernet3/4
switchport access vlan 20
load-interval 30
duplex full
speed 100
!        
interface FastEthernet3/6
switchport access vlan 71
load-interval 30
duplex full
speed 100
!
interface FastEthernet3/8
switchport access vlan 60
load-interval 30
duplex full
speed 100
!
interface FastEthernet3/9
switchport access vlan 21
load-interval 30
duplex full
speed 100
spanning-tree portfast
!
interface FastEthernet3/10
switchport access vlan 51
load-interval 30
duplex full
speed 100
!
interface FastEthernet3/11
switchport access vlan 30
load-interval 30
!
interface FastEthernet3/12
switchport access vlan 70
load-interval 30
duplex full
speed 100
!
interface FastEthernet3/13
switchport access vlan 2
load-interval 30
duplex full
speed 100
!
interface FastEthernet3/14
switchport access vlan 1001
load-interval 30
no keepalive
!
interface FastEthernet3/15
switchport trunk native vlan 1000
switchport trunk allowed vlan 1,21,1000,1002-1005
switchport mode trunk
load-interval 30
duplex full
speed 100
!
interface Vlan1
no ip address
shutdown
!
interface Vlan2
ip address 10.100.2.3 255.255.255.128
standby ip 10.100.2.1
standby timers msec 500 1
standby priority 110
standby preempt
!
interface Vlan11
ip address 10.205.1.3 255.255.255.128
glbp 0 ip 10.205.1.1
!
interface Vlan20
ip address 10.203.1.3 255.255.255.0
standby ip 10.203.1.1
standby timers 1 2
!
interface Vlan21
ip address 10.203.2.3 255.255.255.0
ip helper-address 10.206.1.35
ip helper-address 10.206.1.37
standby ip 10.203.2.1
standby timers 1 2
!
interface Vlan30
ip address 8.10.109.1 255.255.255.0
!
interface Vlan51
ip address 10.206.1.3 255.255.255.128
standby ip 10.206.1.1
!
interface Vlan52
ip address 8.10.2.131 255.255.255.128
ip helper-address 8.10.3.35
ip helper-address 8.10.3.37
standby ip 8.10.2.129
!        
interface Vlan60
ip address 10.204.1.3 255.255.255.128
glbp 0 ip 10.204.1.1
!
interface Vlan61
ip address 8.10.103.1 255.255.255.0
!
interface Vlan62
ip address 8.10.104.1 255.255.255.0
!
interface Vlan70
ip address 10.160.207.3 255.255.255.128
ip helper-address 10.204.1.19
ip helper-address 10.204.1.21
standby ip 10.160.207.1
standby timers msec 300 1
!
interface Vlan71
ip address 10.204.4.3 255.255.255.128
standby ip 10.204.4.1
standby timers msec 300 1
!
interface Vlan1000
ip address 10.1.1.10 255.255.255.252
!
interface Vlan1001
ip address 10.1.1.21 255.255.255.252
!
router eigrp 100
redistribute static
redistribute ospf 1
passive-interface default
no passive-interface FastEthernet2/0
no passive-interface Vlan1000
network 8.0.0.0
network 10.0.0.0
default-metric 100000 10 255 1 1514
no auto-summary
!
router ospf 1
router-id 192.168.1.2
log-adjacency-changes
timers throttle spf 5 100 5000
timers throttle lsa all 5 100 5000
timers lsa arrival 50
redistribute eigrp 100 subnets
passive-interface default
no passive-interface Vlan1001
network 10.1.1.20 0.0.0.3 area 0
network 10.100.2.0 0.0.0.255 area 0
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 172.22.102.1
ip route 172.30.18.0 255.255.255.0 8.10.103.4
!
!
snmp-server community public RO
control-plane
!
mgcp behavior g729-variants static-pt
line con 0
line aux 0
line vty 0 4
exec-timeout 3660 0
logging synchronous
login
terminal-type mon

Router2#sho proc cpu sort
CPU utilization for five seconds: 86%/70%; one minute: 75%; five minutes: 75%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
188     7614256    884556       8607  4.99%  0.65%  0.47%   0 SNMP ENGINE     
  16   456989136 106171824       4304  2.94%  2.94%  2.96%   0 ARP Input       
  64     4304512   1686473       2552  2.37%  0.30%  0.20%   0 IP SNMP         
  59   115965776  51699731       2243  1.80%  0.76%  0.69%   0 IP Input        
   5    34986620   1552834      22530  1.31%  0.26%  0.18%   0 Check heaps     
183     1719988    848105       2028  0.98%  0.13%  0.07%   0 PDU DISPATCHER  
184    57672684  54767397       1053  0.49%  0.58%  0.57%   0 HSRP (Standby)  
  54    85767848   7410211      11574  0.49%  0.44%  0.43%   0 tCOUNTER        
  52    60119908  34976406       1718  0.40%  0.36%  0.35%   0 IGMP Snooping Re
196        1260       557       2262  0.32%  0.02%  0.00% 130 Virtual Exec    
  95     9096200  73609914        123  0.16%  0.19%  0.18%   0 RBSCP Background
112     1945164   7420532        262  0.08%  0.02%  0.00%   0 RUDPV1 Main Proc
  79     4361616  28971622        150  0.08%  0.07%  0.08%   0 SSS Feature Time
  30     1405448   7409457        189  0.08%  0.01%  0.00%   0 TTY Background  
  47    12718124   1551488       8197  0.08%  0.08%  0.08%   0 Compute load avg
107    21986460  12413900       1771  0.08%  0.15%  0.16%   0 CEF process     
192     7355868   6431667       1143  0.08%  0.04%  0.06%   0 IP-EIGRP: HELLO 
  27     4455184   8156560        546  0.08%  0.03%  0.05%   0 Net Background  
  80       55900     59561        938  0.08%  0.00%  0.00%   0 TCP Timer       

Rest are 0%

Router2#sho interfaces switching
FastEthernet2/0
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs    1606754      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    1606296  118888059    1604913  118760471
            Cache misses          0          -          -          -
                    Fast   80189881  890053776  393675494  439905742
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        520      31200        520      31200
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  CDP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     123629   42775634     123623   42773558
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0     741581   44494860
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.

FastEthernet3/1

    All statistics for this interface are zero.
FastEthernet3/2
          Throttle count          0
                   Drops         RP       3909         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

    NOTE: all counts are cumulative and reset only after a reload.
FastEthernet3/3
          Throttle count          0
                   Drops         RP      10713         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     123612   47219784          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
FastEthernet3/4
          Throttle count          0
                   Drops         RP      21167         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     247207   13596385          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

FastEthernet3/8
          Throttle count          0
                   Drops         RP       2242         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

FastEthernet3/9
          Throttle count          0
                   Drops         RP       1257         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

FastEthernet3/10
          Throttle count          0
                   Drops         RP      24166         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     247218   13596990          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

FastEthernet3/11
          Throttle count          0
                   Drops         RP          1         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

FastEthernet3/12
          Throttle count          0
                   Drops         RP        241         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     295624   27494699          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

FastEthernet3/13
          Throttle count          0
                   Drops         RP        144         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

FastEthernet3/14
          Throttle count          0
                   Drops         RP        518         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

FastEthernet3/15
          Throttle count          0
                   Drops         RP       7176         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          0      Drops          0

Vlan2
          Throttle count          0
                   Drops         RP         62         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs   12917815      Drops         59

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     311090   20013366     310324   19497064
            Cache misses          0          -          -          -
                    Fast    2699440  162096376    1766101  105966136
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process   12856531  771391860       3754     225240
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         21       1680          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan20
          Throttle count          0
                   Drops         RP         26         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs  170864020      Drops         25

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    7668910  475704532    7422530  460196842
            Cache misses          0          -          -          -
                    Fast  389284739    9986005   79179155  653772410
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process  171095975 1675823908      62888    3773280
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     234729   14138460          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan21
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs      10036      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    7647297  473593702    7422692  460212500
            Cache misses          0          -          -          -
                    Fast  176622731 2839806715  193103180  467221336
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       9998     602140    2710126  162607560
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        233     329462          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan30
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs    2808317      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         56       3584          0          0
            Cache misses          0          -          -          -
                    Fast          0          0   20274149 1589339948
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    2808317  168558948    6003049  360182940
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan51
          Throttle count          0
                   Drops         RP    2927720         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs   15772438      Drops    2815542

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    6429109  910157615    2470549  153179634
            Cache misses          0          -          -          -
                    Fast  209354743 3774049520  176645242 2850922825
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process   15772366  946350016      23823    1429380
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       5385    1559364          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan60
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs    3341636      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    7719393  902981695    2545349  277617298
            Cache misses          0          -          -          -
                    Fast  329904518 2733552572  340631949 3701461606
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    3341637  200498220       7082     424920
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     247339   14842560          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan61
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs      84450      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    9204817  680176034     222668  123284020
            Cache misses          0          -          -          -
                    Fast   70095026 3973635844  664982555 2069783134
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      84450    5067032      78206    4692360
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan62
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs     101046      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     222524  123278296     222541  123279486
            Cache misses          0          -          -          -
                    Fast  473026379 2436528465 1248128210 1527019850
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     101046    6062768      74638    4478280
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan70
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs    7092096      Drops      13968

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          2        120
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan71
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs    5693699      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    6055858  377177859    5966708  369935886
            Cache misses          0          -          -          -
                    Fast       5711     544967          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    5693699  341621940        932      55920
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         51       4080          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
    NOTE: all counts are cumulative and reset only after a reload.
Vlan1000
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs       1260      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    1602437  118623019    4328858  309473290
            Cache misses          0          -          -          -
                    Fast 1932870753  804829182  615725003 2786812294
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        523      31380        527      31620
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Vlan1001
          Throttle count          0
                   Drops         RP        205         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs        563      Drops          0

    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    3540867 1012635054     769858   76510068
            Cache misses          0          -          -          -
                    Fast  111272160 3163087992    1977301  125917204
               Auton/SSE          0          0          0          0

    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        533      31980        540      32400
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      12315     948136          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

Router2#show ip cef detail

IP CEF with switching (Table Version 1258), flags=0x0
  218 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 2
  5 instant recursive resolutions, 0 used background process
  221 leaves, 39 nodes, 74152 bytes, 1115 inserts, 894 invalidations
  0 load sharing elements, 0 bytes, 0 references
  universal per-destination load sharing algorithm, id 1782FBB1
  2(0) CEF resets, 144 revisions of existing leaves
  Resolution Timer: Exponential (currently 1s, peak 1s)
  146 in-place/0 aborted modifications
  refcounts:  11066 leaf, 10240 node

  Table epoch: 0 (221 entries at this epoch)

Adjacency Table has 141 adjacencies

Router2#show align
Alignment data for:
3600 Software (C3640-IK9O3S-M), Version 12.4(25b), RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Compiled Wed 12-Aug-09 12:52 by prod_rel_team

No alignment data has been recorded.

Total Spurious Accesses 2, Recorded 1

Address  Count  Traceback
      24      2  0x60315230  0x604EF18C  0x6050B52C  0x605AEC8C
                 0x605AEC70

Router2#show inte stats
FastEthernet1/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor    1130688  114851873     577963   84852860
             Route cache      13605    1463761    7093825 1429828267
                   Total    1144293  116315634    7671788 1514681127
FastEthernet2/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor      82362    7701122     117635    9812316
             Route cache        174      44028    6902103  935404127
                   Total      82536    7745150    7019738  945216443

FastEthernet3/2
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor  444153254  990538626  495449540   94331715
             Route cache          0          0          0          0
                   Total  444153254  990538626  495449540   94331715
FastEthernet3/3
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor  300779026 3717677162  310480461 2513167776
             Route cache          0          0          0          0
                   Total  300779026 3717677162  310480461 2513167776
FastEthernet3/4
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor   19920203 1802497784     360403   25485486
             Route cache          0          0          0          0
                   Total   19920203 1802497784     360403   25485486

FastEthernet3/6
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor    2723632  178469348    1175480   79254804
             Route cache          0          0          0          0
                   Total    2723632  178469348    1175480   79254804
FastEthernet3/8
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor    7571699 1499308330     129790   16567181
             Route cache          0          0          0          0
                   Total    7571699 1499308330     129790   16567181
FastEthernet3/9
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor    3848919 1277090111    5072160  572453048
             Route cache          0          0          0          0
                   Total    3848919 1277090111    5072160  572453048
FastEthernet3/10
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor    6294469  783755818    3973472 1286991286
             Route cache          0          0          0          0
                   Total    6294469  783755818    3973472 1286991286
FastEthernet3/11
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor     113194    7244564     358735   24649757
             Route cache          0          0          0          0
                   Total     113194    7244564     358735   24649757
FastEthernet3/14
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor   16719928 1470627500      43080    5965025
             Route cache          0          0          0          0
                   Total   16719928 1470627500      43080    5965025
FastEthernet3/15
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor  806356059 2629024827  758498534 1550085454
             Route cache          0          0          0          0
                   Total  806356059 2629024827  758498534 1550085454
Vlan20
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor   12283275  736996500     354353   21966890
             Route cache    6902185  938906470        164      39008
                   Total   19185460 1675902970     354517   22005898
Vlan21
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor         55       3400     482598   29661584
             Route cache    3848913 1261699109    4217966  497819614
                   Total    3848968 1261702509    4700564  527481198
Vlan30
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor     113159    6789540     352842   21170520
             Route cache          0          0          0          0
                   Total     113159    6789540     352842   21170520
Vlan51
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor     817737   74014504     118680    7356228
             Route cache    4976447  652147812    3848913 1261699060
                   Total    5794184  726162316    3967593 1269055288
Vlan60
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor     164013    9840780     118010   12024672
             Route cache    7066433 1427419466          0          0
                   Total    7230446 1437260246     118010   12024672
Vlan61
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor     544000   38931488      14494    6102582
             Route cache  443610386 3469406212  495437410 2400890752
                   Total  444154386 3508337700  495451904 2406993334
Vlan62
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor      15479    6166622      14265    6093782
             Route cache  300586917 2496259380  310471628 1266036642
                   Total  300602396 2502426002  310485893 1272130424
Vlan71
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor    1180407   70824420    1169613   72515510
             Route cache       1691     165718          0          0
                   Total    1182098   70990138    1169613   72515510
Vlan1000
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor         92       6744     490337   34629060
             Route cache  805909450 3667001890  757521229 2743052024
                   Total  805909542 3667008634  758011566 2777681084
Vlan1001
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor     414503  141596118      36827    3684512
             Route cache   16225116 1254014696        362      64016
                   Total   16639619 1395610814      37189    3748528

Router2#sho cef not-cef-switched
CEF Packets passed on to next switching layer
Slot  No_adj No_encap Unsupp'ted Redirect  Receive  Options   Access     Frag
RP    706334       0           0    11994   651421        0        0        0

Router2#sho ip cache flow   
IP packet size distribution (3990061 total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .000 .644 .029 .008 .035 .068 .016 .007 .011 .017 .023 .006 .005 .006 .007

    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .005 .004 .004 .073 .023 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
  0 active, 4096 inactive, 1137 added
  320921 ager polls, 0 flow alloc failures
  Active flows timeout in 30 minutes
  Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 21640 bytes
  0 active, 1024 inactive, 114 added, 114 added to flow
  0 alloc failures, 0 force free
  1 chunk, 1 chunk added
  last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet           5      0.0       151    40      0.0      54.5      15.4
UDP-NTP            126      0.0         1    76      0.0       0.0      15.2
UDP-TFTP             1      0.0         1    48      0.0       0.0      15.6
UDP-other          485      0.0       192   251      0.0     235.8      15.4
ICMP               217      0.0         4   135      0.0       1.4      15.3
IP-other           303      0.0     12854   182      0.9     561.8      15.5
Total:            1137      0.0      3509   184      0.9     250.8      15.4

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts

Everyone's tags (3)
4 REPLIES

Re: Cisco 3640 with NW-16ESW, high CPU

From your output:

Router2#sho proc cpu  sort
CPU utilization for five seconds: 86%/70%; one minute: 75%; five  minutes: 75%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min    5Min TTY Process
188     7614256    884556       8607  4.99%   0.65%  0.47%   0 SNMP ENGINE     
  16   456989136 106171824        4304  2.94%  2.94%  2.96%   0 ARP Input       
  64     4304512    1686473       2552  2.37%  0.30%  0.20%   0 IP SNMP    

I see that first 3 lines are related to SNMP processes.

Try a "show snmp" command. Do you see a lot of errors there?

According to this: http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a00800948e6.shtml

There are two probable causes for high CPU  utilization occuring in relation to Simple Network Management Protocol  (SNMP):

  1. Large route and/or ARP tables being polled by the NMS station

  2. Polling certain MIBs

How big is your ARP table?

If you are not using SNMP, try turn it off (no snmp-server), and check if there is still high utilisation on the CPU.

Let me know if this helps.

Community Member

Re: Cisco 3640 with NW-16ESW, high CPU

CPU utilization for five seconds: 86%/70%; one minute: 75%; five  minutes: 75%

Isn't the utilization of the processes only 16% ( 86% - 70% ), and the 70% is interrupts, ie. packet routing.

SNMP is begin utilizated by MRTG which created the graphs that you can see in the original post.  The utilization was high before I implemented MRTG, and I setup MRTG to try and get some good data for troubleshooting the high CPU, it is not the cause.  MRTG shows that the CPU utilization graph follows the unicast packets per second graph pretty closely.

I do have an issue with ARPs, but the ARP process is part of the 16%, which is insignificant compared to the 70% interrupts.  I will address the ARPs after I get the 70% down.

Again, the CPU Utilization graph follows the UPPS graph, so I'm inferring that it is not the ARPs nor SNMP.

Community Member

Re: Cisco 3640 with NW-16ESW, high CPU

Hi Johnny,

please attach the output of 'show process 188' and save the output of 'sh run | in snmp' ,

NOTE:

"snmp-server community public RO"

**Your configuration has the community string public which is a security loop hole please do not give out your community strings on public forums**

We can try to reset the snmp engine and check if the High CPU problem fixes or not.

before we try the same please update the 'show process' command output.

Please do try changing the public string to somethink else and if you are using a monitoring tool for this device update the monitoring tool with the changed string, and if you are not using any monitoring tool you can turn off the SNMP ENGINE by using the 'no snmp-server' command on the conf t mode.

Please let me know if this was helpful.

Regards,

podhillo

Community Member

Re: Cisco 3640 with NW-16ESW, high CPU

Please see the above message about the SNMP.

This is in an isolated lab, so I'm not concerned about the read only community strings, but thank you for observing a potental secutiry problem.

1227
Views
0
Helpful
4
Replies
CreatePlease to create content