Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Extremely High CPU utilization 2821 Routers

I am experiencing extremely high CPU utilization on my Cisco 2821. I am using this router to handle several (120-125) GRE tunnels. I am also experiencing large amounts of input errors on both of my gigabit ethernet interfaces. All of my GRE tunnels are configured the same and I have added a sample of one of the tunnel configurations at the end of this post. I had been running over 200 tunnels on a Cisco 1760 router. another part of the router configuration that may be of some use if that I am using a route-map to route traffic from specific hosts to destinations beyond the other end of the tunnel. There is a single route-map for that but has as many entries as there are tunnels.

interface Tunnel###

ip address 10.10.10.77 255.255.255.252
no ip redirects
ip mtu 1440
ip tcp adjust-mss 1350
keepalive 20 9
tunnel source Loopback0
tunnel destination 10.1.1.240

6 REPLIES
Cisco Employee

Re: Extremely High CPU utilization 2821 Routers

Hello Aaron,

In order to have a look, would it be possible to provide us the following:

show process cpu sorted (to show the high cpu utilization)

show interfaces

Bert

New Member

Re: Extremely High CPU utilization 2821 Routers

#sh proc cpu sorted
CPU utilization for five seconds: 80%/79%; one minute: 79%; five minutes: 79%

PIDRuntime(ms)InvokeduSecs5Sec1Min5MinTTYProcess
6626946744832459893230.40%0.39%0.35%0IP Input
34269410867444703990.40%0.11%0.10%0Net Background
1291801540110300163330.24%0.02%0.00%0Per-minute Jobs
235406813041142710.08%0.02%0.00%0Load Meter
903478329573957360.08%0.04%0.02%0CEF process
9738460651962950.08%0.00%0.00%0RUDPV1 Main Proc
49486562608087410.08%0.01%0.00%0Netclock Backgro
2375320813039525770.08%0.03%0.02%0HC Counter Timer
80100.00%0.00%0.00%0IPC Zone Manager
921952651963030.00%0.00%0.00%0IPC Periodic Tim
1018856651963020.00%0.00%0.00%0IPC Deferred Por
120100.00%0.00%0.00%0IPC BackPressure
110100.00%0.00%0.00%0IPC Seat Manager
140100.00%0.00%0.00%0Crash writer
130100.00%0.00%0.00%0OIR Handler
16992523645322720.00%0.00%0.00%0ARP Input
5440851948480.00%0.00%0.00%0Pool Manager
4586137676977076140.00%0.04%0.05%0Check heaps
15479922173492200.00%0.00%0.00%0Environmental mo
200100.00%0.00%0.00%0Policy Manager
210200.00%0.00%0.00%0DDR Timers
3926632130.00%0.00%0.00%0TACACS+
71156108675100.00%0.00%0.00%0IPC Dynamic Cach
170200.00%0.00%0.00%0ATM Idle Timer
250100.00%0.00%0.00%0RO Notify Timers
260200.00%0.00%0.00%0SMART
2710952652053610.00%0.00%0.00%0GraphIt
280200.00%0.00%0.00%0Dialer event
290100.00%0.00%0.00%0SERIAL A'detect
300200.00%0.00%0.00%0XML Proxy Client
60200.00%0.00%0.00%0Timers
1903700.00%0.00%0.00%0AAA_SERVER_DEADT
180200.00%0.00%0.00%0AAA high-capacit
350200.00%0.00%0.00%0IDB Work
36602653220.00%0.00%0.00%0Logger
224220000.00%0.00%0.00%0Entity MIB API
38670606520538100.00%0.01%0.00%0Per-Second Jobs
39124443469620.00%0.00%0.00%0c2800 Periodic
400100.00%0.00%0.00%0AggMgr Process
410100.00%0.00%0.00%0dev_device_inser
420100.00%0.00%0.00%0dev_device_remov
430100.00%0.00%0.00%0sal_dpc_process
440100.00%0.00%0.00%0ARL Table Manage
112135880.00%0.00%0.00%0Chunk Manager
460200.00%0.00%0.00%0Eswilp Storm Con
470200.00%0.00%0.00%0ESWILPPM
240200.00%0.00%0.00%0Serial Backgroun
320100.00%0.00%0.00%0Inode Table Dest
500200.00%0.00%0.00%0SM Monitor
51846912170.00%0.02%0.00%322Virtual Exec
5214268651962920.00%0.00%0.00%0Ether-Switch RBC
530100.00%0.00%0.00%0IGMP Snooping Pr
540100.00%0.00%0.00%0IGMP Snooping Re
5582421734930.00%0.00%0.00%0Call Management
560200.00%0.00%0.00%0dot1x
570200.00%0.00%0.00%0DTP Protocol
5814616651963020.00%0.00%0.00%0PI MATM Aging Pr
59178465204320.00%0.00%0.00%0EtherChnl
600100.00%0.00%0.00%0L2X Data Daemon
610200.00%0.00%0.00%0AAA Dictionary R
6228018401520.00%0.00%0.00%0AAA Server
330100.00%0.00%0.00%0Critical Bkgnd
640100.00%0.00%0.00%0ACCT Periodic Pr
3738040651962650.00%0.00%0.00%0TTY Background
670100.00%0.00%0.00%0ICMP event handl
68013400.00%0.00%0.00%0TurboACL
690200.00%0.00%0.00%0TurboACL chunk
7019210827170.00%0.00%0.00%0MOP Protocols
710300.00%0.00%0.00%0PPP Hooks
730100.00%0.00%0.00%0SSS Manager
450200.00%0.00%0.00%0ESWPPM
750100.00%0.00%0.00%0SSS Feature Mana
76530282546971020.00%0.01%0.00%0SSS Feature Time
770100.00%0.00%0.00%0VPDN call manage
780100.00%0.00%0.00%0L2X Socket proce
790100.00%0.00%0.00%0L2X SSS manager
800200.00%0.00%0.00%0L2TP mgmt daemon
810100.00%0.00%0.00%0X.25 Encaps Mana
82424881727932450.00%0.00%0.00%0IP Background
8326856679283950.00%0.00%0.00%0IP RIB Update
840200.00%0.00%0.00%0PPP IP Route
850200.00%0.00%0.00%0PPP IPCP
8611312647374410.00%0.00%0.00%0Socket Timers
87121223811500.00%0.00%0.00%0TCP Timer
88106489211920.00%0.00%0.00%0TCP Protocols
890100.00%0.00%0.00%0COPS
480200.00%0.00%0.00%0Eswilp Storm Con
910200.00%0.00%0.00%0L2MM
920100.00%0.00%0.00%0MRD
930100.00%0.00%0.00%0IGMPSN
940200.00%0.00%0.00%0SNMP Timers
950200.00%0.00%0.00%0SCTP Main Proces
960100.00%0.00%0.00%0IUA Main Process
6350833771500.00%0.00%0.00%0AAA ACCT Proc
980100.00%0.00%0.00%0bsm_timers
9913516651963120.00%0.00%0.00%0bsm_xmt_proc
1010200.00%0.00%0.00%0Dialer Forwarder
102131961086751210.00%0.00%0.00%0IP Cache Ager
10315970010867714690.00%0.00%0.00%0Adj Manager
104642173820.00%0.00%0.00%0HTTP CORE
1050100.00%0.00%0.00%0IP Traceroute
1060100.00%0.00%0.00%0RARP Input
1070100.00%0.00%0.00%0PAD InCall
1080200.00%0.00%0.00%0X.25 Background
1090200.00%0.00%0.00%0PPP Bind
1100200.00%0.00%0.00%0PPP SSS
11168426082420.00%0.00%0.00%0CRM_CALL_UPDATE_
1120200.00%0.00%0.00%0ENABLE AAA
1130100.00%0.00%0.00%0EM Background Pr
1140100.00%0.00%0.00%0Key chain liveke
1150500.00%0.00%0.00%0LINE AAA
1160200.00%0.00%0.00%0LOCAL AAA
11719532247967870.00%0.01%0.00%0TPLUS
1180100.00%0.00%0.00%0PM Callback
1203969284260.00%0.00%0.00%0AAA SEND STOP EV
121182065204320.00%0.00%0.00%0RMON Recycle Pro
1220200.00%0.00%0.00%0RMON Deferred Se
1230100.00%0.00%0.00%0Syslog Traps
1248240000.00%0.00%0.00%0VLAN Manager
12523613831700.00%0.00%0.00%0Syslog
1260100.00%0.00%0.00%0VPDN Scal
12717216891010.00%0.00%0.00%0Net Input
1281967492130413715080.00%0.06%0.07%0Compute load avg
74392886939140.00%0.00%0.00%0SSS Test Client
13014116226112620.00%0.00%0.00%0CEF Scanner
1310100.00%0.00%0.00%0tHUB
1320200.00%0.00%0.00%0tENM
1338423369646111870.00%0.02%0.00%0HSRP (Standby)
13444648651961160.00%0.00%0.00%0Track
1361269283288513850.00%0.00%0.00%0IP SNMP
1371248361634657630.00%0.00%0.00%0PDU DISPATCHER
138130253616347579670.00%0.00%0.00%0SNMP ENGINE
1390100.00%0.00%0.00%0SNMP ConfCopyPro
14067213275060.00%0.00%0.00%0SNMP Traps
14248823260200.00%0.00%0.00%0IP-EIGRP Router
14363415647886271320.00%0.01%0.00%0IP-EIGRP: PDM
14464024115260.00%0.00%0.00%0IP-EIGRP Router
14557906044605901290.00%0.00%0.00%0IP-EIGRP: PDM
1465037165850050860.00%0.00%0.00%0IP-EIGRP: HELLO
1474935125725897860.00%0.00%0.00%0IP-EIGRP: HELLO

#sh interface gi0/0
GigabitEthernet0/0 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 001b.d439.65b0 (bia 001b.d439.65b0)
  Description: Control DMZ (to Firewall)
  Internet address is 10.10.30.2/28
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 2/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 12:40:48
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 11416000 bits/sec, 9247 packets/sec
  5 minute output rate 5008000 bits/sec, 10118 packets/sec
     198456731 packets input, 1474299571 bytes, 0 no buffer
     Received 261932 broadcasts, 0 runts, 0 giants, 0 throttles
     1514 input errors, 0 CRC, 0 frame, 0 overrun, 1514 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     221189679 packets output, 1584642221 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 2786 pause output
     0 output buffer failures, 0 output buffers swapped out

#sh interface gi0/1
GigabitEthernet0/1 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 001b.d439.65b1 (bia 001b.d439.65b1)
  Description: Core DMZ (to transport routers)
  Internet address is 10.10.20.2/26
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 2/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 12:40:50
  Input queue: 0/75/2/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 9163000 bits/sec, 12797 packets/sec
  5 minute output rate 15089000 bits/sec, 11908 packets/sec
     283011297 packets input, 1756970027 bytes, 0 no buffer
     Received 109970 broadcasts, 0 runts, 0 giants, 2 throttles
     279476 input errors, 0 CRC, 0 frame, 0 overrun, 279476 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     259671581 packets output, 381573842 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 280171 pause output
     0 output buffer failures, 0 output buffers swapped out

Cisco Employee

Re: Extremely High CPU utilization 2821 Routers

Thanks for the output Aaron.

As we can see from the show process cpu:

CPU utilization for five seconds: 80%/79% => 79% of the CPU utilization occurs under interrupt.

A Cisco 2821 is using interrupts mostly to perform CEF packet forwarding (which is the optimal forwarding method on the 2821).  Besides some exceptions, this basically means that you are running close to the platform limitations here.

The ignored counter for the interfaces indicates the amount of received packets ignored by the                     interface because the interface hardware ran low on internal buffers.  Basically, this occurs due to short bursts of traffic and again are an indication that at some points, the devices limits are being reached.

It may be possible to reduce this ( flow-control / QOS ), however, this will depend on how large the bursts are.

In summary, I believe you are reaching the platform limitations of the 2821 and you should consider either offloading traffic to a different device or upgrading to a more powerfull router.

HTH,

Bert

New Member

Re: Extremely High CPU utilization 2821 Routers

I actually have a 2851 setup as my standby router but I don't think I would gain anything by using it over the 2821. I did find that some of my traffic is passing through the router twice and one a lot of the trips it is going in and out the same interface. I am actually changing this tonight to hopefully drop the traffic load enough to handle the bursts and drop the CPU.

New Member

Re: Extremely High CPU utilization 2821 Routers

Aaron,

Cisco 2821 is hitting the limits of the platform. You need to upgrade to a bigger device. Make sure you are not running IP ACCOUNTING.

HTH,

Elyinn.-

Cisco Employee

Re: Extremely High CPU utilization 2821 Routers

Hello Aaron,

In regards to changing the 2821 to the standby 2851, this may make a small difference.

Some basic performance figures of the devices can be found here:

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

Off course, try first to offload traffic which shouldn't hit the device (eg. redirected traffic).

HTH,

Bert

725
Views
5
Helpful
6
Replies