High CPU without High Percentages in Processes

salauolayemi · ‎09-12-2008

Hi Guys, this sound a bit wierd, but I've been on a mission to discover why this happen in some Cisco routers. It could be a bug for all I can tell, but I need someone to agree with me.

From the sh proc cpu below, you can tell total 5secs CPU utilisation is 42% while the max CPU process used by any process is 1.56% actually by my VTY session. Now where is the balance in maths here?

Also, does fragmentation and/or encryption affects loss of packets on tunnel interfaces? please find show int tun below and the Total Output drops.

Thanks, please make my day!

R3#sh processes cpu sorte

CPU utilization for five seconds: 42%/38%; one minute: 41%; five minutes: 41%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

115 348 3442 101 1.96% 0.24% 0.05% 514 Virtual Exec

109 144708588 52506594 2756 1.14% 1.18% 1.55% 0 IP Input

108 8919028 190989094 46 0.16% 0.19% 0.18% 0 IP ARP Retry Age

104 8711088 190989085 45 0.16% 0.17% 0.16% 0 ACCT Periodic Pr

2 1423212 1241867 1146 0.08% 0.10% 0.08% 0 Load Meter

136 64560 6205318 10 0.08% 0.00% 0.00% 0 Socket Timers

273 1184608 7369767 160 0.08% 0.03% 0.04% 0 HSRP IPv4

179 856556 12105730 70 0.08% 0.01% 0.00% 0 IPS Timer

39 299984 3937037 76 0.08% 0.02% 0.00% 0 Net Background

171 1517744 61771366 24 0.08% 0.08% 0.08% 0 RBSCP Background

10 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

9 828 103490 8 0.00% 0.00% 0.00% 0 IPC Dynamic Cach

8 2150108 9960387 215 0.00% 0.03% 0.05% 0 OSPF-1 Hello

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

11 52008 6206061 8 0.00% 0.00% 0.00% 0 IPC Periodic Tim

12 47040 6206059 7 0.00% 0.00% 0.00% 0 IPC Deferred Por

13 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager

18 897108 791926 1132 0.00% 0.00% 0.00% 0 ARP Input

14 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressure

20 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

21 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

15 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler

R3#sh int tun1

Tunnel1 is up, line protocol is up

Hardware is Tunnel

Description: Tunnel interface for GRE

Internet address is 10.5.22.2/24

MTU 1514 bytes, BW 10000 Kbit, DLY 500000 usec,

reliability 255/255, txload 88/255, rxload 4/255

Encapsulation TUNNEL, loopback not set

Keepalive set (10 sec), retries 3

Tunnel source 10.5.23.2 (Loopback1), destination 10.5.23.1

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255

Fast tunneling enabled

Tunnel transmit bandwidth 10000 (kbps)

Tunnel receive bandwidth 10000 (kbps)

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 4w0d

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 55891

Queueing strategy: fifo (QOS pre-classification)

Output queue: 0/0 (size/max)

5 minute input rate 162000 bits/sec, 28 packets/sec

5 minute output rate 3469000 bits/sec, 730 packets/sec

46429564 packets input, 1158083652 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

856993433 packets output, 2052328059 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

salauolayemi · ‎09-12-2008

Here is the sh ver:

R3#sh ver

Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(15)

1, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 18-Jul-07 06:21 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

R3 uptime is 10 weeks, 1 day, 20 hours, 50 minutes

System returned to ROM by power-on

System restarted at 13:27:24 UTC Wed Jul 2 2008

System image file is "flash:c2800nm-advipservicesk9-mz.124-15.T1.bin"

Cisco 2811 (revision 53.51) with 249856K/12288K bytes of memory.

Processor board ID FCZ110672E3

3 FastEthernet interfaces

1 Serial(sync/async) interface

2 Virtual Private Network (VPN) Modules

DRAM configuration is 64 bits wide with parity enabled.

239K bytes of non-volatile configuration memory.

125440K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

ajay chauhan · ‎09-12-2008

Yeah because this CPU utilization is due to Interrupts which doesnt comes as a process .

For more information have a look on :

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

Hope this helps :)

Ajay

salauolayemi · ‎09-12-2008

Thanks Ajay, I will have to do more research into the interrupt driven events which can jack up CPUs on Cisco routers.

But what about the loss of outbound packets on an IPSEC/GRE tunnel interface?

Joseph W. Doherty · ‎09-12-2008

One of the most common causes of high interrupt CPU, documented in Ajay's reference, is the CPU is busy forwarding packets. On software routers, you push up the throughput, you'll see the CPU rise too. The 4% spread between total CPU and interrupt CPU, is normally, fairly good. (There's a reason why the 7200 with NSE-G2 is much "faster" than something like a 2801, the former allows much higher throughput.)

As to your question about drops on tunnel interfaces, I've seen it quite often, however I don't have an explanation for you. Haven't found one on Cisco's site, haven't openned a TAC case on it.