Re: 871 high CPU utilization

catalin.stan · ‎08-29-2013

Hi,

I have a 871 and when I'm downloading a bigger file (over 100 MB) or uploading via FTP the CPU goes to 99%.

I have tested this with basic configuration (only nat and dhcp service configured). I have install the latest IOS version that is supported on the router (c870-advipservicesk9-mz.124-24.T8.bin) but I still have the same problem.

Can someone please let me know if this is a normal behavior for 871. Will a flash/RAM upgrade will help?

Thank you

Seb Rupik · ‎08-29-2013

Hi Catalin,

During one of these high CPU states, what is the output of:

show proc cpu sorted

show proc cpu sorted 1min

cheers,

Seb.

catalin.stan · ‎08-29-2013

Hi Seb,

Thank you for your reply.

Here is the output:

Router#sh proc cpu sorted

CPU utilization for five seconds: 89%/87%; one minute: 71%; five minutes: 25%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

49 48046 377383 127 0.63% 0.87% 0.35% 0 COLLECT STAT COU

120 1144 2680526 0 0.07% 0.03% 0.01% 0 HQF Shaper Backg

77 18078 55311 326 0.07% 0.11% 0.10% 0 IP Input

121 464 751995 0 0.07% 0.01% 0.00% 0 RBSCP Background

85 7452 38960 191 0.07% 0.01% 0.00% 0 Spanning Tree

111 1272 116689 10 0.07% 0.00% 0.00% 0 CEF: IPv4 proces

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

5 4 5 800 0.00% 0.00% 0.00% 0 Pool Manager

7 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

8 13229 41788 316 0.00% 0.04% 0.06% 0 ARP Input

11 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

9 76 78717 0 0.00% 0.00% 0.00% 0 ARP Background

2 1420 15058 94 0.00% 0.00% 0.00% 0 Load Meter

10 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

4 748895 42062 17804 0.00% 0.79% 0.94% 0 Check heaps

16 16 70 228 0.00% 0.00% 0.00% 0 EEM ED Syslog

17 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

18 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers

19 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa

12 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

3 4457 412 10817 0.00% 0.03% 0.02% 0 Exec

22 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

23 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

24 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client

13 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

14 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers

27 0 1 0 0.00% 0.00% 0.00% 0 IDB Work

15 0 2 0 0.00% 0.00% 0.00% 0 Entity MIB API

29 32 75243 0 0.00% 0.00% 0.00% 0 TTY Background

20 0 2 0 0.00% 0.00% 0.00% 0 SMART

31 0 6 0 0.00% 0.00% 0.00% 0 IF-MGR control p

32 0 20 0 0.00% 0.00% 0.00% 0 IF-MGR event pro

33 0 7 0 0.00% 0.00% 0.00% 0 Collection proce

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

34 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

35 0 2 0 0.00% 0.00% 0.00% 0 AggMgr Process

36 0 1 0 0.00% 0.00% 0.00% 0 Token Daemon

37 0 1 0 0.00% 0.00% 0.00% 0 Transport Port A

38 28 22583 1 0.00% 0.00% 0.00% 0 HC Counter Timer

21 200 75280 2 0.00% 0.00% 0.00% 0 GraphIt

40 224 501887 0 0.00% 0.00% 0.00% 0 WLAN LED Timers

41 0 2 0 0.00% 0.00% 0.00% 0 AUX

25 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

43 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con

44 20 11 1818 0.00% 0.00% 0.00% 0 USB Startup

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

45 5249 65464 80 0.00% 0.00% 0.00% 0 Net Input

46 84 15063 5 0.00% 0.00% 0.00% 0 Compute load avg

47 20197 1282 15754 0.00% 0.02% 0.00% 0 Per-minute Jobs

48 12 6 2000 0.00% 0.00% 0.00% 0 Switch Link Moni

26 504 51912 9 0.00% 0.00% 0.00% 0 Net Background

50 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Pr

28 0 32 0 0.00% 0.00% 0.00% 0 Logger

52 16 75319 0 0.00% 0.00% 0.00% 0 Crypto Device Up

53 0 1 0 0.00% 0.00% 0.00% 0 Multi-ISA Event

54 0 1 0 0.00% 0.00% 0.00% 0 Multi-ISA Cleanu

30 396 75329 5 0.00% 0.00% 0.00% 0 Per-Second Jobs

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

56 0 2 0 0.00% 0.00% 0.00% 0 SEC BATCH

57 28 75247 0 0.00% 0.00% 0.00% 0 PI MATM Aging Pr

58 4 2 2000 0.00% 0.00% 0.00% 0 DTP Protocol

59 4 6 666 0.00% 0.00% 0.00% 0 Dot1x Mgr Proces

39 0 2 0 0.00% 0.00% 0.00% 0 LED Timers

61 0 1 0 0.00% 0.00% 0.00% 0 EAP Framework

62 116 75283 1 0.00% 0.00% 0.00% 0 linktest

63 0 2 0 0.00% 0.00% 0.00% 0 Dot11 Mgmt & Ass

64 0 2 0 0.00% 0.00% 0.00% 0 Dot11 aaa proces

65 0 1256 0 0.00% 0.00% 0.00% 0 pmkid

66 0 2 0 0.00% 0.00% 0.00% 0 Dot11 auth Dot1x

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

67 0 1 0 0.00% 0.00% 0.00% 0 Dot11 Mac Auth

68 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

69 0 2 0 0.00% 0.00% 0.00% 0 AAA Server

70 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc

71 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr

72 108 8798 12 0.00% 0.00% 0.00% 0 CDP Protocol

73 0 2 0 0.00% 0.00% 0.00% 0 CEF switching ba

74 0 5 0 0.00% 0.00% 0.00% 0 ADJ resolve proc

75 12 7 1714 0.00% 0.00% 0.00% 0 IP ARP Adjacency

76 4 1 4000 0.00% 0.00% 0.00% 0 IP ARP Retry Age

42 72 6 12000 0.00% 0.00% 0.00% 0 ESWPPM

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

78 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl

79 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Echo event

80 0 3 0 0.00% 0.00% 0.00% 0 PPP Hooks

82 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager

83 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana

84 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Time

51 204 1891 107 0.00% 0.00% 0.00% 0 IGMP Snooping Re

86 0 24 0 0.00% 0.00% 0.00% 0 SSM connection m

87 0 1 0 0.00% 0.00% 0.00% 0 AC Switch

88 0 252 0 0.00% 0.00% 0.00% 0 Authentication P

89 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

90 0 2 0 0.00% 0.00% 0.00% 0 EAPoUDP Process

91 0 2 0 0.00% 0.00% 0.00% 0 IP Host Track Pr

92 152 1446 105 0.00% 0.00% 0.00% 0 IP Background

93 32 1064 30 0.00% 0.00% 0.00% 0 IP RIB Update

94 0 1 0 0.00% 0.00% 0.00% 0 L2X Data Daemon

95 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route

96 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP

97 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

98 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

99 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

100 56 74 756 0.00% 0.00% 0.00% 0 L2MM

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

101 0 1 0 0.00% 0.00% 0.00% 0 MRD

102 484 1952 247 0.00% 0.00% 0.00% 0 IGMPSN

55 296 26 11384 0.00% 0.00% 0.00% 0 crypto engine pr

104 0 1 0 0.00% 0.00% 0.00% 0 TCP Protocols

105 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

106 0 252 0 0.00% 0.00% 0.00% 0 HTTP CORE

107 4 1511 2 0.00% 0.00% 0.00% 0 CEF background p

108 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers

109 0 1 0 0.00% 0.00% 0.00% 0 COPS

110 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder

1 12 33 363 0.00% 0.00% 0.00% 0 Chunk Manager

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

112 0 4 0 0.00% 0.00% 0.00% 0 ADJ background

113 0 3 0 0.00% 0.00% 0.00% 0 Flow Exporter Ti

114 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute

115 8 91 87 0.00% 0.00% 0.00% 0 RARP Input

116 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Inspect Tim

117 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind

118 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS

119 0 1 0 0.00% 0.00% 0.00% 0 MQC Flow Event B

60 0 1 0 0.00% 0.00% 0.00% 0 MAB Framework

103 0 1 0 0.00% 0.00% 0.00% 0 TCP Timer

122 4 7 571 0.00% 0.00% 0.00% 0 SCTP Main Proces

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

123 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage

124 0 1 0 0.00% 0.00% 0.00% 0 IPS Process

125 0 2 0 0.00% 0.00% 0.00% 0 IPS Auto Update

126 0 2 0 0.00% 0.00% 0.00% 0 SDEE Management

127 196 146914 1 0.00% 0.00% 0.00% 0 Inspect process

128 0 628 0 0.00% 0.00% 0.00% 0 DHCPD Timer

129 4 6019 0 0.00% 0.00% 0.00% 0 FW DP Inspect pr

130 0 1 0 0.00% 0.00% 0.00% 0 tHUB

131 80 146914 0 0.00% 0.00% 0.00% 0 CCE DP URLF cach

132 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc

133 0 1 0 0.00% 0.00% 0.00% 0 Select Timers

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

134 32 2 16000 0.00% 0.00% 0.00% 0 HTTP Process

135 0 2 0 0.00% 0.00% 0.00% 0 CIFS API Process

136 4 2 2000 0.00% 0.00% 0.00% 0 CIFS Proxy Proce

137 4 1 4000 0.00% 0.00% 0.00% 0 Crypto HW Proc

139 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve

140 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA

141 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr

142 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke

143 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

144 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA

145 0 2 0 0.00% 0.00% 0.00% 0 TPLUS

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

146 0 1 0 0.00% 0.00% 0.00% 0 FW_TEST_TRP

147 0 1 0 0.00% 0.00% 0.00% 0 EPM MAIN PROCESS

148 0 3 0 0.00% 0.00% 0.00% 0 Crypto WUI

149 0 2 0 0.00% 0.00% 0.00% 0 Crypto Support

150 0 1 0 0.00% 0.00% 0.00% 0 IPSECv6 PS Proc

151 0 4 0 0.00% 0.00% 0.00% 0 Crypto CA

152 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL

153 0 1 0 0.00% 0.00% 0.00% 0 Key Proc

154 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc

155 36149 2592 13946 0.00% 0.00% 0.00% 0 crypto sw pk pro

156 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

157 0 3 0 0.00% 0.00% 0.00% 0 Crypto IKE Dispa

158 4 3 1333 0.00% 0.00% 0.00% 0 Crypto IKMP

159 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key

160 148 3767 39 0.00% 0.00% 0.00% 0 IPSEC key engine

161 0 1 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce

162 4 6 666 0.00% 0.00% 0.00% 0 Crypto ACL

163 0 1 0 0.00% 0.00% 0.00% 0 Crypto PAS Proc

164 0 1 0 0.00% 0.00% 0.00% 0 GDOI GM Process

165 0 1 0 0.00% 0.00% 0.00% 0 UNICAST REKEY

166 0 1 0 0.00% 0.00% 0.00% 0 UNICAST REKEY AC

167 20 34977 0 0.00% 0.00% 0.00% 0 PM Callback

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

168 0 2 0 0.00% 0.00% 0.00% 0 Control-plane ho

169 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV

170 4 2 2000 0.00% 0.00% 0.00% 0 EEM ED Resource

171 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Routing

172 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Track

173 0 7528 0 0.00% 0.00% 0.00% 0 RMON Recycle Pro

174 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se

175 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

176 8 2353 3 0.00% 0.00% 0.00% 0 Crypto cTCP proc

177 28 2 14000 0.00% 0.00% 0.00% 0 VLAN Manager

178 0 1255 0 0.00% 0.00% 0.00% 0 DHCPD Database

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

179 4 52 76 0.00% 0.00% 0.00% 0 EEM Server

180 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc

181 0 3 0 0.00% 0.00% 0.00% 0 EEM ED CLI

182 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Counter

183 0 3 0 0.00% 0.00% 0.00% 0 EM ED GOLD

184 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Interface

185 4 3 1333 0.00% 0.00% 0.00% 0 EEM ED IOSWD

186 4 3 1333 0.00% 0.00% 0.00% 0 EEM ED Ipsla

187 0 3 0 0.00% 0.00% 0.00% 0 EEM ED None

188 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Nf

189 0 3 0 0.00% 0.00% 0.00% 0 EEM ED OIR

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

190 0 1 0 0.00% 0.00% 0.00% 0 VPDN Test

191 0 3 0 0.00% 0.00% 0.00% 0 EEM ED SNMP

192 0 2 0 0.00% 0.00% 0.00% 0 EEM ED SNMP Noti

193 0 1886 0 0.00% 0.00% 0.00% 0 EEM ED Timer

194 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Test

195 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Config

196 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Env

197 0 18 0 0.00% 0.00% 0.00% 0 Syslog

198 0 3 0 0.00% 0.00% 0.00% 0 EEM ED RPC

201 8 43 186 0.00% 0.00% 0.00% 0 SSH Event handle

202 168 1039 161 0.00% 0.00% 0.00% 0 DHCPD Receive

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

203 260 75287 3 0.00% 0.00% 0.00% 0 DHCP Client

205 445 146917 3 0.00% 0.00% 0.00% 0 IP NAT Ager

206 0 1 0 0.00% 0.00% 0.00% 0 IP NAT WLAN

207 0 1 0 0.00% 0.00% 0.00% 0 IP VFR proc

Router#sh proc cpu sorted 1min

CPU utilization for five seconds: 18%/17%; one minute: 17%; five minutes: 19%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

4 750096 42128 17805 0.00% 1.30% 1.01% 0 Check heaps

49 48986 377900 129 0.55% 0.29% 0.29% 0 COLLECT STAT COU

77 18210 55706 326 0.07% 0.07% 0.09% 0 IP Input

8 13321 42023 316 0.07% 0.05% 0.05% 0 ARP Input

3 4541 670 6777 0.00% 0.05% 0.03% 0 Exec

47 20229 1284 15754 0.31% 0.03% 0.00% 0 Per-minute Jobs

45 5281 65640 80 0.00% 0.00% 0.00% 0 Net Input

26 512 51963 9 0.07% 0.00% 0.00% 0 Net Background

120 1168 2684138 0 0.00% 0.00% 0.00% 0 HQF Shaper Backg

9 80 78824 1 0.00% 0.00% 0.00% 0 ARP Background

10 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

Router#sh proc cpu sorted 1min

CPU utilization for five seconds: 91%/89%; one minute: 22%; five minutes: 20%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

4 750669 42135 17815 0.55% 1.24% 1.01% 0 Check heaps

49 51082 377908 135 1.18% 0.37% 0.30% 0 COLLECT STAT COU

77 18222 55709 327 0.00% 0.06% 0.08% 0 IP Input

8 13353 42027 317 0.07% 0.05% 0.06% 0 ARP Input

3 4793 677 7079 0.07% 0.05% 0.03% 0 Exec

47 20229 1284 15754 0.00% 0.03% 0.00% 0 Per-minute Jobs

45 5281 65642 80 0.00% 0.00% 0.00% 0 Net Input

2 1480 15079 98 0.07% 0.00% 0.00% 0 Load Meter

40 228 502581 0 0.07% 0.00% 0.00% 0 WLAN LED Timers

30 504 75436 6 0.07% 0.00% 0.00% 0 Per-Second Jobs

26 512 51963 9 0.00% 0.00% 0.00% 0 Net Background

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

120 1172 2684159 0 0.00% 0.00% 0.00% 0 HQF Shaper Backg

12 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

13 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

14 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers

15 0 2 0 0.00% 0.00% 0.00% 0 Entity MIB API

17 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

16 16 70 228 0.00% 0.00% 0.00% 0 EEM ED Syslog

19 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify Wa

20 0 2 0 0.00% 0.00% 0.00% 0 SMART

18 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers

22 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

23 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

24 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client

21 200 75387 2 0.00% 0.00% 0.00% 0 GraphIt

11 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

27 0 1 0 0.00% 0.00% 0.00% 0 IDB Work

28 0 32 0 0.00% 0.00% 0.00% 0 Logger

29 32 75349 0 0.00% 0.00% 0.00% 0 TTY Background

10 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

25 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

32 0 20 0 0.00% 0.00% 0.00% 0 IF-MGR event pro

33 0 7 0 0.00% 0.00% 0.00% 0 Collection proce

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

34 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

35 0 2 0 0.00% 0.00% 0.00% 0 AggMgr Process

36 0 1 0 0.00% 0.00% 0.00% 0 Token Daemon

37 0 1 0 0.00% 0.00% 0.00% 0 Transport Port A

38 28 22615 1 0.00% 0.00% 0.00% 0 HC Counter Timer

31 0 6 0 0.00% 0.00% 0.00% 0 IF-MGR control p

9 80 78828 1 0.00% 0.00% 0.00% 0 ARP Background

41 0 2 0 0.00% 0.00% 0.00% 0 AUX

42 72 6 12000 0.00% 0.00% 0.00% 0 ESWPPM

43 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con

44 20 11 1818 0.00% 0.00% 0.00% 0 USB Startup

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

7 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

46 84 15084 5 0.00% 0.00% 0.00% 0 Compute load avg

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

48 12 6 2000 0.00% 0.00% 0.00% 0 Switch Link Moni

5 4 5 800 0.00% 0.00% 0.00% 0 Pool Manager

50 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Pr

1 12 33 363 0.00% 0.00% 0.00% 0 Chunk Manager

52 16 75426 0 0.00% 0.00% 0.00% 0 Crypto Device Up

53 0 1 0 0.00% 0.00% 0.00% 0 Multi-ISA Event

54 0 1 0 0.00% 0.00% 0.00% 0 Multi-ISA Cleanu

55 296 26 11384 0.00% 0.00% 0.00% 0 crypto engine pr

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

56 0 2 0 0.00% 0.00% 0.00% 0 SEC BATCH

57 28 75353 0 0.00% 0.00% 0.00% 0 PI MATM Aging Pr

58 4 2 2000 0.00% 0.00% 0.00% 0 DTP Protocol

59 4 6 666 0.00% 0.00% 0.00% 0 Dot1x Mgr Proces

39 0 2 0 0.00% 0.00% 0.00% 0 LED Timers

61 0 1 0 0.00% 0.00% 0.00% 0 EAP Framework

62 116 75389 1 0.00% 0.00% 0.00% 0 linktest

63 0 2 0 0.00% 0.00% 0.00% 0 Dot11 Mgmt & Ass

64 0 2 0 0.00% 0.00% 0.00% 0 Dot11 aaa proces

65 0 1258 0 0.00% 0.00% 0.00% 0 pmkid

66 0 2 0 0.00% 0.00% 0.00% 0 Dot11 auth Dot1x

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

67 0 1 0 0.00% 0.00% 0.00% 0 Dot11 Mac Auth

68 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R

69 0 2 0 0.00% 0.00% 0.00% 0 AAA Server

70 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc

71 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr

72 108 8813 12 0.00% 0.00% 0.00% 0 CDP Protocol

73 0 2 0 0.00% 0.00% 0.00% 0 CEF switching ba

74 0 5 0 0.00% 0.00% 0.00% 0 ADJ resolve proc

75 12 7 1714 0.00% 0.00% 0.00% 0 IP ARP Adjacency

76 4 1 4000 0.00% 0.00% 0.00% 0 IP ARP Retry Age

51 204 1895 107 0.00% 0.00% 0.00% 0 IGMP Snooping Re

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

78 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl

79 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Echo event

80 0 3 0 0.00% 0.00% 0.00% 0 PPP Hooks

82 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager

83 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana

84 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Time

85 7460 39118 190 0.00% 0.00% 0.00% 0 Spanning Tree

86 0 24 0 0.00% 0.00% 0.00% 0 SSM connection m

87 0 1 0 0.00% 0.00% 0.00% 0 AC Switch

88 0 252 0 0.00% 0.00% 0.00% 0 Authentication P

89 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

90 0 2 0 0.00% 0.00% 0.00% 0 EAPoUDP Process

91 0 2 0 0.00% 0.00% 0.00% 0 IP Host Track Pr

92 252 1448 174 0.00% 0.00% 0.00% 0 IP Background

93 32 1066 30 0.00% 0.00% 0.00% 0 IP RIB Update

94 0 1 0 0.00% 0.00% 0.00% 0 L2X Data Daemon

95 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route

96 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP

97 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

98 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

99 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant

100 56 74 756 0.00% 0.00% 0.00% 0 L2MM

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

101 0 1 0 0.00% 0.00% 0.00% 0 MRD

102 484 1956 247 0.00% 0.00% 0.00% 0 IGMPSN

60 0 1 0 0.00% 0.00% 0.00% 0 MAB Framework

104 0 1 0 0.00% 0.00% 0.00% 0 TCP Protocols

105 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

106 0 252 0 0.00% 0.00% 0.00% 0 HTTP CORE

107 4 1512 2 0.00% 0.00% 0.00% 0 CEF background p

108 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers

109 0 1 0 0.00% 0.00% 0.00% 0 COPS

110 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder

111 1500 116852 12 0.00% 0.00% 0.00% 0 CEF: IPv4 proces

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

112 0 4 0 0.00% 0.00% 0.00% 0 ADJ background

113 0 3 0 0.00% 0.00% 0.00% 0 Flow Exporter Ti

114 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute

115 8 91 87 0.00% 0.00% 0.00% 0 RARP Input

116 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Inspect Tim

117 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind

118 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS

119 0 1 0 0.00% 0.00% 0.00% 0 MQC Flow Event B

103 0 1 0 0.00% 0.00% 0.00% 0 TCP Timer

121 476 753027 0 0.00% 0.00% 0.00% 0 RBSCP Background

122 4 7 571 0.00% 0.00% 0.00% 0 SCTP Main Proces

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

123 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage

124 0 1 0 0.00% 0.00% 0.00% 0 IPS Process

125 0 2 0 0.00% 0.00% 0.00% 0 IPS Auto Update

126 0 2 0 0.00% 0.00% 0.00% 0 SDEE Management

127 296 147119 2 0.00% 0.00% 0.00% 0 Inspect process

128 0 629 0 0.00% 0.00% 0.00% 0 DHCPD Timer

129 4 6027 0 0.00% 0.00% 0.00% 0 FW DP Inspect pr

130 0 1 0 0.00% 0.00% 0.00% 0 tHUB

131 84 147119 0 0.00% 0.00% 0.00% 0 CCE DP URLF cach

132 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc

133 0 1 0 0.00% 0.00% 0.00% 0 Select Timers

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

134 32 2 16000 0.00% 0.00% 0.00% 0 HTTP Process

135 0 2 0 0.00% 0.00% 0.00% 0 CIFS API Process

136 4 2 2000 0.00% 0.00% 0.00% 0 CIFS Proxy Proce

137 4 1 4000 0.00% 0.00% 0.00% 0 Crypto HW Proc

139 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve

140 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA

141 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr

142 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke

143 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA

144 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA

145 0 2 0 0.00% 0.00% 0.00% 0 TPLUS

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

146 0 1 0 0.00% 0.00% 0.00% 0 FW_TEST_TRP

147 0 1 0 0.00% 0.00% 0.00% 0 EPM MAIN PROCESS

148 0 3 0 0.00% 0.00% 0.00% 0 Crypto WUI

149 0 2 0 0.00% 0.00% 0.00% 0 Crypto Support

150 0 1 0 0.00% 0.00% 0.00% 0 IPSECv6 PS Proc

151 0 4 0 0.00% 0.00% 0.00% 0 Crypto CA

152 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL

153 0 1 0 0.00% 0.00% 0.00% 0 Key Proc

154 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc

155 36149 2592 13946 0.00% 0.00% 0.00% 0 crypto sw pk pro

156 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

157 0 3 0 0.00% 0.00% 0.00% 0 Crypto IKE Dispa

158 4 3 1333 0.00% 0.00% 0.00% 0 Crypto IKMP

159 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key

160 148 3772 39 0.00% 0.00% 0.00% 0 IPSEC key engine

161 0 1 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce

162 4 6 666 0.00% 0.00% 0.00% 0 Crypto ACL

163 0 1 0 0.00% 0.00% 0.00% 0 Crypto PAS Proc

164 0 1 0 0.00% 0.00% 0.00% 0 GDOI GM Process

165 0 1 0 0.00% 0.00% 0.00% 0 UNICAST REKEY

166 0 1 0 0.00% 0.00% 0.00% 0 UNICAST REKEY AC

167 32 35030 0 0.00% 0.00% 0.00% 0 PM Callback

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

168 0 2 0 0.00% 0.00% 0.00% 0 Control-plane ho

169 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV

170 4 2 2000 0.00% 0.00% 0.00% 0 EEM ED Resource

171 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Routing

172 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Track

173 0 7538 0 0.00% 0.00% 0.00% 0 RMON Recycle Pro

174 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se

175 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

176 8 2356 3 0.00% 0.00% 0.00% 0 Crypto cTCP proc

177 28 2 14000 0.00% 0.00% 0.00% 0 VLAN Manager

178 0 1257 0 0.00% 0.00% 0.00% 0 DHCPD Database

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

179 4 52 76 0.00% 0.00% 0.00% 0 EEM Server

180 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc

181 0 3 0 0.00% 0.00% 0.00% 0 EEM ED CLI

182 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Counter

183 0 3 0 0.00% 0.00% 0.00% 0 EM ED GOLD

184 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Interface

185 4 3 1333 0.00% 0.00% 0.00% 0 EEM ED IOSWD

186 4 3 1333 0.00% 0.00% 0.00% 0 EEM ED Ipsla

187 0 3 0 0.00% 0.00% 0.00% 0 EEM ED None

188 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Nf

189 0 3 0 0.00% 0.00% 0.00% 0 EEM ED OIR

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

190 0 1 0 0.00% 0.00% 0.00% 0 VPDN Test

191 0 3 0 0.00% 0.00% 0.00% 0 EEM ED SNMP

192 0 2 0 0.00% 0.00% 0.00% 0 EEM ED SNMP Noti

193 0 1889 0 0.00% 0.00% 0.00% 0 EEM ED Timer

194 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Test

195 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Config

196 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Env

197 0 18 0 0.00% 0.00% 0.00% 0 Syslog

198 0 3 0 0.00% 0.00% 0.00% 0 EEM ED RPC

201 8 43 186 0.00% 0.00% 0.00% 0 SSH Event handle

202 168 1039 161 0.00% 0.00% 0.00% 0 DHCPD Receive

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

203 260 75393 3 0.00% 0.00% 0.00% 0 DHCP Client

205 449 147122 3 0.00% 0.00% 0.00% 0 IP NAT Ager

206 0 1 0 0.00% 0.00% 0.00% 0 IP NAT WLAN

207 0 1 0 0.00% 0.00% 0.00% 0 IP VFR proc

Seb Rupik · ‎08-29-2013

Hi Catalin,

The line:

CPU utilization for five seconds: 91%/89%; one minute: 22%; five minutes: 20%

...indicates 89% of CPU usage is down to intterupts where the packets are being process switched as opposed to processing by CEF. The other 2% being made of running processes.

What is the output from:

show ip cef switching statistics

Also take a look at this document which will run you through some tests you can carry out:

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

cheers,

Seb.

catalin.stan · ‎08-29-2013

Hi Seb,

here is the output:

Router#sh ip cef switching statistics

Reason Drop Punt Punt2Host

RP LES No route 2226 0 1

RP LES Packet destined for us 0 38152 0

RP LES TTL expired 0 0 26128

RP LES Features 0 0 924

RP LES Unclassified reason 22334 0 0

RP LES Total 24560 38152 27053

All Total 24560 38152 27053

Joseph W. Doherty · ‎08-29-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Your CPU stats indicate most of your CPU is "interrupt" CPU, i.e. normal packet forwarding.

An 871 had only a 25 Kpps rate "engine". Cisco recommends the 25 Kbps 860 series for only 4 Mbps of WAN bandwidth.

What's your aggregate bandwidth utilization when CPU tops out?

catalin.stan · ‎08-29-2013

Hi Joseph,

Can you please tell me how do I check that ?

Seb Rupik · ‎08-29-2013

Try this command to show you the per interface pps on the router:

sh int summary

catalin.stan · ‎08-29-2013

Router#sh inter summary

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

------------------------------------------------------------------------

* FastEthernet0 0 0 0 0 2582000 274 60000 107 0

* FastEthernet1 0 0 0 0 61000 107 2594000 270 0

FastEthernet2 0 0 0 0 0 0 0 0 0

FastEthernet3 0 0 0 0 0 0 0 0 0

FastEthernet4 0 0 0 0 0 0 0 0 0

NVI0 0 0 0 0 0 0 0 0 0

* Vlan1 0 0 0 0 2873000 306 56000 118 0

* Vlan192 0 0 0 0 61000 118 2862000 299 0

Seb Rupik · ‎08-29-2013

Was that during a download and high CPU usage?

catalin.stan · ‎08-29-2013

Yes

Seb Rupik · ‎08-29-2013

OK, what's the output from: sh int stats

...compare the ratio between 'Processor' and 'Total', to see what percentage of packets processed by the CPU.

Also the output from:

sh int fa0

sh int fa1

...look at the 'overrun' counter. A high number here would indicate an inerface that is stuggling to process the number of packets being processed.

If any of the above numbers are high, then this is highlighting a hardware limitation. So long as there aren't many packet drops, then this limitation will only manifest itself with increased latency.

catalin.stan · ‎08-29-2013

Router#sh interfaces stats
FastEthernet0
          Switching path    Pkts In   Chars In   Pkts Out Chars Out
               Processor      57707   69990129      21779    1412159
             Route cache          0          0          0          0
                   Total      57707   69990129      21779    1412159
FastEthernet1
          Switching path    Pkts In   Chars In   Pkts Out Chars Out
               Processor      20422    1341906      53711   65306522
             Route cache          0          0          0          0
                   Total      20422    1341906      53711   65306522
FastEthernet2
          Switching path    Pkts In   Chars In   Pkts Out Chars Out
               Processor          0          0          0          0
             Route cache          0          0          0          0
                   Total          0          0          0          0
FastEthernet3
          Switching path    Pkts In   Chars In   Pkts Out Chars Out
               Processor          0          0          0          0
             Route cache          0          0          0          0
                   Total          0          0          0          0
Interface FastEthernet4 is disabled

Interface NVI0 is disabled

Vlan1
          Switching path    Pkts In   Chars In   Pkts Out Chars Out
               Processor        222      63489         18       1629
             Route cache      73197   88759272      27702    1519940
                   Total      73419   88822761      27720    1521569
Vlan192
          Switching path    Pkts In   Chars In   Pkts Out Chars Out
               Processor         94      17399         19       2646
             Route cache      27888    1692644      73693   89384507
                   Total      27982    1710043      73712   89387153
Router#sh int fa 0
FastEthernet0 is up, line protocol is up
Hardware is Fast Ethernet, address is 0019.e75b.1455 (bia 0019.e75b.1455)
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 8/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output never, output hang never
Last clearing of "show interface" counters 00:00:40
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3354000 bits/sec, 353 packets/sec
5 minute output rate 73000 bits/sec, 135 packets/sec
     121840 packets input, 148008746 bytes, 0 no buffer
     Received 124 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     46314 packets output, 2992014 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     2 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Router#sh int fa 1
FastEthernet1 is up, line protocol is up
Hardware is Fast Ethernet, address is 0019.e75b.1456 (bia 0019.e75b.1456)
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 11/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:00:49
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 103000 bits/sec, 190 packets/sec
5 minute output rate 4675000 bits/sec, 485 packets/sec
     58198 packets input, 3774735 bytes, 0 no buffer
     Received 98 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     152620 packets output, 185746450 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

Seb Rupik · ‎08-29-2013

Looks like every packet is handled by the processor:

Router#sh interfaces stats

FastEthernet0

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 57707 69990129 21779 1412159

Route cache 0 0 0 0

Total 57707 69990129 21779 1412159

FastEthernet1

Switching path Pkts In Chars In Pkts Out Chars Out

Processor 20422 1341906 53711 65306522

Route cache 0 0 0 0

Total 20422 1341906 53711 65306522

...so to answer your original question, this appears to be normal behaviour!

catalin.stan · ‎08-29-2013

OK. Thank you!

I see that the route cache for the interfaces vlan1 and vlan192 is very high. Is this the normal behavior ?

It's very strange that I can't download large files from ftp with this model.

Will an upgrade of RAM help ?