High CPU utilization on 7200 Router due to interrupts

Unanswered Question
Dec 5th, 2012
User Badges:

Hi Everybody,

 

I have on  7200 series router with NPE-G1 module which is facing high CPU utilization. I have not found any particular process causing this high CPU utilization as it is caused due to interrupts. I have already enabled fast switching by "ip cef" command. Please suggest how to normalize the utilization as it is impacting the network and causing slowness. Please find below the output of show process cpu and also find attached the show tech of the device. Also let me know if any other output is required.

 

TCISL-GGCOLO-7206-RO#sh proc cpu sorted

CPU utilization for five seconds: 75%/71%; one minute: 70%; five minutes: 69%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

184   3311658521741691145        190  1.47%  1.41%  1.41%   0 IP Input       

  41   351217284  21287490      16498  1.14%  1.23%  1.22%   0 Per-Second Jobs

  25    76426548 963268136         79  0.24%  0.29%  0.29%   0 ARP Input      

287     1505324 629990778          2  0.16%  0.10%  0.08%   0 PPP Events     

  37    28118300  16690436       1684  0.16%  0.11%  0.10%   0 Net Background 

193         140      2417         57  0.16%  0.02%  0.00%   2 Virtual Exec   

113      431068  20149940         21  0.08%  0.04%  0.06%   0 hal_periodic 1 

286     1082160 628735884          1  0.08%  0.11%  0.10%   0 PPP manager    

224        4720  20151164          0  0.08%  0.00%  0.00%   0 RUDPV1 Main Proc

207     1174992  36046711         32  0.08%  0.10%  0.08%   0 CEF process    

177      545544  20149939         27  0.08%  0.05%  0.06%   0 hal_periodic 2 

274     4786700   4074008       1174  0.08%  0.04%  0.06%   0 Compute load avg

186     1293884  20640618         62  0.08%  0.05%  0.06%   0 CDP Protocol   

300     1078960  23234478         46  0.08%  0.00%  0.00%   0 OSPF-1 Router  

  15           0         1          0  0.00%  0.00%  0.00%   0 Crash writer   

  14           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager 

  17     4630400  20156773        229  0.00%  0.00%  0.00%   0 EnvMon         

  13           0        81          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT

  12         700      6153        113  0.00%  0.00%  0.00%   0 AAA high-capacit

  16           0         1          0  0.00%  0.00%  0.00%   0 RO Notify Timers

  21        2544  20146727          0  0.00%  0.00%  0.00%   0 IPC Periodic Tim

  11           0         2          0  0.00%  0.00%  0.00%   0 ATM VC Auto Crea

  23           0         1          0  0.00%  0.00%  0.00%   0 IPC Seat Manager

  24           0         1          0  0.00%  0.00%  0.00%   0 IPC BackPressure

  18           0         1          0  0.00%  0.00%  0.00%   0 OIR Handler    

  10           0         2          0  0.00%  0.00%  0.00%   0 ATM AutoVC Perio

  27         504     12210         41  0.00%  0.00%  0.00%   0 DDR Timers     

  19          32    335833          0  0.00%  0.00%  0.00%   0 IPC Dynamic Cach

  20           0         1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager

  22        2216  20146739          0  0.00%  0.00%  0.00%   0 IPC Deferred Por

  31       19080  20149945          0  0.00%  0.00%  0.00%   0 GraphIt        

  32           0         2          0  0.00%  0.00%  0.00%   0 Dialer event   

  33           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client

   7           0         2          0  0.00%  0.00%  0.00%   0 Timers         

  36           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd 

   6       60596    222285        272  0.00%  0.00%  0.00%   0 Pool Manager   

  38           0         2          0  0.00%  0.00%  0.00%   0 IDB Work       

  39        1680     35559         47  0.00%  0.00%  0.00%   0 Logger         

  40        4016  20146730          0  0.00%  0.00%  0.00%   0 TTY Background 

   5    27720652   2487467      11144  0.00%  0.06%  0.05%   0 Check heaps    

  26        6176   4029876          1  0.00%  0.00%  0.00%   0 HC Counter Timer

  28           0         5          0  0.00%  0.00%  0.00%   0 Entity MIB API 

  44           0         1          0  0.00%  0.00%  0.00%   0 POS APS Event Pr

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  45           0         2          0  0.00%  0.00%  0.00%   0 VNM DSPRM MAIN 

  46           0         1          0  0.00%  0.00%  0.00%   0 Coproc Event Pro

  47           0         1          0  0.00%  0.00%  0.00%   0 CES Line Conditi

  48           0         2          0  0.00%  0.00%  0.00%   0 Flash MIB Update

  29         228     18090         12  0.00%  0.00%  0.00%   0 EEM ED Syslog  

  50           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/1 bert

  30           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect

  52           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/3 bert

  35           0         1          0  0.00%  0.00%  0.00%   0 Inode Table Dest

  54           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/5 bert

  55           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/6 bert

  56           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/7 bert

  57           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/8 bert

  58           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/9 bert

  59           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/10 ber

  60           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/11 ber

  42           0         1          0  0.00%  0.00%  0.00%   0 CSP Timer      

  62           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/13 ber

  63           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/14 ber

  64           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/15 ber

  65           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/16 ber

  66           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/17 ber

  67           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/18 ber

  68           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/19 ber

  69           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/20 ber

  70           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/21 ber

  71           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/1 bert

  72           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/2 bert

   4           0         1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN     

   9           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer 

  75           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/5 bert

  43           0         1          0  0.00%  0.00%  0.00%   0 SONET alarm time

  49           0         1          0  0.00%  0.00%  0.00%   0 Flash Card Oir 

  78           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/8 bert

  79           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/9 bert

  80           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/10 ber

  81           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/11 ber

  82           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/12 ber

  83           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/13 ber

  84           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/14 ber

  85           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/15 ber

  86           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/16 ber

  87           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/17 ber

  88           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/18 ber

  51           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/2 bert

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  90           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/20 ber

  91           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/21 ber

  92           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/1 bert

  53           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/4 bert

   2      226688   4029991         56  0.00%  0.01%  0.00%   0 Load Meter     

  95           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/4 bert

  96           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/5 bert

  61           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.1/12 ber

  98           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/7 bert

  99           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/8 bert

100           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/9 bert

101           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/10 ber

102           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/11 ber

103           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/12 ber

  73           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/3 bert

105           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/14 ber

106           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/15 ber

107           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/16 ber

108           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/17 ber

109           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/18 ber

110           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/19 ber

111           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/20 ber

112           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/21 ber

  74           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/4 bert

114           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/1 bert

115           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/2 bert

116           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/3 bert

117           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/4 bert

118           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/5 bert

119           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/6 bert

120           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/7 bert

121           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/8 bert

122           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/9 bert

123           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/10 ber

124           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/11 ber

125           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/12 ber

126           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/13 ber

127           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/14 ber

128           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/15 ber

129           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/16 ber

130           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/17 ber

131           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/18 ber

132           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/19 ber

133           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/20 ber

134           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.1/21 ber

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

135           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/1 bert

136           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/2 bert

137           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/3 bert

   8           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun

139           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/5 bert

140           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/6 bert

141           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/7 bert

142           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/8 bert

   3         328     32620         10  0.00%  0.00%  0.00%   0 TACACS+        

  76           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/6 bert

145           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/11 ber

146           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/12 ber

147           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/13 ber

148           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/14 ber

149           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/15 ber

  77           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/7 bert

151           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/17 ber

  89           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.2/19 ber

153           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/19 ber

154           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/20 ber

155           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/21 ber

156           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/1 bert

157           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/2 bert

158           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/3 bert

159           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/4 bert

160           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/5 bert

161           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/6 bert

162           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/7 bert

163           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/8 bert

164           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/9 bert

165           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/10 ber

166           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/11 ber

167           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/12 ber

168           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/13 ber

169           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/14 ber

170           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/15 ber

171           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/16 ber

172           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/17 ber

173           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/18 ber

174           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/19 ber

175           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/20 ber

176           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.3/21 ber

  93           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/2 bert

178           0         8          0  0.00%  0.00%  0.00%   0 TurboACL       

179           0         2          0  0.00%  0.00%  0.00%   0 TurboACL chunk 

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

180         984      9503        103  0.00%  0.00%  0.00%   0 AAA Server     

181        1100     11699         94  0.00%  0.00%  0.00%   0 AAA ACCT Proc  

182           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr

183           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R

  94           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/3 bert

185           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl

  97           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/6 bert

187           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager     

188         972   2686633          0  0.00%  0.00%  0.00%   0 SSS Test Client

189           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana

190       78352  78699756          0  0.00%  0.00%  0.00%   0 SSS Feature Time

191         740     13735         53  0.00%  0.00%  0.00%   0 PPP Hooks      

192           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana

104           0         3          0  0.00%  0.00%  0.00%   0 hal 1/0.3/13 ber

194           0         1          0  0.00%  0.00%  0.00%   0 VPDN call manage

195           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket proce

196           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS manager

197           0         2          0  0.00%  0.00%  0.00%   0 L2TP mgmt daemon

198           0         8          0  0.00%  0.00%  0.00%   0 IPv6 RIB Redistr

199       37548   4799266          7  0.00%  0.00%  0.00%   0 IP Background  

200      118704    982981        120  0.00%  0.00%  0.00%   0 IP RIB Update  

201           0         2          0  0.00%  0.00%  0.00%   0 SNMP Timers    

202       61684     24579       2509  0.00%  0.00%  0.00%   0 PPP IP Route   

203        2400     24939         96  0.00%  0.00%  0.00%   0 PPP IPCP       

204           0         2          0  0.00%  0.00%  0.00%   0 ATM OAM Input  

205           0         2          0  0.00%  0.00%  0.00%   0 ATM OAM TIMER  

206      318040    338733        938  0.00%  0.00%  0.00%   0 Adj Manager    

   1           0        65          0  0.00%  0.00%  0.00%   0 Chunk Manager  

208      121848  10170577         11  0.00%  0.00%  0.00%   0 TCP Timer      

209        5256      5885        893  0.00%  0.00%  0.00%   0 TCP Protocols  

210      452088  33290469         13  0.00%  0.00%  0.00%   0 Socket Timers  

211         468     67172          6  0.00%  0.00%  0.00%   0 HTTP CORE      

212         140     44028          3  0.00%  0.00%  0.00%   0 Transport Port A

213           0         1          0  0.00%  0.00%  0.00%   0 IP Traceroute  

214          44       466         94  0.00%  0.00%  0.00%   0 RARP Input     

215    11471192  66714322        171  0.00%  0.03%  0.05%   0 DHCPD Receive  

216        1296    335828          3  0.00%  0.00%  0.00%   0 IP Cache Ager  

217           0         1          0  0.00%  0.00%  0.00%   0 COPS           

218           0         2          0  0.00%  0.00%  0.00%   0 PPP SSS        

219           0         2          0  0.00%  0.00%  0.00%   0 PPP Bind       

220       56072 201283790          0  0.00%  0.02%  0.02%   0 RBSCP Background

221           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall      

222           0         2          0  0.00%  0.00%  0.00%   0 X.25 Background

223           0         2          0  0.00%  0.00%  0.00%   0 Dialer Forwarder

138           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/4 bert

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

225           0         1          0  0.00%  0.00%  0.00%   0 bsm_timers     

226        1360  20146733          0  0.00%  0.00%  0.00%   0 bsm_xmt_proc   

227           0         2          0  0.00%  0.00%  0.00%   0 RLM groups Proce

228           0         3          0  0.00%  0.00%  0.00%   0 gk process     

229           0         2          0  0.00%  0.00%  0.00%   0 Border Element p

230        1008    806000          1  0.00%  0.00%  0.00%   0 CRM_CALL_UPDATE_

231           0         2          0  0.00%  0.00%  0.00%   0 LOCAL AAA      

232           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA     

233           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA       

234       53864     94613        569  0.00%  0.00%  0.00%   0 TPLUS          

235           0         1          0  0.00%  0.00%  0.00%   0 Key chain liveke

236           0         2          0  0.00%  0.00%  0.00%   0 VSP_MGR        

237           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr

238           0         1          0  0.00%  0.00%  0.00%   0 VoIP AAA       

239           0         1          0  0.00%  0.00%  0.00%   0 QOS_MODULE_MAIN

240           0         1          0  0.00%  0.00%  0.00%   0 RPMS_PROC_MAIN 

241           0         1          0  0.00%  0.00%  0.00%   0 CCVPM_HTSP     

242           0         1          0  0.00%  0.00%  0.00%   0 CCVPM_R2       

243           0         1          0  0.00%  0.00%  0.00%   0 CCSWVOICE      

244           0         1          0  0.00%  0.00%  0.00%   0 xcpa-driver    

245           4         1       4000  0.00%  0.00%  0.00%   0 http client proc

246        1728   2014974          0  0.00%  0.00%  0.00%   0 OSPFv3-10 Hello

247         100     22389          4  0.00%  0.00%  0.00%   0 FB/KS Log HouseK

248           0         2          0  0.00%  0.00%  0.00%   0 EPHONE MWI BG Pr

249           0         2          0  0.00%  0.00%  0.00%   0 Proxy Session Ap

250         384   2014966          0  0.00%  0.00%  0.00%   0 RMON Recycle Pro

251           0         2          0  0.00%  0.00%  0.00%   0 RMON Deferred Se

252           0         1          0  0.00%  0.00%  0.00%   0 SYSMGT Events  

253        1080     12735         84  0.00%  0.00%  0.00%   0 AAA SEND STOP EV

254           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps   

255           0         1          0  0.00%  0.00%  0.00%   0 IMA Traps      

256           0         3          0  0.00%  0.00%  0.00%   0 EEM ED CLI     

257           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Counter 

258           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Interface

259           0         3          0  0.00%  0.00%  0.00%   0 EEM ED IOSWD   

260           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Memory-th

261           0         2          0  0.00%  0.00%  0.00%   0 EEM ED None    

262           0         2          0  0.00%  0.00%  0.00%   0 EM ED OIR      

263           0         2          0  0.00%  0.00%  0.00%   0 EEM ED SNMP    

264         128    335855          0  0.00%  0.00%  0.00%   0 EEM ED Timer   

265        1096   4041466          0  0.00%  0.00%  0.00%   0 EEM Server     

266           0         1          0  0.00%  0.00%  0.00%   0 RMON Packets   

267        1960     17865        109  0.00%  0.00%  0.00%   0 Syslog         

268           0         1          0  0.00%  0.00%  0.00%   0 VPDN Scal      

269        2468  20136413          0  0.00%  0.00%  0.00%   0 trunk conditioni

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

270           0         1          0  0.00%  0.00%  0.00%   0 trunk conditioni

271           0         2          0  0.00%  0.00%  0.00%   0 EEM Policy Direc

272       37660     19608       1920  0.00%  0.00%  0.07%   3 Virtual Exec   

273     2334936  69759518         33  0.00%  0.01%  0.00%   0 Net Input      

143           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/9 bert

275     7435728    375772      19787  0.00%  0.01%  0.00%   0 Per-minute Jobs

276         452    167918          2  0.00%  0.00%  0.00%   0 CC-API_VCM     

277       79220 201307791          0  0.00%  0.03%  0.02%   0 CCPROXY_CT     

278      311348    699974        444  0.00%  0.00%  0.00%   0 CEF Scanner    

280           0         2          0  0.00%  0.00%  0.00%   0 IP Flow Backgrou

281          92    335896          0  0.00%  0.00%  0.00%   0 IPv6 Input     

282        2100   4904125          0  0.00%  0.00%  0.00%   0 IPv6 IDB       

283         516    115849          4  0.00%  0.00%  0.00%   0 IPv6 ND        

284      148220  80588022          1  0.00%  0.01%  0.00%   0 MLD            

285       41720  19757276          2  0.00%  0.00%  0.00%   0 OSPFv3-10 Router

144           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/10 ber

150           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/16 ber

288        3132  20147912          0  0.00%  0.00%  0.00%   0 Multilink PPP  

289      623396   8131126         76  0.00%  0.00%  0.00%   0 OSPF-1 Hello   

290         304       321        947  0.00%  0.00%  0.00%   0 TCP Listener   

291     1066944   6757845        157  0.00%  0.00%  0.00%   0 IP SNMP        

292      454052   3841146        118  0.00%  0.00%  0.00%   0 PDU DISPATCHER 

293     2206124   3846304        573  0.00%  0.00%  0.00%   0 SNMP ENGINE    

294           0         2          0  0.00%  0.00%  0.00%   0 IP SNMPV6      

295      939228     49998      18785  0.00%  0.00%  0.00%   0 SNMP ConfCopyPro

296           0         1          0  0.00%  0.00%  0.00%   0 SNMP Traps     

297      170172  20492075          8  0.00%  0.00%  0.00%   0 NTP            

298         256    167917          1  0.00%  0.00%  0.00%   0 DHCPD Timer    

299          84    335838          0  0.00%  0.00%  0.00%   0 DHCPD Database 

152           0         3          0  0.00%  0.00%  0.00%   0 hal 2/0.2/18 ber

301      711820  57447570         12  0.00%  0.00%  0.00%   0 BGP Router     

302     2376008   9713766        244  0.00%  0.00%  0.00%   0 BGP I/O        

303     1627216   1349715       1205  0.00%  0.00%  0.00%   0 BGP Scanner   

TCISL-GGCOLO-7206-RO#

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Shaheen Sidana Thu, 12/06/2012 - 02:17
User Badges:

Hii,


Thanks for the link. I have already gone through this document but nothing found relevant which can solve the issue. Please go through the show tech once and suggest what I can do to lower down the utilization.


Thanks ,

Shaheen

Pulikkal Sekhar... Thu, 12/06/2012 - 02:32
User Badges:
  • Cisco Employee,

7200 is a CEF router. So if the traffic rate goes high, you will see the CPU utilization going high because of interrupts because they use CPU to switch the traffic


As per below link, the cef switching performacne for NPE-G1 is 521 Mbps


http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf


your box is switching more than 400 Mbps of traffic


GigabitEthernet0/1 is up, line protocol is up

  30 second input rate 317693000 bits/sec, 42778 packets/sec

  30 second output rate 111276000 bits/sec, 34165 packets/sec


GigabitEthernet0/3 is up, line protocol is up

  5 minute input rate 93029000 bits/sec, 29066 packets/sec

  5 minute output rate 270483000 bits/sec, 35920 packets/sec


So what you are sseeing should be an expected behavior for the amount of traffic you have


Thanks

Raju

Joseph W. Doherty Thu, 12/06/2012 - 07:47
User Badges:
  • Super Bronze, 10000 points or more

Disclaimer


The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.


Liability Disclaimer


In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.


Posting


I agree with Raju, CPU usage appears reasonable for traffic volume.


If the CPU routinely doesn't top 75%, you might have other cause(s) for your "slowness" issue.

AmitLohumi Thu, 12/06/2012 - 22:54
User Badges:

Hi All,


Following are a a coupls of CEF outputs from the same router :


###################################Show cef not-cef-switched######################
TCISL-GGCOLO-7206-RO#Show cef not-cef-switched
CEF Packets passed on to next switching layer
Slot  No_adj No_encap Unsupp'ted Redirect  Receive  Options   Access     Frag
RP    114933       0           0        0   269738      257        0        0

##################################Show cef drop##################################
TCISL-GGCOLO-7206-RO#Show cef drop
CEF Drop Statistics
Slot  Encap_fail  Unresolved Unsupported    No_route      No_adj  ChkSum_Err
RP    2486523203           4      139268     8313115           0        2721



We can see a large no. of 'Encap_fail' packets which would indicate the no. of packets throttled by CEF after exceeding the limit for packets punted to the processor due to missing CEF adjacency information. Would this not indicate malicious traffic from amy of the overloaded ports ?? I am also looking at Bug ID : CSCso80549.


Kindly suggest your views on this .


Thanks

Amit

Joseph W. Doherty Fri, 12/07/2012 - 04:27
User Badges:
  • Super Bronze, 10000 points or more

Disclaimer


The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.


Liability Disclaimer


In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.


Posting


My view would be since the encap_fail indicates a possible CEF adjacency issue, you might examine Technote:

http://www.cisco.com/en/US/tech/tk827/tk831/technologies_tech_note09186a0080094303.shtml and see if there's anything that helps.  Otherwise, if you have a support contract, you might open a case with TAC.

AmitLohumi Sat, 12/08/2012 - 09:25
User Badges:

Hi Joseph,


Thanks for sharing your views. I was going through this document which does not seem to suggest a straightforward way to troubleshoot this problem. What do you think about this Bug that seems to match the symptoms in this case. ?


Regards,

Amit

Giuseppe Larosa Mon, 12/10/2012 - 02:04
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Amit,

the router has many subinterfaces each of them with a policer applied inbound and a policer applied outbound of different rates


The aggregate traffic volume is high as noted by Raju and I agree that given the software only nature of C7200 with NPE-G1 your device is not behaving bad but as it can be expected.


About the bug you are mentioning. I find this bug description to be vague.


your router is running

disk2:c7200-is-mz.124-16b.bin


Release 12.4(16)B is not listed among the affected IOS images in the list of affected IOS versions.


Given the number of subinterfaces defined with policers and the level of aggregate traffic it looks like that the router is behaving correctly.


the order of packets received and sent on GE interrfaces is like the following from show tech

Gi0/1

17336889990 packets input

15158090407 packets output


Gi0/2

1317444924 packets input

1623446401 packets output


Gi0/3

12719037739 packets input

14364366054 packets output


The rate of enc failure compared to packets input on gi0/1


2486523203 /17336889990 = 0.14%


Even if the number looks like great it is not in comparison to total packets passing on interfaces

So this can be seen as a confirmation the trouble described in the bug should not apply to your router.


Hope to help

Giuseppe

AmitLohumi Wed, 12/12/2012 - 02:03
User Badges:

Hi Giuseppe,


Many thanks for your detailed analysis. Have a query though.


If the CEF switching limit for NPE-G1 has not yet been crossed over, why should there be any interrupts going on to the CPU on account of the high volume of traffic.


Regards,

Amit

AmitLohumi Wed, 12/12/2012 - 02:05
User Badges:


Would it be because CEF switching s being implemented in Software without dedicated ASICs ?


Regards,

Amit

Pulikkal Sekhar... Wed, 12/12/2012 - 05:18
User Badges:
  • Cisco Employee,

Hi Amit,


7200 is a CEF router. It doesn't use any ASIC to forward the packet. Any CEF router uses interrupts to switch the packets


Thanks

Raju

AmitLohumi Wed, 12/12/2012 - 11:57
User Badges:

Hi raj,


What would you say of the possibility of a DOS attack, since the utilization has shot up without any major changesbeing made. Do you think using the 'Scheduler Allocate' would help here without any negative side-affects or do we need to go for a traffic analysis exercise using Wireshark to identify the source of this unexpected traffic ?


Regards,

Amit

Pulikkal Sekhar... Wed, 12/12/2012 - 18:49
User Badges:
  • Cisco Employee,

Hi Amit,


My suggestion would be to sniff the traffic if you are seeing increased traffc rate on any of the interfaces


Thanks

Raju

Joseph W. Doherty Thu, 12/13/2012 - 02:05
User Badges:
  • Super Bronze, 10000 points or more

Disclaimer


The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.


Liability Disclaimer


In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.


Posting


Amit, you seem to discounting the possibility that the high interrupt CPU might be "normal" as it might be just caused by your traffic loading.


Two things that might help demonstrate this could be your root cause, if you can graph overall CPU utilization with overall traffic volume transiting the router, you might see a very strong relationship.


Second, pick a traffic generator of your choice and push some synthetic traffic through your router.  Watch interrupt CPU as you push different volumes of such test traffic.  Again, you likely see interrupt CPU correspond with the amount of test traffic. 

Actions

This Discussion

Related Content