High CPU load on 7600 router

Unanswered Question
Nov 20th, 2009
User Badges:

Hello :


  We have been observing high CPU load on 7600 router.


------------------ show process cpu ------------------


CPU utilization for five seconds: 99%/7%; one minute: 66%; five minutes: 54%


201        7984      3663       2179  0.00%  0.00%  0.00%   0 Tagcon Addr     
202    13173672  93099702        141  0.08%  0.03%  0.00%   0 OSPF-1 Router   
203  1734370096 574262673       3020 43.25% 45.25% 42.53%   0 BGP Router      
204    14859436 203301904         73  0.08%  0.07%  0.08%   0 BGP I/O         
205   170487696   3326213      51255  0.00%  0.61%  1.05%   0 BGP Scanner     
207        8608   1140673          7  0.00%  0.00%  0.00%   0 IPHC Admin  



From BGP peers we see the following;



edge2-cky#sh ip bgp summ
BGP router identifier 86.43.248.2, local AS number 5466
BGP table version is 176686504, main routing table version 176686504
338525 network entries using 39607425 bytes of memory
1015435 path entries using 52802620 bytes of memory
80214/57143 BGP path/bestpath attribute entries using 9946536 bytes of memory
560 BGP rrinfo entries using 13440 bytes of memory
60898 BGP AS-PATH entries using 1553658 bytes of memory
3660 BGP community entries using 271030 bytes of memory
179 BGP extended community entries using 4296 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 104199005 total bytes of memory
BGP activity 14841182/14496911 prefixes, 91093682/90068358 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
95.45.198.66    4 39915       0       0        0    0    0 never    Active
180.134.191.148 4  5466 59102668  675415 176686507    6    0 15w5d      338523
180.134.191.149 4  5466 59167342  675225 176686507    1    0 15w5d      338520
180.134.191.152 4  5466 61355269  675417 176686507    4    0 15w5d      338390


edge2-cky#sh ip bgp vpnv
edge2-cky#sh ip bgp vpnv4 all summ
BGP router identifier 86.43.248.2, local AS number 5466
BGP table version is 3727638, main routing table version 3727638
5018 network entries using 692484 bytes of memory
7749 path entries using 526932 bytes of memory
31 multipath network entries and 72 multipath paths
80237/383 BGP path/bestpath attribute entries using 9949388 bytes of memory
572 BGP rrinfo entries using 13792 bytes of memory
60906 BGP AS-PATH entries using 1553898 bytes of memory
3665 BGP community entries using 271214 bytes of memory
183 BGP extended community entries using 4392 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 13012100 total bytes of memory
BGP activity 14841188/14496911 prefixes, 91093711/90068380 paths, scan interval 15 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
33.43.146.202   4 65500  675462  677872  3727638    0    0 25w4d          13
33.43.230.6     4 65500  388757  567485  3727638    0    0 25w4d           6
33.43.230.10    4 65500  320616  320991  3727638    0    0 14w0d           3
33.43.230.14    4 65500  361344  405323  3727638    0    0 34w3d           9
33.43.230.18    4 65500  340442  503526  3727638    0    0 6d18h           1
33.43.230.22    4 65500   82037   82104  3727638    0    0 7w4d            2
33.43.230.26    4 65500       0       0        0    0    0 never    Active
33.43.230.30    4 65500   70335   87735  3727638    0    0 5w4d            3
33.43.230.34    4 65500       0       0        0    0    0 never    Idle
176.134.127.130 4  5466 36361403  678080  3727638    0    0 50w6d        2642
186.134.127.142 4  5466 45709963  678109  3727638    0    0 8w0d         2642


The IOS is running is 12.4(8d) . Cu has several 7200 NPE-G1 but none of them show such a CPU load. Next I plean to do CPU profiling.

But before that I would like to know if anyone has come across such an issue and how to troubleshoot it fi

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Giuseppe Larosa Fri, 11/20/2009 - 09:58
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Arjun,

first of all if the IOS image is


>>

The IOS is running is 12.4(8d) . Cu has several 7200 NPE-G1 but none of them show such a CPU load. Next I plean to do CPU profiling.

But before that I would like to know if anyone has come across such an issue and how to troubleshoot.



this device is not a C7600 that is the device involved in your other thread but a C7200 VXR with an NPE-G1 as the other ones you are mentioning.


I suggest you to take from time to time the output of


sh proc cpu sorted 1min


so that you can see what processes are using most resources.


if this is a C7206 it is still a software based where CEF can be used but it misses dedicated HW ASICs to perform forwarding.


When I tested NPE-G1 I was able to load cpu to 100% by simply sending 500 Mbps of traffic made of small packets.


So compare also the traffic volumes handled by the different c7206 you have.


Hope to help

Giuseppe

Yasir Ashfaque Sat, 11/21/2009 - 03:02
User Badges:

12.4d on 7600 ??, which superviso engine is it ?? and whatz the output of


show ip route summary

sampusarkar Sun, 11/22/2009 - 00:40
User Badges:

Hello :


  Sorry for the confusion. It is 7206VXR with NPE-G1. The thing is that the CPU load fluctuates bw 100% 70% depending on traffic.

I would appreciate if you could let me know how to troubleshoot the issue further.


Regards

Arjun

Giuseppe Larosa Sun, 11/22/2009 - 06:12
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Arjun,

as I wrote in my first post being a software based router, main cpu load increases with traffic volume.


Check if CEF is enabled.


cpu can go to 100% even with CEF enabled when the number of packets processed per second is near to 1,000,000.


How much traffic is on the C7206 ports, how many packets per second?


you need to sum tx and rx pps on all interfaces that gives you the total load in pps


Hope to help

Giuseppe

Yasir Ashfaque Sun, 11/22/2009 - 06:19
User Badges:

I think its issue of BGP, as it has already consumed 45% of CPU, what is the output of show ip route summary, and also see if route dampening is enabled or not.

sampusarkar Sun, 11/22/2009 - 08:46
User Badges:

Hello Vasir:


   Thanks. Route dampending is disable. I will paste the "show ip route sum" tomorrow.


Regards

Arjun

sampusarkar Mon, 11/23/2009 - 01:48
User Badges:

Hello Vasir :


   Please find the show ip route summ output;



sh ip route summ


IP routing table name is Default-IP-Routing-Table(0) IP routing table maximum-paths is 16
Route Source    Networks          Subnets     Overhead     Memory
(bytes)
connected        0            4                324
544
static           0            1                72
136
ospf 1           7            1374          99432
187816
  Intra-area: 703 Inter-area: 608 External-1: 52 External-2: 18
  NSSA External-1: 0 NSSA External-2: 0
bgp 5466           134714       203723     24367464      46455832
  External: 0 Internal: 338437 Local: 0
internal              3939
4553484
Total                 138660       205102     24467292
51197812

Yasir Ashfaque Mon, 11/23/2009 - 03:33
User Badges:

I think its over burdened, BGP is consuming hell lot of Memory and CPU Utlization, i think you will have to filter unneccessory BGP Routes, or best way to handle it open a TICKET with Cisco TAC, and make it sure with Cisco if 7200 NP1 can support this kind of burden, it could be a BUG too, why dont you move to 12.2SR series ??

sampusarkar Sun, 11/22/2009 - 08:44
User Badges:

Hello Giuseppe :


  Yes . CEF is enabled. I also do not see any traffic being process switched with " show interface switching" output. Here is an example

of one show interface output.



Serial3/0/1:1 is up, line protocol is up
  Hardware is CE3
  Description: By VPNSC:   Internet address is

  MTU 1500 bytes, BW 1984 Kbit, DLY 20000 usec,
     reliability 255/255, txload 55/255, rxload 33/255
  Encapsulation PPP, LCP Open
  Open: IPCP, crc 16, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:13, output 00:00:00, output hang never
  Last clearing of "show interface" counters 01:23:18
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1
  Queueing strategy: Class-based queueing
  Output queue: 2/1000/64/1 (size/max total/threshold/drops)
     Conversations  1/46/256 (active/max active/max total)
     Reserved Conversations 1/1 (allocated/max allocated)
     Available Bandwidth 496 kilobits/sec
  5 minute input rate 260000 bits/sec, 522 packets/sec
  5 minute output rate 428000 bits/sec, 403 packets/sec
     2282137 packets input, 141041214 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     1727450 packets output, 217454949 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions no alarm present
  Timeslot(s) Used:1-31, subrate: 64Kb/s, transmit delay is 0 flags

mbambic Wed, 12/23/2009 - 07:52
User Badges:

Try this command:


sh proc cpu sorted 5min | exc 0.00


Paste back the output and it will show the highest used processes sorted by the recent runtime


Here's some docs to help:

High CPU Utilization

Actions

This Discussion