11-20-2009 07:24 AM - edited 03-04-2019 06:45 AM
Hello :
We have been observing high CPU load on 7600 router.
------------------ show process cpu ------------------
CPU utilization for five seconds: 99%/7%; one minute: 66%; five minutes: 54%
201 7984 3663 2179 0.00% 0.00% 0.00% 0 Tagcon Addr
202 13173672 93099702 141 0.08% 0.03% 0.00% 0 OSPF-1 Router
203 1734370096 574262673 3020 43.25% 45.25% 42.53% 0 BGP Router
204 14859436 203301904 73 0.08% 0.07% 0.08% 0 BGP I/O
205 170487696 3326213 51255 0.00% 0.61% 1.05% 0 BGP Scanner
207 8608 1140673 7 0.00% 0.00% 0.00% 0 IPHC Admin
From BGP peers we see the following;
edge2-cky#sh ip bgp summ
BGP router identifier 86.43.248.2, local AS number 5466
BGP table version is 176686504, main routing table version 176686504
338525 network entries using 39607425 bytes of memory
1015435 path entries using 52802620 bytes of memory
80214/57143 BGP path/bestpath attribute entries using 9946536 bytes of memory
560 BGP rrinfo entries using 13440 bytes of memory
60898 BGP AS-PATH entries using 1553658 bytes of memory
3660 BGP community entries using 271030 bytes of memory
179 BGP extended community entries using 4296 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 104199005 total bytes of memory
BGP activity 14841182/14496911 prefixes, 91093682/90068358 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
95.45.198.66 4 39915 0 0 0 0 0 never Active
180.134.191.148 4 5466 59102668 675415 176686507 6 0 15w5d 338523
180.134.191.149 4 5466 59167342 675225 176686507 1 0 15w5d 338520
180.134.191.152 4 5466 61355269 675417 176686507 4 0 15w5d 338390
edge2-cky#sh ip bgp vpnv
edge2-cky#sh ip bgp vpnv4 all summ
BGP router identifier 86.43.248.2, local AS number 5466
BGP table version is 3727638, main routing table version 3727638
5018 network entries using 692484 bytes of memory
7749 path entries using 526932 bytes of memory
31 multipath network entries and 72 multipath paths
80237/383 BGP path/bestpath attribute entries using 9949388 bytes of memory
572 BGP rrinfo entries using 13792 bytes of memory
60906 BGP AS-PATH entries using 1553898 bytes of memory
3665 BGP community entries using 271214 bytes of memory
183 BGP extended community entries using 4392 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 13012100 total bytes of memory
BGP activity 14841188/14496911 prefixes, 91093711/90068380 paths, scan interval 15 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
33.43.146.202 4 65500 675462 677872 3727638 0 0 25w4d 13
33.43.230.6 4 65500 388757 567485 3727638 0 0 25w4d 6
33.43.230.10 4 65500 320616 320991 3727638 0 0 14w0d 3
33.43.230.14 4 65500 361344 405323 3727638 0 0 34w3d 9
33.43.230.18 4 65500 340442 503526 3727638 0 0 6d18h 1
33.43.230.22 4 65500 82037 82104 3727638 0 0 7w4d 2
33.43.230.26 4 65500 0 0 0 0 0 never Active
33.43.230.30 4 65500 70335 87735 3727638 0 0 5w4d 3
33.43.230.34 4 65500 0 0 0 0 0 never Idle
176.134.127.130 4 5466 36361403 678080 3727638 0 0 50w6d 2642
186.134.127.142 4 5466 45709963 678109 3727638 0 0 8w0d 2642
The IOS is running is 12.4(8d) . Cu has several 7200 NPE-G1 but none of them show such a CPU load. Next I plean to do CPU profiling.
But before that I would like to know if anyone has come across such an issue and how to troubleshoot it fi
11-20-2009 09:58 AM
Hello Arjun,
first of all if the IOS image is
>>
The IOS is running is 12.4(8d) . Cu has several 7200 NPE-G1 but none of them show such a CPU load. Next I plean to do CPU profiling.
But before that I would like to know if anyone has come across such an issue and how to troubleshoot.
this device is not a C7600 that is the device involved in your other thread but a C7200 VXR with an NPE-G1 as the other ones you are mentioning.
I suggest you to take from time to time the output of
sh proc cpu sorted 1min
so that you can see what processes are using most resources.
if this is a C7206 it is still a software based where CEF can be used but it misses dedicated HW ASICs to perform forwarding.
When I tested NPE-G1 I was able to load cpu to 100% by simply sending 500 Mbps of traffic made of small packets.
So compare also the traffic volumes handled by the different c7206 you have.
Hope to help
Giuseppe
11-21-2009 03:02 AM
12.4d on 7600 ??, which superviso engine is it ?? and whatz the output of
show ip route summary
11-22-2009 12:40 AM
Hello :
Sorry for the confusion. It is 7206VXR with NPE-G1. The thing is that the CPU load fluctuates bw 100% 70% depending on traffic.
I would appreciate if you could let me know how to troubleshoot the issue further.
Regards
Arjun
11-22-2009 06:12 AM
Hello Arjun,
as I wrote in my first post being a software based router, main cpu load increases with traffic volume.
Check if CEF is enabled.
cpu can go to 100% even with CEF enabled when the number of packets processed per second is near to 1,000,000.
How much traffic is on the C7206 ports, how many packets per second?
you need to sum tx and rx pps on all interfaces that gives you the total load in pps
Hope to help
Giuseppe
11-22-2009 06:19 AM
I think its issue of BGP, as it has already consumed 45% of CPU, what is the output of show ip route summary, and also see if route dampening is enabled or not.
11-22-2009 08:46 AM
Hello Vasir:
Thanks. Route dampending is disable. I will paste the "show ip route sum" tomorrow.
Regards
Arjun
11-23-2009 01:48 AM
Hello Vasir :
Please find the show ip route summ output;
sh ip route summ
IP routing table name is Default-IP-Routing-Table(0) IP routing table maximum-paths is 16
Route Source Networks Subnets Overhead Memory
(bytes)
connected 0 4 324
544
static 0 1 72
136
ospf 1 7 1374 99432
187816
Intra-area: 703 Inter-area: 608 External-1: 52 External-2: 18
NSSA External-1: 0 NSSA External-2: 0
bgp 5466 134714 203723 24367464 46455832
External: 0 Internal: 338437 Local: 0
internal 3939
4553484
Total 138660 205102 24467292
51197812
11-23-2009 03:33 AM
I think its over burdened, BGP is consuming hell lot of Memory and CPU Utlization, i think you will have to filter unneccessory BGP Routes, or best way to handle it open a TICKET with Cisco TAC, and make it sure with Cisco if 7200 NP1 can support this kind of burden, it could be a BUG too, why dont you move to 12.2SR series ??
11-22-2009 08:44 AM
Hello Giuseppe :
Yes . CEF is enabled. I also do not see any traffic being process switched with " show interface switching" output. Here is an example
of one show interface output.
Serial3/0/1:1 is up, line protocol is up
Hardware is CE3
Description: By VPNSC: Internet address is
MTU 1500 bytes, BW 1984 Kbit, DLY 20000 usec,
reliability 255/255, txload 55/255, rxload 33/255
Encapsulation PPP, LCP Open
Open: IPCP, crc 16, loopback not set
Keepalive set (10 sec)
Last input 00:00:13, output 00:00:00, output hang never
Last clearing of "show interface" counters 01:23:18
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1
Queueing strategy: Class-based queueing
Output queue: 2/1000/64/1 (size/max total/threshold/drops)
Conversations 1/46/256 (active/max active/max total)
Reserved Conversations 1/1 (allocated/max allocated)
Available Bandwidth 496 kilobits/sec
5 minute input rate 260000 bits/sec, 522 packets/sec
5 minute output rate 428000 bits/sec, 403 packets/sec
2282137 packets input, 141041214 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1727450 packets output, 217454949 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions no alarm present
Timeslot(s) Used:1-31, subrate: 64Kb/s, transmit delay is 0 flags
12-23-2009 07:52 AM
Try this command:
sh proc cpu sorted 5min | exc 0.00
Paste back the output and it will show the highest used processes sorted by the recent runtime
Here's some docs to help:
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: