Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

7206 Npe G2 high cpu problems

Hi,

I have a lot of cisco 7206 vxr deviceses. I have a high cpu problem.

I have a 7206 vxr g2. I used to use ebgp for three upstream carrier. I used to bgp full route table method. I have about 800 Mbps active traffics and behing the router about 1600 active customers.

Could you advice me how can I trouble shoot this high cpu problem?

#sh int gigabitEthernet 0/1

GigabitEthernet0/1 is up, line protocol is up

  Hardware is MV64460 Internal MAC, address is 000c.cf1d.d01b (bia 000c.cf1d.d01b)

  Description: ***** GW Interface *****

  Internet address is 213.144.96.1/24

  MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 31/255, rxload 45/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  Keepalive set (10 sec)

  Full Duplex, 1000Mbps, media type is RJ45

  output flow-control is XON, input flow-control is unsupported

  ARP type: ARPA, ARP Timeout 03:20:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters 4w4d

  Input queue: 0/4096/4860/1347216 (size/max/drops/flushes); Total output drops: 13800835

  Queueing strategy: fifo

  Output queue: 0/4096 (size/max)

  30 second input rate 177466000 bits/sec, 35236 packets/sec

  30 second output rate 125457000 bits/sec, 33418 packets/sec

     2016938526 packets input, 1849362236 bytes, 4794 no buffer

     Received 50788313 broadcasts (15690571 IP multicasts)

     0 runts, 0 giants, 0 throttles

     51133 input errors, 44 CRC, 0 frame, 0 overrun, 51089 ignored

     0 watchdog, 27438132 multicast, 0 pause input

     1827169002 packets output, 1537189058 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

#

#sh int gigabitEthernet 0/2

GigabitEthernet0/2 is up, line protocol is up

  Hardware is MV64460 Internal MAC, address is 000c.cf1d.d01a (bia 000c.cf1d.d01a)

  Description: ***** Internet Access Port *****

  MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 95/255, rxload 84/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  Keepalive set (10 sec)

  Full Duplex, 1000Mbps, media type is RJ45

  output flow-control is XON, input flow-control is unsupported

  ARP type: ARPA, ARP Timeout 03:20:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters 4w4d

  Input queue: 0/4096/15826/2230509 (size/max/drops/flushes); Total output drops: 17298

  Queueing strategy: fifo

  Output queue: 0/4096 (size/max)

  30 second input rate 332999000 bits/sec, 63563 packets/sec

  30 second output rate 373472000 bits/sec, 71761 packets/sec

     1133386686 packets input, 849294076 bytes, 17917 no buffer

     Received 37718109 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     421 input errors, 0 CRC, 0 frame, 0 overrun, 421 ignored

     0 watchdog, 2554 multicast, 0 pause input

     2675409271 packets output, 2343996450 bytes, 0 underruns

     4 output errors, 0 collisions, 4 interface resets

     0 babbles, 0 late collision, 0 deferred

     4 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

#

#sh int gigabitEthernet 0/3

GigabitEthernet0/3 is up, line protocol is up

  Hardware is MV64460 Internal MAC, address is 000c.cf1d.d019 (bia 000c.cf1d.d019)

  Description: ***** Metro Ethernet *****

  MTU 1700 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 64/255, rxload 59/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  Keepalive set (10 sec)

  Full Duplex, 1000Mbps, media type is RJ45

  output flow-control is XON, input flow-control is unsupported

  ARP type: ARPA, ARP Timeout 03:20:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters 4w4d

  Input queue: 0/4096/99466/2578258 (size/max/drops/flushes); Total output drops: 45184295

  Queueing strategy: fifo

  Output queue: 0/4096 (size/max)

  30 second input rate 234300000 bits/sec, 51018 packets/sec

  30 second output rate 251936000 bits/sec, 44635 packets/sec

     1015412816 packets input, 1352601046 bytes, 41095 no buffer

     Received 157083858 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     296819 input errors, 413 CRC, 0 frame, 0 overrun, 296406 ignored

     0 watchdog, 157681185 multicast, 0 pause input

     2426528064 packets output, 508387291 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

#

#sh ver

Cisco IOS Software, 7200 Software (C7200P-SPSERVICESK9-M), Version 12.2(33)SRD2, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2009 by Cisco Systems, Inc.

Compiled Wed 20-May-09 21:32 by prod_rel_team

ROM: System Bootstrap, Version 12.4(4r)XD5, RELEASE SOFTWARE (fc1)

7206_VXR_G2 uptime is 26 weeks, 5 days, 19 hours, 17 minutes

System returned to ROM by power-on

System restarted at 15:58:49 GMT+2 Tue Aug 30 2011

System image file is "disk2:/c7200p-spservicesk9-mz.122-33.SRD2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

Cisco 7206VXR (NPE-G2) processor (revision A) with 917504K/65536K bytes of memory.

Processor board ID 29813402

MPC7448 CPU at 1666Mhz, Implementation 0, Rev 2.2

6 slot VXR midplane, Version 2.7

Last reset from power-on

PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.

Current configuration on bus mb1 has a total of 90 bandwidth points.

This configuration is within the PCI bus capacity and is supported.

PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.

Current configuration on bus mb2 has a total of 0 bandwidth points.

This configuration is within the PCI bus capacity and is supported.

Please refer to the following document "Cisco 7200 Series Port Adaptor

Hardware Configuration Guidelines" on Cisco.com <http://www.cisco.com>

for c7200 bandwidth points oversubscription and usage guidelines.

1 FastEthernet interface

3 Gigabit Ethernet interfaces

3 Serial interfaces

8 Channelized E1/PRI ports

1 Channelized E3 port

2045K bytes of NVRAM.

250880K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

#

#sh processes cpu sorted

CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

246   371404700   1324881     280347  7.91%  2.23%  1.71%   0 BGP Scanner

249    834493961271325781         65  0.79%  0.94%  0.96%   0 EIGRP-IPv4: PDM

  85   1989393561828373765        108  0.79%  0.84%  0.80%   0 IP Input

245    35318844 989199499         35  0.47%  0.59%  0.54%   0 EIGRP-IPv4: PDM

125    85488388 108901503        785  0.47%  0.55%  0.56%   0 IP RIB Update

133    90593692  95054980        953  0.47%  0.56%  0.56%   0 static

241    30375216 679746761         44  0.31%  0.32%  0.32%   0 EIGRP-IPv4: PDM

153   254854208 212933212       1196  0.31%  0.61%  0.68%   0 BGP Router

  24    42637432 339131952        125  0.23%  0.20%  0.22%   0 ARP Input

239    30037548 233726388        128  0.23%  0.11%  0.11%   0 EIGRP-IPv4: PDM

   3         164       625        262  0.15%  0.04%  0.00%   2 SSH Process

214     9704812  36008323        269  0.15%  0.07%  0.07%   0 SNMP ENGINE

  48     2342552    292826       7999  0.15%  0.01%  0.00%   0 Per-minute Jobs

  12       19224  16121638          1  0.07%  0.00%  0.00%   0 IPC Periodic Tim

128      5841603820724583          0  0.07%  0.08%  0.07%   0 HQF Shaper Backg

261    12606892 460359880         27  0.07%  0.07%  0.07%   0 EIGRP-IPv4: HELL

221      410440 488309363          0  0.07%  0.04%  0.05%   0 PPP manager

236     10576081928546169          0  0.07%  0.09%  0.07%   0 IP SLAs Event Pr

129      4362563820729036          0  0.07%  0.07%  0.07%   0 HQF Input Shaper

250     7143556 346717390         20  0.07%  0.05%  0.06%   0 EIGRP-IPv4: HELL

117     1054272  22516667         46  0.07%  0.06%  0.07%   0 CEF: IPv4 proces

247      838676  42421526         19  0.07%  0.00%  0.00%   0 EIGRP-IPv4: HELL

101     1316444  65438354         20  0.07%  0.00%  0.00%   0 Spanning Tree

  89    14028036 117415925        119  0.07%  0.06%  0.07%   0 ADJ resolve proc

222      357576 488309506          0  0.07%  0.02%  0.02%   0 PPP Events

212     6874068  69959410         98  0.07%  0.05%  0.06%   0 IP SNMP

  26           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer

  27           0         1          0  0.00%  0.00%  0.00%   0 ATM ASYNC PROC

  25      285844  16814593         16  0.00%  0.00%  0.00%   0 ARP Background

  30           0         4          0  0.00%  0.00%  0.00%   0 Entity MIB API

  28           0         5          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT

  29           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager

  33         116     16995          6  0.00%  0.00%  0.00%   0 Serial Backgroun

  31        5372    160564         33  0.00%  0.00%  0.00%   0 EEM ED Syslog

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  35           0         1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa

  36           0        19          0  0.00%  0.00%  0.00%   0 DDR Timers

  37           0         2          0  0.00%  0.00%  0.00%   0 SMART

  32         784         5     156800  0.00%  0.00%  0.00%   0 PrstVbl

  39       17824  16121633          1  0.00%  0.00%  0.00%   0 GraphIt

  40           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect

  15           0         1          0  0.00%  0.00%  0.00%   0 IPC Seat Manager

  42           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd

  43     2817912   4977553        566  0.00%  0.03%  0.01%   0 Net Background

  23        3352   3232113          1  0.00%  0.00%  0.00%   0 ALARM_TRIGGER_SC

  22           0         1          0  0.00%  0.00%  0.00%   0 Exception contro

  34           0         1          0  0.00%  0.00%  0.00%   0 RO Notify Timers

  38           0         1          0  0.00%  0.00%  0.00%   0 RFS server proce

  14       12304  16121640          0  0.00%  0.00%  0.00%   0 IPC Deferred Por

  49          20        84        238  0.00%  0.00%  0.00%   0 IF-MGR control p

  50          20       393         50  0.00%  0.00%  0.00%   0 IF-MGR event pro

  51           0         1          0  0.00%  0.00%  0.00%   0 Inode Table Dest

  52           0         2          0  0.00%  0.00%  0.00%   0 INTR MGR PROCESS

  53        9712   5276901          1  0.00%  0.00%  0.00%   0 Transport Port A

  54       51772   4807188         10  0.00%  0.00%  0.00%   0 HC Counter Timer

  41           0         2          0  0.00%  0.00%  0.00%   0 Dialer event

  56           0         1          0  0.00%  0.00%  0.00%   0 POS APS Event Pr

  57           0         1          0  0.00%  0.00%  0.00%   0 SONET alarm time

  58           0         2          0  0.00%  0.00%  0.00%   0 Hawkeye Backgrou

  59           0         1          0  0.00%  0.00%  0.00%   0 CSP Timer

  60           0         2          0  0.00%  0.00%  0.00%   0 FPD Management P

  44           0         2          0  0.00%  0.00%  0.00%   0 IDB Work

  62      159304    919190        173  0.00%  0.00%  0.00%   0 Collection proce

  63           0         2          0  0.00%  0.00%  0.00%   0 VNM DSPRM MAIN

  64           0         2          0  0.00%  0.00%  0.00%   0 Flash MIB Update

  65           8         6       1333  0.00%  0.00%  0.00%   0 Flash Card Oir

  66           0         1          0  0.00%  0.00%  0.00%   0 CES Line Conditi

  45        4968    342666         14  0.00%  0.00%  0.00%   0 Logger

  68           0         1          0  0.00%  0.00%  0.00%   0 Async write proc

  46       27764  16121595          1  0.00%  0.00%  0.00%   0 TTY Background

  70           0         1          0  0.00%  0.00%  0.00%   0 Async write proc

#

#sh processes memory sorted

Processor Pool Total:  850287212 Used:  508707208 Free:  341580004

      I/O Pool Total:   67108864 Used:    6691312 Free:   60417552

Transient Pool Total:   16777216 Used:     122356 Free:   16654860

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process

153   0 2167646800 3258059216  369532500      52448          0 BGP Router

125   0 2566262772  823460356   78550656          0          0 IP RIB Update

   0   0   51236456    7371184   39439020          0          0 *Init*

   0   0 3116389520 2968229116    7423404    3661344      17420 *Dead*

   5   0  358534536  332228156    4457884  133743676  135805988 Pool Manager

239   0 2924544240 2945053956    4331500      19108     719136 EIGRP-IPv4: PDM

   0   0          0          0    2033364          0          0 *MallocLite*

249   0 1396961512 1353257724    1081536     146924    1013924 EIGRP-IPv4: PDM

  96   0     796772      47276     710012          0          0 PPP SIP

  84   0     535788      25248     526860          0          0 IP ARP Adjacency

207   0     446176          0     507300          0          0 CCPROXY_CT

105   0     580832      88200     500112          0          0 SSS Manager

222   0     499536      13776     494768        780          0 PPP Events

124   0     449952     107988     342492          0          0 IP Background

100   0     311152        188     324088          0          0 SSM connection m

133   0     820972     615892     323388          0          0 static

  31   0     279992          0     289504     113400          0 EEM ED Syslog

  89   0 1027472104 1153134984     265224    6600880   30557704 ADJ resolve proc

241   0 3208006648 3210118876     224048      48436     392504 EIGRP-IPv4: PDM

  62   0    3442292    1015964     210036          0          0 Collection proce

   1   0   37852324   37808240     182816          0          0 Chunk Manager

188   0     142428       9180     148940          0          0 EEM Server

  85   0  731131288  534554412     119264   44456588   10697448 IP Input

164   0      65848          0      90972          0          0 QOS_MODULE_MAIN

117   0      90752         76      90752          0          0 CEF: IPv4 proces

  43   0     436500        248      83176      65260          0 Net Background

256   0     285140      11248      82884          0          0 Tag Control

115   0      65752          0      78876          0          0 CEF background p

167   0     111976       3912      67600          0          0 Crypto ACL

  82   0      49404          0      62528          0          0 IPAM Manager

  24   0 3298366036 3366734816      62384          0   13954704 ARP Input

#

Message was edited by: Umit AYDINLI because i added sh tech output.

Message was edited by: Umit AYDINLI

Everyone's tags (6)
2 ACCEPTED SOLUTIONS

Accepted Solutions
Hall of Fame Super Bronze

7206 Npe G2 high cpu problems

Your interfaces are reporting a high number of input and output drops for the 4week period.

Can you make sure the duplex/speed are matching at either end and also verify cabling?

Packet drops in the router can create high CPU conditions.

What other services you have enabled in the router?

Can you post your sanitize config?

Regards,

Edison

Cisco Employee

7206 Npe G2 high cpu problems

Hi Umit

With the information you provided, we cannot arrive at a conclusion. The CPU utilisation will depend on several factors like-

1)features configured

2)Switching path taken by packets

3)Broadcasts and multicast packets received

4)Packets direct to the router

From the outputs, I see that you have 62 % interrupts and remaining CPU is accounted by various process which is pretty normal.

CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%

I notice that you have significant traffic flowing through the router. I would say you are hitting the hardware limitation with the current flow of traffic.

As per documents NPE-G2 can handle up to 1024 Mbps traffic, but never expect more than 800 Mbps with heavy configuration.

As you add ACL's, encryption, QOS, etc - performance will decline significantly from the given numbers, unless it is a hardware-assisted platform, such as the ASR 1000, 7600 or 12000

You can compare the router performance in below link-

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

The only logical thing we can do now is to perform a CPU profiling and identify the functions that are interrupting the CPU. Once the function is identified, we can try tweaking/ optimizing  the associated configuration . However in your case I doubt there is anything more we can do to reduce CPU other than redirecting some traffic.

Regards

Vishnu

4 REPLIES
Hall of Fame Super Bronze

7206 Npe G2 high cpu problems

Your interfaces are reporting a high number of input and output drops for the 4week period.

Can you make sure the duplex/speed are matching at either end and also verify cabling?

Packet drops in the router can create high CPU conditions.

What other services you have enabled in the router?

Can you post your sanitize config?

Regards,

Edison

Super Bronze

7206 Npe G2 high cpu problems

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

From what you've posted, you might just be occasionally offering too much load to this router.  A G2 can struggle with two gig links.

Super Bronze

Re: 7206 Npe G2 high cpu problems

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

A G2 can struggle with two gig links.

BTW, reason for this statement, gig Ethernet can require up to 1.488 Mpps.  So two gig links could require up to about 3 Mpps, and your three gig links could require up to about 4.5 Mpps.  Since a -G2 is rated at 2 Mpps, see the reason why it can struggle?

Even the "fastest" 7200 "engine", the discontinued NSE-1, rated at 3.5 Mpps (PXF), might struggle with your three gig links.  A 5 Mpps ASR would probably be the most suitable choice for your current usage.

Looking at your stats, there is some "low hanging fruit", i.e. some configuration options that might help.  These include: insuring your BGP peering is using MTU, not just 576; insuring buffer usage is just "hits"; enabling compiled ACLs (if supported and assuming you're using more than a few ACLs); tuning SPD; and perhaps reducing your TCP window size to only accept 2 to 4 packets (to throttle BGP updates).

Cisco Employee

7206 Npe G2 high cpu problems

Hi Umit

With the information you provided, we cannot arrive at a conclusion. The CPU utilisation will depend on several factors like-

1)features configured

2)Switching path taken by packets

3)Broadcasts and multicast packets received

4)Packets direct to the router

From the outputs, I see that you have 62 % interrupts and remaining CPU is accounted by various process which is pretty normal.

CPU utilization for five seconds: 75%/62%; one minute: 71%; five minutes: 69%

I notice that you have significant traffic flowing through the router. I would say you are hitting the hardware limitation with the current flow of traffic.

As per documents NPE-G2 can handle up to 1024 Mbps traffic, but never expect more than 800 Mbps with heavy configuration.

As you add ACL's, encryption, QOS, etc - performance will decline significantly from the given numbers, unless it is a hardware-assisted platform, such as the ASR 1000, 7600 or 12000

You can compare the router performance in below link-

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

The only logical thing we can do now is to perform a CPU profiling and identify the functions that are interrupting the CPU. Once the function is identified, we can try tweaking/ optimizing  the associated configuration . However in your case I doubt there is anything more we can do to reduce CPU other than redirecting some traffic.

Regards

Vishnu

2995
Views
0
Helpful
4
Replies