Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Community Member

7304 High CPU utilization not cef switched

I have a cisco 7304 thats is currently suffering from high cpu utilization due to traffic being processed switched as opposed to cef switched.

The reason for the cef drops are due to "unsupported features" however I am having trouble pin pointing why the packets being received are outside of the cef supported features. I would appriceate any insight you guys can come up with that could explain the above behaviour

I have cef enabled with valid adjacencies so as far as i can tell all packets should be cef switched. We are running a very basic config and aren’t using any of the features that are traditionally not supported by cef such as: NAT,QoS,AcL,L2VPN,PBR etc.

Hopefully the below output will provide some useful information.

High CPU

show proc cpu sort

CPU utilization for five seconds: 63%/30%; one minute: 67%; five minutes: 65%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

131  11585018281234091803        938 31.91% 33.73% 33.25%   0 IP Input

   6     8305880    723241      11484  1.03%  0.24%  0.22%   0 Check heaps

  59         660       303       2178  0.39%  0.64%  0.18%   2 SSH Process

CEF enabled on interface

show run | i cef

ip cef

!

show cef interface

GigabitEthernet0/0 is up (if_number 4)

  Corresponding hwidb fast_if_number 4

  Corresponding hwidb firstsw->if_number 4

  Internet address is 10.200.0.13/30

  ICMP redirects are never sent

  IP unicast RPF check is disabled

  Output features: Post-Ingress-NetFlow

  IP policy routing is disabled

  BGP based policy accounting on input is disabled

  BGP based policy accounting on output is disabled

  Hardware idb is GigabitEthernet0/0

  Fast switching type 1, interface type 27

IP CEF switching enabled

  IP CEF switching turbo vector

  IP Null turbo vector

  IP prefix lookup IPv4 mtrie generic

  Input fast flags 0x0, Output fast flags 0x0

  ifindex 2(2)

  Slot 0 Slot unit 0 VC -1

  Transmit limit accumulator 0x0 (0x0)

  IP MTU 1500

slo-ce-core1a#show cef interface gi0/1

GigabitEthernet0/1 is up (if_number 5)

  Corresponding hwidb fast_if_number 5

  Corresponding hwidb firstsw->if_number 5

  Internet address is 10.200.0.246/30

  ICMP redirects are never sent

  IP unicast RPF check is disabled

  Output features: Post-Ingress-NetFlow

  IP policy routing is disabled

  BGP based policy accounting on input is disabled

  BGP based policy accounting on output is disabled

  Hardware idb is GigabitEthernet0/1

  Fast switching type 1, interface type 27

  IP CEF switching enabled

  IP CEF switching turbo vector

  IP Null turbo vector

  IP prefix lookup IPv4 mtrie generic

  Input fast flags 0x0, Output fast flags 0x0

  ifindex 3(3)

  Slot 0 Slot unit 1 VC -1

  Transmit limit accumulator 0x0 (0x0)

  IP MTU 1500

slo-ce-core1a#show cef interface gi4/1/0

GigabitEthernet4/1/0 is up (if_number 10)

  Corresponding hwidb fast_if_number 10

  Corresponding hwidb firstsw->if_number 10

  Internet address is 10.200.0.253/30

  ICMP redirects are never sent

  IP unicast RPF check is disabled

  Input features: Ingress-NetFlow

  Output features: Post-Ingress-NetFlow

  IP policy routing is disabled

  BGP based policy accounting on input is disabled

  BGP based policy accounting on output is disabled

  Hardware idb is GigabitEthernet4/1/0

  Fast switching type 1, interface type 27

IP CEF switching enabled

  IP CEF switching turbo vector

  IP Null turbo vector

  IP prefix lookup IPv4 mtrie generic

  Input fast flags 0x0, Output fast flags 0x0

  ifindex 8(8)

  Slot 4 Slot unit 8 VC -1

  Transmit limit accumulator 0x0 (0x0)

  IP MTU 1500

!

CEF unsupported drops

show ip cef switching statistics

       Reason                          Drop       Punt  Punt2Host

RP LES No route                         558          0          8

RP LES Packet destined for us             0   14193479          0

RP LES TTL expired                        0          0          3

RP LES Bad IP packet length               8          0          0

RP LES Unclassified reason                0 12015125162          0

RP LES Total                            566 12029318641         11

All    Total                            566 12029318641         11

IP Traffic

show ip traffic

IP statistics:

  Rcvd:  3439917492 total, 14384000 local destination

         3 format errors, 0 checksum errors, 0 bad hop count

         0 unknown protocol, 0 not a gateway

         0 security failures, 0 bad options, 0 with options

  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route

         0 timestamp, 0 extended security, 0 record route

         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

         0 other

  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble

         0 fragmented, 0 couldn't fragment

  Bcast: 178931 received, 0 sent

  Mcast: 0 received, 0 sent

  Sent:  11638103 generated, 3411999017 forwarded

  Drop:  17259721 encapsulation failed, 0 unresolved, 0 no adjacency

         8 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr

         0 options denied, 0 source IP address zero

Platform info

show ver

Cisco IOS Software, 7300 Software (C7300-A3JK91S-M), Version 12.2(33)SB10, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2011 by Cisco Systems, Inc.

Compiled Wed 06-Apr-11 14:36 by prod_rel_team

ROM: System Bootstrap, Version 12.1(12r)EX1, RELEASE SOFTWARE (fc1)

BOOTLDR: Cisco IOS Software, 7300 Software (C7300-BOOT-M), Version 12.2(33)SB10, RELEASE SOFTWARE (fc1)

slo-ce-core1a uptime is 5 weeks, 6 days, 13 hours, 29 minutes

Uptime for this control processor is 5 weeks, 6 days, 13 hours, 29 minutes

System returned to ROM by error - an Error Interrupt, PC 0x4098D0C0 at 00:09:46 BST Tue Jul 23 2013

System restarted at 00:44:14 BST Tue Jul 23 2013

System image file is "disk0:/c7300-a3jk91s-mz.122-33.SB10.bin"

Last reload type: Normal Reload

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco 7300 (NSE100) processor (revision E) with 491520K/32768K bytes of memory.

Processor board ID SMQ1136NCK3

R7000 CPU at 350Mhz, Implementation 0x27, Rev 3.3, 256KB L2, 1024KB L3 Cache

4 slot midplane, Version 69.48

Last reset from software reset or reload

PXF processor tmc0 'system:pxf/ucode1' is running ( v1.0 ).

PXF processor tmc1 'system:pxf/ucode1' is running ( v1.0 ).

5 FastEthernet interfaces

4 Gigabit Ethernet interfaces

509K bytes of non-volatile configuration memory.

125440K bytes of ATA compact flash in bootdisk (Sector size 512 bytes).

250880K bytes of ATA compact flash in disk0 (Sector size 512 bytes).

Configuration register is 0x2102

slo-ce-core1a#

7 REPLIES
Cisco Employee

7304 High CPU utilization not cef switched

Hi ,

Curious to know whether are we using any GRE tunnel (MPLS to Ip)  in this router ?

If yes please get the output of

- show pxf accounting punt (Run multipl times)
- show mpls forwarding table details


Regards

Partha

Community Member

7304 High CPU utilization not cef switched

Hi Partha,

Thanks for your response.

No we are not running any GRE tunnels or MPSL on this router. its a very basic config with 2 bgp neighbours, 2 additional layer 3 interfaces and a few static routes.

Below is the output from "show pxf acc punt" I can see there is an increment on the "Null adjacency punt"  however when i run the "show cef not-cef-switched" command i can see there are no "no_adj" it seems there is some contradiction between the two show commands ?

PXF punt

show pxf accounting punt

PXF Punt Reasons:

  Non IP Punt              : 727

  RP IPC PAK Punt          : 77212652

  Broadcasts/Promiscuous Multicasts: 67912

  Local Address Punt       : 14214751

  Null Adjacency Punt      : 12044971484

  Unsupported iFeature Punt: 566

show pxf accounting punt

PXF Punt Reasons:

  Non IP Punt              : 727

  RP IPC PAK Punt          : 77217269

  Broadcasts/Promiscuous Multicasts: 67915

  Local Address Punt       : 14215104

Null Adjacency Punt      : 12045090424

  Unsupported iFeature Punt: 566

not cef switched

show cef not-cef-switched

% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Packets passed on to next switching layer

Slot  No_adj No_encap Unsupp'ted Redirect  Receive  Options   Access     Frag

RP         0       0  3455158711        0 14215321        0        0        0

Cisco Employee

7304 High CPU utilization not cef switched

Hi,

what is the status of arp table ?

Also ,

Can you do: clear counters and capture the 'sh int stat'

30 seconds apart for 3 interations.

Just a single snapshot of 'sh int stat' isn't very helpful.

Are the counters going up continuously?

Also check: sh ip route | incl 00:00 and see if you are

seeing route churn.

Regards

Partha

Community Member

7304 High CPU utilization not cef switched

Hi Partha,

The arp table and default route are both stable.

After clearing the counters I can still see traffic being dropped by CEF due to an unclassifyed reason.

#show ip traffic

IP statistics:

  Rcvd:  17843637 total, 14911948 local destination

         3 format errors, 0 checksum errors, 0 bad hop count

         0 unknown protocol, 0 not a gateway

         0 security failures, 0 bad options, 0 with options

  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route

         0 timestamp, 0 extended security, 0 record route

         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

         0 other

  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble

         0 fragmented, 0 couldn't fragment

  Bcast: 191149 received, 0 sent

  Mcast: 0 received, 0 sent

  Sent:  12182883 generated, 4284497968 forwarded

  Drop:  17260643 encapsulation failed, 0 unresolved, 0 no adjacency

         8 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr

         0 options denied, 0 source IP address zero

#show ip traffic

IP statistics:

  Rcvd:  19058786 total, 14913038 local destination

         3 format errors, 0 checksum errors, 0 bad hop count

         0 unknown protocol, 0 not a gateway

         0 security failures, 0 bad options, 0 with options

  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route

         0 timestamp, 0 extended security, 0 record route

         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

         0 other

  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble

         0 fragmented, 0 couldn't fragment

  Bcast: 191174 received, 0 sent

  Mcast: 0 received, 0 sent

  Sent:  12184259 generated, 4285712493 forwarded

  Drop:  17260647 encapsulation failed, 0 unresolved, 0 no adjacency

         8 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr

         0 options denied, 0 source IP address zero

Community Member

7304 High CPU utilization not cef switched

Due to the fact that I am unable to find a reason why the traffic is not being cef switched I plan on changing the hardware. Im going to load up the same config on a new 7304 and see what the results produce.

Super Bronze

Re: 7304 High CPU utilization not cef switched

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

It's been years since I worked with a 7304 with an NSE-100.

Ideally, most traffic should be forwarded by the PXF.  I recall PXF was extra sensitive to the "kind" of traffic.  Later IOS releases expanded the "kind" of traffic supported by the PXF.  I haven't looked, but is the 12.2(33)SB10 current?  If not, you might consider an IOS upgrade.  Also with the PXF, often with particular IOS upgrades you were supposed to insure module firmware was current with that IOS version.  Is yours?

When the PXF can't handle the traffic, then the "normal" CPU, basically a "NPE-400", forwards the traffic.  Don't recall whether traffic not forwarded by the PXF shows as not CEF forwarded, or whether the non-PXF traffic is CEF forwarded or not based on its characteristics.

The reason I'm mentioning this, is because you might want to troubleshoot for PXF not forwarding, vs. CEF not forwarding first, and then perhaps non-PXF CEF vs. process switched.

Community Member

Re: 7304 High CPU utilization not cef switched

Hi Joseph,

Many thanks for your post I appreciate your time.

Firstly we are running a basic config with only 2 bgp connections and a few static routes. I have cross checked our config with the non compatible PXF features such as NAT, QoS, VPN etc and we dont have any of those features running on this device.

Regarding the IOS upgrade, it is something I would like to do for sure however we have another 7304 running a very similar config and the same IOS, FPGA and PXF processor version and im not seeing any issues with the CEF forwarding there.

One thing I don't understand entirely is the amount of drops in "null adjacency punt" from the show PXF

accounting punt command as we have a fully populated CEF table

Good DeviceBad Device
show pxf accounting puntshow pxf accounting punt
PXF Punt Reasons:PXF Punt Reasons:
  Non IP Punt             :   320110 Non IP Punt             : 222
  IPv4 Options Punt       :   118107 RP IPC PAK Punt         : 21023815
  RP IPC PAK Punt         :   680342883 Broadcasts/Promiscuous   Multicasts: 20837
  Broadcasts/Promiscuous Multicasts: 42175 Local Address Punt       : 3295324
  Local Address Punt       :   8116540 Null Adjacency Punt     : 5649246490
  Null Adjacency Punt     :

We already have a replacement device configured and ready to go. My concern is that I replace the device and then everything works ok. Although this would resolve the issue it wouldn't satisfy my curiosity =p

713
Views
0
Helpful
7
Replies
CreatePlease to create content