7304 High CPU utilization not cef switched

Aaron Huggins · ‎09-02-2013

I have a cisco 7304 thats is currently suffering from high cpu utilization due to traffic being processed switched as opposed to cef switched.

The reason for the cef drops are due to "unsupported features" however I am having trouble pin pointing why the packets being received are outside of the cef supported features. I would appriceate any insight you guys can come up with that could explain the above behaviour

I have cef enabled with valid adjacencies so as far as i can tell all packets should be cef switched. We are running a very basic config and aren’t using any of the features that are traditionally not supported by cef such as: NAT,QoS,AcL,L2VPN,PBR etc.

Hopefully the below output will provide some useful information.

High CPU

show proc cpu sort

CPU utilization for five seconds: 63%/30%; one minute: 67%; five minutes: 65%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

131 11585018281234091803 938 31.91% 33.73% 33.25% 0 IP Input

6 8305880 723241 11484 1.03% 0.24% 0.22% 0 Check heaps

59 660 303 2178 0.39% 0.64% 0.18% 2 SSH Process

CEF enabled on interface

show run | i cef

ip cef

!

show cef interface

GigabitEthernet0/0 is up (if_number 4)

Corresponding hwidb fast_if_number 4

Corresponding hwidb firstsw->if_number 4

Internet address is 10.200.0.13/30

ICMP redirects are never sent

IP unicast RPF check is disabled

Output features: Post-Ingress-NetFlow

IP policy routing is disabled

BGP based policy accounting on input is disabled

BGP based policy accounting on output is disabled

Hardware idb is GigabitEthernet0/0

Fast switching type 1, interface type 27

IP CEF switching enabled

IP CEF switching turbo vector

IP Null turbo vector

IP prefix lookup IPv4 mtrie generic

Input fast flags 0x0, Output fast flags 0x0

ifindex 2(2)

Slot 0 Slot unit 0 VC -1

Transmit limit accumulator 0x0 (0x0)

IP MTU 1500

slo-ce-core1a#show cef interface gi0/1

GigabitEthernet0/1 is up (if_number 5)

Corresponding hwidb fast_if_number 5

Corresponding hwidb firstsw->if_number 5

Internet address is 10.200.0.246/30

ICMP redirects are never sent

IP unicast RPF check is disabled

Output features: Post-Ingress-NetFlow

IP policy routing is disabled

BGP based policy accounting on input is disabled

BGP based policy accounting on output is disabled

Hardware idb is GigabitEthernet0/1

Fast switching type 1, interface type 27

IP CEF switching enabled

IP CEF switching turbo vector

IP Null turbo vector

IP prefix lookup IPv4 mtrie generic

Input fast flags 0x0, Output fast flags 0x0

ifindex 3(3)

Slot 0 Slot unit 1 VC -1

Transmit limit accumulator 0x0 (0x0)

IP MTU 1500

slo-ce-core1a#show cef interface gi4/1/0

GigabitEthernet4/1/0 is up (if_number 10)

Corresponding hwidb fast_if_number 10

Corresponding hwidb firstsw->if_number 10

Internet address is 10.200.0.253/30

ICMP redirects are never sent

IP unicast RPF check is disabled

Input features: Ingress-NetFlow

Output features: Post-Ingress-NetFlow

IP policy routing is disabled

BGP based policy accounting on input is disabled

BGP based policy accounting on output is disabled

Hardware idb is GigabitEthernet4/1/0

Fast switching type 1, interface type 27

IP CEF switching enabled

IP CEF switching turbo vector

IP Null turbo vector

IP prefix lookup IPv4 mtrie generic

Input fast flags 0x0, Output fast flags 0x0

ifindex 8(8)

Slot 4 Slot unit 8 VC -1

Transmit limit accumulator 0x0 (0x0)

IP MTU 1500

!

CEF unsupported drops

show ip cef switching statistics

Reason Drop Punt Punt2Host

RP LES No route 558 0 8

RP LES Packet destined for us 0 14193479 0

RP LES TTL expired 0 0 3

RP LES Bad IP packet length 8 0 0

RP LES Unclassified reason 0 12015125162 0

RP LES Total 566 12029318641 11

All Total 566 12029318641 11

IP Traffic

show ip traffic

IP statistics:

Rcvd: 3439917492 total, 14384000 local destination

3 format errors, 0 checksum errors, 0 bad hop count

0 unknown protocol, 0 not a gateway

0 security failures, 0 bad options, 0 with options

Opts: 0 end, 0 nop, 0 basic security, 0 loose source route

0 timestamp, 0 extended security, 0 record route

0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

0 other

Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble

0 fragmented, 0 couldn't fragment

Bcast: 178931 received, 0 sent

Mcast: 0 received, 0 sent

Sent: 11638103 generated, 3411999017 forwarded

Drop: 17259721 encapsulation failed, 0 unresolved, 0 no adjacency

8 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr

0 options denied, 0 source IP address zero

Platform info

show ver

Cisco IOS Software, 7300 Software (C7300-A3JK91S-M), Version 12.2(33)SB10, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 06-Apr-11 14:36 by prod_rel_team

ROM: System Bootstrap, Version 12.1(12r)EX1, RELEASE SOFTWARE (fc1)

BOOTLDR: Cisco IOS Software, 7300 Software (C7300-BOOT-M), Version 12.2(33)SB10, RELEASE SOFTWARE (fc1)

slo-ce-core1a uptime is 5 weeks, 6 days, 13 hours, 29 minutes

Uptime for this control processor is 5 weeks, 6 days, 13 hours, 29 minutes

System returned to ROM by error - an Error Interrupt, PC 0x4098D0C0 at 00:09:46 BST Tue Jul 23 2013

System restarted at 00:44:14 BST Tue Jul 23 2013

System image file is "disk0:/c7300-a3jk91s-mz.122-33.SB10.bin"

Last reload type: Normal Reload

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco 7300 (NSE100) processor (revision E) with 491520K/32768K bytes of memory.

Processor board ID SMQ1136NCK3

R7000 CPU at 350Mhz, Implementation 0x27, Rev 3.3, 256KB L2, 1024KB L3 Cache

4 slot midplane, Version 69.48

Last reset from software reset or reload

PXF processor tmc0 'system:pxf/ucode1' is running ( v1.0 ).

PXF processor tmc1 'system:pxf/ucode1' is running ( v1.0 ).

5 FastEthernet interfaces

4 Gigabit Ethernet interfaces

509K bytes of non-volatile configuration memory.

125440K bytes of ATA compact flash in bootdisk (Sector size 512 bytes).

250880K bytes of ATA compact flash in disk0 (Sector size 512 bytes).

Configuration register is 0x2102

slo-ce-core1a#

Partha Dasgupta · ‎09-02-2013

Hi ,

Curious to know whether are we using any GRE tunnel (MPLS to Ip) in this router ?

If yes please get the output of

- show pxf accounting punt (Run multipl times)
- show mpls forwarding table details


Regards

Partha

Aaron Huggins · ‎09-02-2013

Hi Partha,

Thanks for your response.

No we are not running any GRE tunnels or MPSL on this router. its a very basic config with 2 bgp neighbours, 2 additional layer 3 interfaces and a few static routes.

Below is the output from "show pxf acc punt" I can see there is an increment on the "Null adjacency punt" however when i run the "show cef not-cef-switched" command i can see there are no "no_adj" it seems there is some contradiction between the two show commands ?

PXF punt

show pxf accounting punt

PXF Punt Reasons:

Non IP Punt : 727

RP IPC PAK Punt : 77212652

Broadcasts/Promiscuous Multicasts: 67912

Local Address Punt : 14214751

Null Adjacency Punt : 12044971484

Unsupported iFeature Punt: 566

show pxf accounting punt

PXF Punt Reasons:

Non IP Punt : 727

RP IPC PAK Punt : 77217269

Broadcasts/Promiscuous Multicasts: 67915

Local Address Punt : 14215104

Null Adjacency Punt : 12045090424

Unsupported iFeature Punt: 566

not cef switched

show cef not-cef-switched

% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Packets passed on to next switching layer

Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag

RP 0 0 3455158711 0 14215321 0 0 0

Partha Dasgupta · ‎09-02-2013

Hi,

what is the status of arp table ?

Also ,

Can you do: clear counters and capture the 'sh int stat'

30 seconds apart for 3 interations.

Just a single snapshot of 'sh int stat' isn't very helpful.

Are the counters going up continuously?

Also check: sh ip route | incl 00:00 and see if you are

seeing route churn.

Regards

Partha

Aaron Huggins · ‎09-04-2013

Hi Partha,

The arp table and default route are both stable.

After clearing the counters I can still see traffic being dropped by CEF due to an unclassifyed reason.

#show ip traffic

IP statistics:

Rcvd: 17843637 total, 14911948 local destination

3 format errors, 0 checksum errors, 0 bad hop count

0 unknown protocol, 0 not a gateway

0 security failures, 0 bad options, 0 with options

Opts: 0 end, 0 nop, 0 basic security, 0 loose source route

0 timestamp, 0 extended security, 0 record route

0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

0 other

Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble

0 fragmented, 0 couldn't fragment

Bcast: 191149 received, 0 sent

Mcast: 0 received, 0 sent

Sent: 12182883 generated, 4284497968 forwarded

Drop: 17260643 encapsulation failed, 0 unresolved, 0 no adjacency

8 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr

0 options denied, 0 source IP address zero

#show ip traffic

IP statistics:

Rcvd: 19058786 total, 14913038 local destination

3 format errors, 0 checksum errors, 0 bad hop count

0 unknown protocol, 0 not a gateway

0 security failures, 0 bad options, 0 with options

Opts: 0 end, 0 nop, 0 basic security, 0 loose source route

0 timestamp, 0 extended security, 0 record route

0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump

0 other

Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble

0 fragmented, 0 couldn't fragment

Bcast: 191174 received, 0 sent

Mcast: 0 received, 0 sent

Sent: 12184259 generated, 4285712493 forwarded

Drop: 17260647 encapsulation failed, 0 unresolved, 0 no adjacency

8 no route, 0 unicast RPF, 0 forced drop, 0 unsupported-addr

0 options denied, 0 source IP address zero

Aaron Huggins · ‎09-16-2013

Due to the fact that I am unable to find a reason why the traffic is not being cef switched I plan on changing the hardware. Im going to load up the same config on a new 7304 and see what the results produce.

Joseph W. Doherty · ‎09-16-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

It's been years since I worked with a 7304 with an NSE-100.

Ideally, most traffic should be forwarded by the PXF. I recall PXF was extra sensitive to the "kind" of traffic. Later IOS releases expanded the "kind" of traffic supported by the PXF. I haven't looked, but is the 12.2(33)SB10 current? If not, you might consider an IOS upgrade. Also with the PXF, often with particular IOS upgrades you were supposed to insure module firmware was current with that IOS version. Is yours?

When the PXF can't handle the traffic, then the "normal" CPU, basically a "NPE-400", forwards the traffic. Don't recall whether traffic not forwarded by the PXF shows as not CEF forwarded, or whether the non-PXF traffic is CEF forwarded or not based on its characteristics.

The reason I'm mentioning this, is because you might want to troubleshoot for PXF not forwarding, vs. CEF not forwarding first, and then perhaps non-PXF CEF vs. process switched.

Aaron Huggins · ‎09-17-2013

Hi Joseph,

Many thanks for your post I appreciate your time.

Firstly we are running a basic config with only 2 bgp connections and a few static routes. I have cross checked our config with the non compatible PXF features such as NAT, QoS, VPN etc and we dont have any of those features running on this device.

Regarding the IOS upgrade, it is something I would like to do for sure however we have another 7304 running a very similar config and the same IOS, FPGA and PXF processor version and im not seeing any issues with the CEF forwarding there.

One thing I don't understand entirely is the amount of drops in "null adjacency punt" from the show PXF

accounting punt command as we have a fully populated CEF table

Good Device	Bad Device
show pxf accounting punt	show pxf accounting punt
PXF Punt Reasons:	PXF Punt Reasons:
Non IP Punt : 320110	Non IP Punt : 222
IPv4 Options Punt : 118107	RP IPC PAK Punt : 21023815
RP IPC PAK Punt : 680342883	Broadcasts/Promiscuous Multicasts: 20837
Broadcasts/Promiscuous Multicasts: 42175	Local Address Punt : 3295324
Local Address Punt : 8116540	Null Adjacency Punt : 5649246490
Null Adjacency Punt :

We already have a replacement device configured and ready to go. My concern is that I replace the device and then everything works ok. Although this would resolve the issue it wouldn't satisfy my curiosity =p