PHXRTR02#sho cef not-cef-switched
Load for five secs: 88%/58%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:49:50.700 MST Tue Jul 6 2010

% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Packets passed on to next switching layer
Slot  No_adj No_encap Unsupp'ted Redirect  Receive  Options   Access     Frag
RP         0       0   124603860        0     2474        0     5008        0

PHXRTR02#sho ip cef switch stat
Load for five secs: 88%/57%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:50:24.060 MST Tue Jul 6 2010

       Reason                          Drop       Punt  Punt2Host
RP LES Packet destined for us             0       1267       1207
RP LES Encapsulation resource             0       2824          0
RP LES Incomplete adjacency               0          0      10366
RP LES TTL expired                        0          0       2883
RP LES Features                         434          0  124601258
RP LES Total                            434       4091  124615714

All    Total                            434       4091  124615714

PHXRTR02#sho ip cef switch stat feat
Load for five secs: 82%/54%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:50:40.813 MST Tue Jul 6 2010

IPv4 CEF input features:
       Feature                Drop    Consume       Punt  Punt2Host Gave route
       Virtual Fragment         91          0          0          0          0
       Access List               3          0          0       5008          0
       NAT Outside               0          0          0  124602939          0
Total                           94          0          0  124607947          0

IPv4 CEF output features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF post-encap features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
       IPSEC Post-encap        340  141966553          0          0          0
Total                          340  141966553          0          0          0

IPv4 CEF for us features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF punt features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF local features:
       Feature                Drop    Consume       Punt  Punt2Host Gave route
Total                            0          0          0          0          0

PHXRTR02#sho ip traffic
Load for five secs: 89%/59%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:50:52.425 MST Tue Jul 6 2010

IP statistics:
  Rcvd:  2364200412 total, 1190951521 local destination
         0 format errors, 0 checksum errors, 766 bad hop count
         1 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 0 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other
  Frags: 718 reassembled, 0 timeouts, 0 couldn't reassemble
         4970151 fragmented, 9940568 fragments, 530 couldn't fragment
  Bcast: 366 received, 0 sent
  Mcast: 5302283 received, 5267912 sent
  Sent:  38786751 generated, 3812043703 forwarded
  Drop:  780 encapsulation failed, 0 unresolved, 0 no adjacency
         67 no route, 0 unicast RPF, 0 forced drop
         0 options denied
  Drop:  0 packets with source IP address zero
  Drop:  0 packets with internal loop back IP address
         0 physical broadcast

ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 79304 unreachable
        798634 echo, 1044 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 4 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
        271 time exceeded, 0 info replies
  Sent: 0 redirects, 48216 unreachable, 1060 echo, 798634 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 4 timestamp replies
        0 info reply, 760 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
  Rcvd: 133304 total, 9 checksum errors, 6260 no port
  Sent: 166045 total

UDP statistics:
  Rcvd: 12888284 total, 0 checksum errors, 4747 no port
  Sent: 12935831 total, 0 forwarded broadcasts

BGP statistics:
  Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh, 0 unrecognized
  Sent: 0 total, 0 opens, 0 notifications, 0 updates
        0 keepalives, 0 route-refresh

IP-EIGRP statistics:
  Rcvd: 0 total
  Sent: 0 total

PIMv2 statistics: Sent/Received
  Total: 0/0, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  Queue drops: 0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0
  Queue drops: 0

OSPF statistics:
  Rcvd: 5314417 total, 0 checksum errors
        4083340 hello, 933 database desc, 60 link state req
        1228547 link state updates, 1450 link state acks

  Sent: 5288704 total
        4115824 hello, 754 database desc, 43 link state req
        653251 link state updates, 518832 link state acks

ARP statistics:
  Rcvd: 190483 requests, 119 replies, 0 reverse, 0 other
  Sent: 1591 requests, 2726 replies (0 proxy), 0 reverse
  Drop due to input queue full: 0

Sorry for the multiple posts, in answer to your last question, here is the additional output for cpu utilization:

PHXRTR02#sho proc cpu sort
Load for five secs: 85%/56%; one minute: 85%; five minutes: 85%
Time source is NTP, 15:53:03.658 MST Tue Jul 6 2010

CPU utilization for five seconds: 85%/56%; one minute: 85%; five minutes: 85%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
119  1853535196  1219057691       1520 27.57% 27.47% 27.63%   0 IP Input
184    16137016   411358600         39  0.55%  0.53%  0.52%   0 HQF Shaper Backg
234        4620        2097       2203  0.39%  0.06%  0.07% 514 SSH Process
  37      251044     8403647         29  0.07%  0.01%  0.00%   0 Net Background
  41     1567876    19554786         80  0.07%  0.08%  0.07%   0 Per-Second Jobs
185     1129920   194449152          5  0.07%  0.04%  0.05%   0 RBSCP Background
   6       47840       52759        906  0.00%  0.00%  0.00%   0 Pool Manager

Thanks!

Anyone have any thoughts on this?  It's a shameless bump on my end I know ;D

We really still need to collect more info. The difference between the 2 show ip cef switching statistics commands was 50 seconds. In this time the number of punts for features was 6,689 packets. These are sent to be process switched, but they alone may or may not be the problem. IP Input is ony used to process switch traffic. In a similiar fashion the Input queue of each interface is only used for process switched traffic. I would run the following command to filter down the interfaces:

show interfaces | include protocol|Input

In the output you want to take note of input queue with the most packets. You will probably see a lot of Input queue drops on the same interface. Then you can use the following command to see what that traffic is:

show buffers input-interface packet | include source

The output will be a list of all of the packets which are currently in the Input queue at that instant. The source IP, dest IP, source port, dest port, and protocol will be listed. A lot of times you can find a single stream which is the offending stream. The trick is to determine what shouldn't be there, because some of the traffic is likely to be legitamite.

As an example I had a customer today where we ran the above command sever times and nearly all of the traffic was using the port for SSH. All of the packets came from the same source and were destined to the same destination. The destination IP address wasn't configured on the router so I then checked the routing table. We didn't have a route to the destination which is why the traffic was being punted. If you can't find something glaring you can take the full output form the show buffers command and we can start decoding packets to look for other potential causes.

Hi George,

     The queue is mostly on our FA0/1(OUTSIDE-Internet) interface however there is a smaller queue on the TU3 interface.  When I ran the commands you provided it revealed:

PHXRTR02#sho buffers input-interface fastEthernet 0/1 packet | include source
  source: <PHXRTR02-Outside(FA0/1) IP>, destination: <CGYRTR01-Outside IP>, id: 0x7BCF, ttl: 239, prot: 50

     I am not sure about the port in use but showing IP cache flow did give me SrcP in HEX of 32E1(13025 DEC) and DstP in HEX of FE54(65108 DEC):

PHXRTR02#sho ip cache flow | inc <PHX-Outside-IP>

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts
Fa0/1         <CGY-Outside-IP>   Local         <PHX-Outside-IP>  32 32E1 FE54    14K
Fa0/1         <CGY-Outside-IP>   Local         <PHX-Outside-IP>  32 32E1 FE54    10K

   Again, it appears to be on port 13025 but I am not entirely sure what this is used for but it might be this is the encrypted traffic for the VPN between the two sites on TU3?

That is all encrypted traffic. Protocol 50 is for ESP. In the show ip cache flow output the Pr column is protocol and it is reported in hex. 0x32 == 50 in decimal. When you use the show buffers command this only displays packets at that given instant. You may need to run it several times to get a good sampling of packets. From the output it is hard to determine how many packets you looked at.

Hmm, isn't the packet count in the ip-cache flow indicitive of the amount of ESP traffic is flowing?  It also says in the command that active flows timeout in 30 minutes.  If that is the case, here is the current flow, but please do note that overall traffic is much lower right now, and CPU utilization as well.

SrcIf            SrcIPaddress         DstIf              DstIPaddress         Pr      SrcP      DstP       Pkts
Fa0/1           Local                32      16F5      F91F        148K
Fa0/1           Local                32      16F5      F91F        223K

As you suggested I ran the command several times yesterday and today and never saw anything but ESP packets in the buffers.  I guess the other thing I am confused about is that this flow is showing up in the "ip cache flow" command, does that mean that it is being properly CEF switched?