Process Switching Question

Unanswered Question
May 5th, 2010

Would the below output snippet of "show interfaces switching" be an acceptable amount of process switched packets?  Or would that be considered high?  Also it appears the CEF is dropping packets due to the Cache misses, is that correct?


Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     198527   20305755     137600   11936300
            Cache misses      58283          -          -          -
                    Fast  114698940 1539757122  176439333 2845662647
               Auton/SSE          0          0          0          0

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
francisco_1 Wed, 05/05/2010 - 08:08

When using process switching, the operation is to populate the fast cache, if the network is not stable, network conditions can cause an increased number of Cache misses (an instance where no cache entry matches a packet ) and can result in a high number of packets be process switched!.



Francisco

kyle01041981 Wed, 05/05/2010 - 08:11

OK, I am troubleshooting High CPU spikes and stumbled accross the

these cache misses.  But...I am unclear on

your answer...so to clearify, are you saying the amount of process switched packets in the output I provided is a High amount?

francisco_1 Wed, 05/05/2010 - 08:17

Can you post "show cef drop" and "show interface switching" (sorry o think you already posted show interface switching)

kyle01041981 Wed, 05/05/2010 - 08:38

See below....


router#show cef drop
CEF Drop Statistics
Slot  Encap_fail  Unresolved Unsupported    No_route      No_adj  ChkSum_Err
RP         88087         195           0       62615           0           0


router#show interfaces switching
FastEthernet0/0
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs     201275      Drops          0


    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     200009   20442235     138702   12022536
            Cache misses      58283          -          -          -
                    Fast  114971009 1593045035  176788338 3081229314
               Auton/SSE          0          0          0          0


    Protocol  DEC MOP            
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0        988      76076
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0


    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
--More--     Process     201217   12073028     224361   13461660
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0


    Protocol  CDP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      46150    5278389      19686    8726589
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0


    Protocol  PPPoE              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       5505     412052     224658   13533231
            Cache misses          0          -          -          -
                    Fast    2830219  540707918    3517128 2544941938
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        147      10770      59641    3578460
            Cache misses          0          -          -          -
--More--            Fast          0          0          0          0
               Auton/SSE          0          0          0          0


    NOTE: all counts are cumulative and reset only after a reload.
FastEthernet0/1
          Throttle count          0
                   Drops         RP        164         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs      13604      Drops          0


    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     642732   58538192     653639   63688078
            Cache misses          9          -          -          -
                    Fast  181742646 1495447728  118768428 1806409122
               Auton/SSE          0          0          0          0


    Protocol  DEC MOP            
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0        988      76076
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
--More       Auton/SSE          0          0          0          0


    Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         87       5220        243      14580
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0


    Protocol  CDP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0       9814    4035204
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      59065    3543900
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0


--More--             NOTE: all counts are cumulative and reset only after a reload.


Interface Serial0/1/0 is disabled



Interface Serial0/2/0 is disabled


Virtual-Access1


    All statistics for this interface are zero.
Virtual-Template1


    All statistics for this interface are zero.
Virtual-Access2


    All statistics for this interface are zero.
Virtual-Access3
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs        333      Drops          0


            Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       1551     119275       1542     118692
            Cache misses       1940          -          -          -
                    Fast     221290   35391142     405650  263132083
               Auton/SSE          0          0          0          0


    Protocol  PPPoE              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          1         18
            Cache misses          0          -          -          -
                    Fast          0          0       2572      25720
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      48841     683677
            Cache misses          0          -          -          -
                    Fast     134612    1608414          0          0
               Auton/SSE          0          0          0          0


    NOTE: all counts are cumulative and reset only after a reload.
Virtual-Access4
--More--                   Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs        184      Drops          0


    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          6        508         94       2352
            Cache misses          0          -          -          -
                    Fast     407691  111053673     619356  426091885
               Auton/SSE          0          0          0          0


    Protocol  PPPoE              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          0          0
            Cache misses          0          -          -          -
                    Fast          0          0       2090      20900
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      32930     460716
--More--                     Cache misses          0          -          -          -
                    Fast      92261    1061814          0          0
               Auton/SSE          0          0          0          0


    NOTE: all counts are cumulative and reset only after a reload.
Virtual-Access5
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs        186      Drops          0


    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         34       2576        103       3240
            Cache misses          8          -          -          -
                    Fast     182462   35921082     271895  186012176
               Auton/SSE          0          0          0          0


    Protocol  PPPoE              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          1         24
            Cache misses          0          -          -          -
--More--                             Fast          0          0       1357      13570
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      19502     272784
            Cache misses          0          -          -          -
                    Fast      49611     642742          0          0
               Auton/SSE          0          0          0          0


    NOTE: all counts are cumulative and reset only after a reload.
Virtual-Access6
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs        570      Drops          0


    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        732      51861        808      56976
            Cache misses       1354          -          -          -
                    Fast     256387   90681993     354825  234555456
--More--                        Auton/SSE          0          0          0          0


    Protocol  PPPoE              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          0          0
            Cache misses          0          -          -          -
                    Fast          0          0       1372      13720
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      23288     325609
            Cache misses          0          -          -          -
                    Fast      60972     760716          0          0
               Auton/SSE          0          0          0          0


    NOTE: all counts are cumulative and reset only after a reload.
Virtual-Access7
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs        440      Drops          0
--More--        
    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        771      70982        785      55626
            Cache misses       1427          -          -          -
                    Fast     344677   87538302     466489  323174015
               Auton/SSE          0          0          0          0


    Protocol  PPPoE              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          0          0
            Cache misses          0          -          -          -
                    Fast          0          0       1015      10150
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      19887     278034
            Cache misses          0          -          -          -
                    Fast      47650     665630          0          0
               Auton/SSE          0          0          0          0


    NOTE: all counts are cumulative and reset only after a reload.
--More--         Virtual-Access8
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs          4      Drops          0


    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          2         48
            Cache misses          0          -          -          -
                    Fast     549878   77643459     937254  676136368
               Auton/SSE          0          0          0          0


    Protocol  PPPoE              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          0          0
            Cache misses          0          -          -          -
                    Fast          0          0       1976      19760
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
--More--                          Process          0          0      55435     776085
            Cache misses          0          -          -          -
                    Fast     158559    1807258          0          0
               Auton/SSE          0          0          0          0


    NOTE: all counts are cumulative and reset only after a reload.
Virtual-Access9
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs        253      Drops          0


    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process         85       6837        158       5904
            Cache misses         64          -          -          -
                    Fast     275892   38647213     451171  354169113
               Auton/SSE          0          0          0          0


    Protocol  PPPoE              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0          0          0
--More--                     Cache misses          0          -          -          -
                    Fast          0          0        108       1080
               Auton/SSE          0          0          0          0


    Protocol  Other              
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      20296     283775
            Cache misses          0          -          -          -
                    Fast      48282     680188          0          0
               Auton/SSE          0          0          0          0


    NOTE: all counts are cumulative and reset only after a reload.
Loopback1


    All statistics for this interface are zero.
Tunnel2


    Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        619      64716        518      43512
            Cache misses          0          -          -          -
                    Fast     603324   65158992     438346   47341368
               Auton/SSE          0          0          0          0
--More--        
    NOTE: all counts are cumulative and reset only after a reload.



router#

francisco_1 Wed, 05/05/2010 - 08:58

Are you having network issues on this router, routing changes or high CPU?


I am seeing number of packets dropped could be related to packet exceeding the limit for packet ounted to the RP!! Also seeing packets being dropped by CEF!!!



router#show cef drop
CEF Drop Statistics
Slot  Encap_fail  Unresolved Unsupported    No_route      No_adj  ChkSum_Err
RP         88087         195           0       62615           0           0


Protocol  ARP                
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
--       Process     201217   12073028     224361   13461660



Protocol  IP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     200009   20442235     138702   12022536
            Cache misses      58283     


http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

kyle01041981 Wed, 05/05/2010 - 09:36

Yes, there are occasional HIGH CPU spikes.  What do you mean "related to packet exceeding the limit for packet ounted to the RP!!"?

francisco_1 Wed, 05/05/2010 - 09:49

sorry i meant punted not ounted.


Based on your CEF output under "Encap_fail", i am seeing cef drops. I got this direct from Cisco doc saying that The "Encap_fail" field indicates the number of packets dropped after exceeding the limit for packets punted to the processor because of missing adjecency information such as an unresolved ARP request. Note that CEF throttles packets punted to the process level at a rate of one packet per second to aid in suspectibility of Dos attacks.

kyle01041981 Mon, 05/10/2010 - 11:38

From this cisco document, there are numerous causes for this issue.

Do any of you know if there is a typical cause?  How would you recomend proceding?  One thing I noticed is I see around 100,000 broadcast packets recieved on the interface which I was suspious of and have condsidered issuing Mac-address accounting in that interface to try and see if all those broadcast were coming from one source address.


The main thing that has got me concerned is the amount of CEF drops or amount of " /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin;} Encap_fail".  Any guidance in this issue would be greatly appreciated.

francisco_1 Tue, 05/11/2010 - 10:04

I would start by debuging cef drops "debug ip cef drops" to see what traffic is being cef-dropped so you can track what apps is affected.



Francisco.

Paolo Bevilacqua Wed, 05/05/2010 - 09:46

These rates are definitely acceptable.


In most cases, process switched packets are sourced or destined for the router.

Actions

This Discussion