×

Warning message

  • Cisco Support Forums is in Read Only mode while the site is being migrated.
  • Cisco Support Forums is in Read Only mode while the site is being migrated.

Traffic shaping queue limit and packet drops

Unanswered Question
Jan 16th, 2014
User Badges:

Hi,  I have an ASR1001 with the below nested policy map applied to a GE WAN interface on the outbound direction.  The parent policy-map is just a shaper and the child policy-map does the actual class prioritization.


policy-map QoS_Out_to_rest
 class Mgmt_Out
  bandwidth 16
  set ip precedence 6
 class Voice_Out
  police 150000000 20480000 40960000 conform-action set-prec-transmit 5 exceed-action drop  violate-action drop 
  priority
 class Platinum_Out
  bandwidth 198000
  random-detect
  random-detect precedence 3 1045 1065
  police 198000000 50181000 10036200 conform-action set-prec-transmit 3 exceed-action set-prec-transmit 3 violate-action set-prec-transmit 3
 class Silver_Out
  bandwidth 12000
  random-detect
  police 12000000 2457000 737200 conform-action set-prec-transmit 1 exceed-action set-prec-transmit 1 violate-action set-prec-transmit 1
 class class-default
  bandwidth 239840
  random-detect
  police 239840000 44997000 90165000 conform-action set-prec-transmit 0 exceed-action set-prec-transmit 0 violate-action set-prec-transmit 0

policy-map Shaping_pmap
 class class-default
  shape average 600000000 60000000 60000000
  service-policy QoS_Out_to_rest




The problem I'm having is a high packet drop rate seen in the class-default on the child policy, which reflects in the parent policy as shown below in red:


WAN-CPE#sh policy-map interface gigabitEthernet 0/0/1 output

GigabitEthernet0/0/1


   Service-policy output: Shaping_pmap

 

    Class-map: class-default (match-any)

      145971879 packets, 46014416510 bytes

      30 second offered rate 174778000 bps, drop rate 114000 bps

      Match: any

      Queueing

      queue limit 2499 packets

      (queue depth/total drops/no-buffer drops) 0/34125/0

      (pkts output/bytes output) 145937765/45977616422

      shape (average) cir 600000000, bc 60000000, be 60000000

      target shape rate 600000000


      Service-policy : QoS_Out_to_rest


         queue stats for all priority classes:

          Queueing

          queue limit 512 packets

          (queue depth/total drops/no-buffer drops) 0/0/0

          (pkts output/bytes output) 86981436/18979811341

'.............................................

...............................................


        Class-map: class-default (match-any)

          57822243 packets, 26207121807 bytes

          30 second offered rate 101754000 bps, drop rate 153000 bps

          Match: any

          Queueing

          queue limit 998 packets

          (queue depth/total drops/no-buffer drops) 0/34125/0

          (pkts output/bytes output) 57788118/26170310963

          bandwidth 239840 kbps

            Exp-weight-constant: 4 (1/16)

            Mean queue depth: 1 packets

            class       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark

                    pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob



            0        57788118/26170310963  10331/11180773  23794/25630071          24            49  1/10

            1               0/0               0/0              0/0                 27            49  1/10

            2               0/0               0/0              0/0                 30            49  1/10

            3               0/0               0/0              0/0                 33            49  1/10

            4               0/0               0/0              0/0                 36            49  1/10

            5               0/0               0/0              0/0                 39            49  1/10

            6               0/0               0/0              0/0                 42            49  1/10

            7               0/0               0/0              0/0                 45            49  1/10

          police:




I tried increasing the queue-limit on the child pmap default class up to 3-fold but the traffic drops continued at virtually the smae rate.  Then I tried increasing the queue limit on the parent pmap but the IOS apparently dioes not support manually setting the queue-limit on the parent pmap. 


Is there any other option I could try?  The shaper average rate is well above the actual average traffic so more bandwidth doesn't seem to be the solution.


Thanks in advance for any suggestions on the matter.


Martin

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Vasilii Mikhail... Thu, 01/16/2014 - 02:06
User Badges:
  • Gold, 750 points or more

Hello.


I think you need to tune min/max threshold as it's done under Platinum_Out class.

I would guess that 500000 bytes (1000-1400 packets)  for max threshold (and twice less for min threshold) could be enough, but try different values.


Please aware that ARS is doing QoS in hardware, so there is a slight delay after configuration before it takes effect. Also examine the log to be aware of any error!

paul driver Thu, 01/16/2014 - 02:11
User Badges:
  • Green, 3000 points or more

Hello


Your qos config  doesn't seem correct me, your policing outbound for one and applying congestion avoidance to the child default class where I think you should be applying some degree of fairness to this catch all class.


For voice traffic I would suggest using LLQ instead of policing and for all the other policed traffic I would personally police this as it enters the router ( lan facing) and then apply some congestion management relating to this traffic on the wan links outbound using hierarichical qos (HQF)



res

Paul







Please don't forget to rate any posts that have been helpful.

Thanks.

Ashish Arora Thu, 01/16/2014 - 04:20
User Badges:
  • Cisco Employee,

Hi,


It seems like a queuing issue , before providing any feedback here , i would like to know the following things:-

1. Show interface gig (repeat output for 3-4 consecutive attempts).

2. Make sure to put load-interval 30 command under the gig interface before providing any outputs.

3. Are you we sure we are getting CIR 600 megs for this cirtuit, Can we test it by applying legacy QOS (rate-limiting) or Policing.

4. Aren't we geeting any burty traffic ?, This can be checked by a packet capture for 10-15 minutes and see the same under wireshark (go to statastics >> IO graph). Please share this output also.

5.since this is 600 megs link , we should need to check the output hold queue as paskets are getting tail dropped. so need to increase the size slowly and see the behaviour.(Note if we will increase the size at a higher number , we may expect voice delay),so please check the production impact befor applying it.

6. What is the actual issue are we experiance , Do you feel slowness or voice issue or any other issue.

7. I will provide you an action plan after seeing the output asked above and :-

   Show interface gig stats

   show interface gig switching.



-THank you

Ashish Arora

MARTIN LAUB Thu, 01/16/2014 - 05:22
User Badges:

Hi Ashish, please see the command outputs below. Keep in mind that the traffic volume dropped significantly since I opened the discussion, and so did packet drops.       


Here are the answers to your questions:

1 and 2:  please see below

3 the WAN circuit is a 1 GigE cirtuit. The provider is rate-limiting to 650 M so I'm shaping at 600M just in case

4 traffic IS bursty.There are some applications sending out 1000-UDP packet bursts or so

5 How can I check this please?

6 slow file transfers (topped at 2M). nuttcp shows a lot of TCP retransmissions due to packet drops.  Voice is fine. NO end user complaints

7. thanks !





WAN-CPEF#sh int gigabitEthernet 0/0/1

GigabitEthernet0/0/1 is up, line protocol is up

  Hardware is ASR1001, address is f0f7.5559.5601 (bia f0f7.5559.5601)

  Description: WAN

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 16/255, rxload 35/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  Keepalive not supported

  Full Duplex, 1000Mbps, link type is force-up, media type is LX

  output flow-control is on, input flow-control is on

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:05:43, output 00:05:43, output hang never

  Last clearing of "show interface" counters 01:31:11

  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 415

  Queueing strategy: Class-based queueing

  Output queue: 0/40 (size/max)

  30 second input rate 137456000 bits/sec, 28430 packets/sec

  30 second output rate 66031000 bits/sec, 23256 packets/sec

     190300536 packets input, 120170194379 bytes, 0 no buffer

     Received 2015 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     152849701 packets output, 55506977247 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out


WAN-CPE#sh int gigabitEthernet 0/0/1

GigabitEthernet0/0/1 is up, line protocol is up

  Hardware is ASR1001, address is f0f7.5559.5601 (bia f0f7.5559.5601)

  Description: WAN

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 16/255, rxload 35/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  Keepalive not supported

  Full Duplex, 1000Mbps, link type is force-up, media type is LX

  output flow-control is on, input flow-control is on

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:05:45, output 00:05:45, output hang never

  Last clearing of "show interface" counters 01:31:12

  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 415

  Queueing strategy: Class-based queueing

  Output queue: 0/40 (size/max)

  30 second input rate 137456000 bits/sec, 28430 packets/sec

  30 second output rate 66031000 bits/sec, 23256 packets/sec

     190300536 packets input, 120170194379 bytes, 0 no buffer

     Received 2015 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     152849701 packets output, 55506977247 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out


WAN-CPE#sh int gigabitEthernet 0/0/1

GigabitEthernet0/0/1 is up, line protocol is up

  Hardware is ASR1001, address is f0f7.5559.5601 (bia f0f7.5559.5601)

  Description: WAN

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 19/255, rxload 34/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  Keepalive not supported

  Full Duplex, 1000Mbps, link type is force-up, media type is LX

  output flow-control is on, input flow-control is on

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:05:46, output 00:05:46, output hang never

  Last clearing of "show interface" counters 01:31:14

  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 415

  Queueing strategy: Class-based queueing

  Output queue: 0/40 (size/max)

  30 second input rate 136538000 bits/sec, 28446 packets/sec

  30 second output rate 76238000 bits/sec, 24406 packets/sec

     190585301 packets input, 120338570846 bytes, 0 no buffer

     Received 2021 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     153116779 packets output, 55627828181 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out



WAN-CPE#sh int gigabitEthernet 0/0/1 switching

GigabitEthernet0/0/1 WAN

          Throttle count          0

                   Drops         RP       2453         SP          0

             SPD Flushes       Fast          0        SSE          0

             SPD Aggress       Fast          0

            SPD Priority     Inputs  336612027      Drops          0



    Protocol  IP

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process        365      34994          0          0

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE 848669476307 608144422524240 702953328190 280895435014649



    Protocol  ARP

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process      41295    2312520     381134   24392576

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE          0          0          0          0



    Protocol  Other

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process          0          0          0          0

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE      41295    2477700     381104   24390656



    NOTE: all counts are cumulative and reset only after a reload.


WAN-CPE#sh int gigabitEthernet 0/0/1 switching

GigabitEthernet0/0/1 WAN

          Throttle count          0

                   Drops         RP       2453         SP          0

             SPD Flushes       Fast          0        SSE          0

             SPD Aggress       Fast          0

            SPD Priority     Inputs  336612075      Drops          0



    Protocol  IP

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process        365      34994          0          0

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE 848669785041 608144625699596 702953598567 280895542993229



    Protocol  ARP

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process      41295    2312520     381134   24392576

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE          0          0          0          0



    Protocol  Other

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process          0          0          0          0

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE      41295    2477700     381104   24390656





WAN-CPEF#sh int gigabitEthernet 0/0/1 switching

GigabitEthernet0/0/1 WAN

          Throttle count          0

                   Drops         RP       2453         SP          0

             SPD Flushes       Fast          0        SSE          0

             SPD Aggress       Fast          0

            SPD Priority     Inputs  336612115      Drops          0



    Protocol  IP

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process        365      34994          0          0

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE 848670068158 608144792696556 702953853921 280895645436152



    Protocol  ARP

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process      41295    2312520     381134   24392576

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE          0          0          0          0



    Protocol  Other

          Switching path    Pkts In   Chars In   Pkts Out  Chars Out

                 Process          0          0          0          0

            Cache misses          0          -          -          -

                    Fast          0          0          0          0

               Auton/SSE      41295    2477700     381104   24390656

Vasilii Mikhail... Thu, 01/16/2014 - 22:09
User Badges:
  • Gold, 750 points or more

Hello, m.laub.


Have you tried the recommendation to add min/max thresholds for class-default?


policy-map QoS_Out_to_rest

  class class-default

random-detect precedence 0 500 900


To adjust policy the good (or even best) practice is to: 1) remove it from interface, 2) adjust policy, 3) apply back to the interface.

Ashish Arora Thu, 01/16/2014 - 06:48
User Badges:
  • Cisco Employee,

hi ,


THank you for pasting the outputs.

Since traffic rate is somethere close to 136 down and 67 Up. It seems to be ok for the link as Saturated bandwidth allowed is 650 Mbps ;however only 40 Packets can be buffered into the output queue , which is quite less when compared to 1 Gig link. I should say try to change it to 100 and keep on incrementing till 200.You can perform as follow:-


Interface gig 0/0/1

hold-queue 70 out



Check the behaviour


hold-queue 100 out



Check the behaviour  now for 5 minutes atleast.


hold-queue 150 out ( monitor it for some time)


We shoudn't increase this a lot beause it may create issues for the voice traffic and experience a little delay in RTP traffic.

Please also try to remove the policy and put it back in.



In order to check the CIR, We should follow the policy :-

Policy-map TEST

class class-default

police rate 600000000

    conform-action transmit

    exceed-action drop



Now we need to send traffic till it reached the max limit and once it would start dropping , CIR can be roughly observed.



-Thank you.

Ashish

Joseph W. Doherty Fri, 01/17/2014 - 04:54
User Badges:
  • Super Bronze, 10000 points or more

Disclaimer


The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.


Liability Disclaimer


In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.


Posting


I haven't thoroughly read all the other posts, but I think I noticed someone else noticed your class-default drops are in your WRED settings, 24/49.  Two third of your drops are WRED tail drops.  Those (default?) values are very likely way too small for 240 Mbps.


BTW, I don't know why you need to rate limit your non-LLQ classes, but if the ASR1001 supports child policy class shaping, I would generally recommend it over policing.  I would also recommend analysis of whether you really need to rate limit any of your non-LLQ classes.  (I'm guessing you're working with some WAN vendor that "sets aside" a certain amount of bandwidth for the non-LLQ classes, but often that bandwidth is proportionally prioritized.  If it is, shaping or policing is not taking advantage of otherwise available bandwidth.)

Actions

This Discussion