Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

drops in queue on subinterface with out congestion (QOS)

Hello colleagues

The problem is, that i have ~ 7-8 Mbps on 65Mbps link, but my service policy seems to drop packets according to queue size.

In my understanding, policy map start to drop traffic only when congestion appear, is it so?

same situation on ASR 1002 and 7206

ios  Version 15.2(4)S4 and  Version 15.1(4)M7

 

Here is config for policy map on WAN link

 

class-map CRITICAL

match precedence 5

class-map match-any IMPORTANT

match precedence 3

match precedence 6

class-map BUSSINESS

match precedence 1

 

policy-map SHAPING

class class-default

  shape average 65000000

  service-policy SP-QOS

 

policy-map SP-QOS

Class CRITICAL

priority percent 70

Class IMPORTANT

bandwidth percent 12

Class BUSSINESS

bandwidth percent 12

random-detect dscp-based

Class class-default

bandwidth percent 6

random-detect

 

interface GigabitEthernet0/0/1.9

encapsulation dot1Q 9

 ip address xxxxxxxxx

 ip nat outside

 ip nbar protocol-discovery

service-policy output SHAPING

 

and here, what I see in statistic

  Service-policy output: SHAPING

 

    Class-map: class-default (match-any)

      129798770 packets, 40181696005 bytes

      5 minute offered rate 7304000 bps, drop rate 0000 bps

      Match: any

      Queueing

      queue limit 270 packets

      (queue depth/total drops/no-buffer drops) 0/11680/0

      (pkts output/bytes output) 129779213/40165064207

      shape (average) cir 65000000, bc 260000, be 260000

      target shape rate 65000000

 

      Service-policy : SP-QOS

 

        queue stats for all priority classes:

          Queueing

          queue limit 512 packets

          (queue depth/total drops/no-buffer drops) 0/0/0

          (pkts output/bytes output) 13484092/3912327041

 

        Class-map: CRITICAL (match-all)

          13480980 packets, 3911647909 bytes

          5 minute offered rate 422000 bps, drop rate 0000 bps

          Match:  precedence 5

          Priority: 70% (45500 kbps), burst bytes 1137500, b/w exceed drops: 0

 

 

        Class-map: IMPORTANT (match-any)

          7721261 packets, 1021340017 bytes

          5 minute offered rate 108000 bps, drop rate 0000 bps

          Match:  precedence 3

          Match:  precedence 6

          Queueing

          queue limit 64 packets

          (queue depth/total drops/no-buffer drops) 0/0/0

          (pkts output/bytes output) 7715467/1020879486

          bandwidth 12% (7800 kbps)

 

        Class-map: BUSSINESS (match-all)

          47724414 packets, 15630803933 bytes

          5 minute offered rate 3393000 bps, drop rate 0000 bps

          Match:  precedence 1

          Queueing

          queue limit 64 packets

          (queue depth/total drops/no-buffer drops) 0/7388/0

          (pkts output/bytes output) 47741826/15628877447

          bandwidth 12% (7800 kbps)

            Exp-weight-constant: 4 (1/16)

            Mean queue depth: 1 packets

            dscp       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark

                    pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

 

            af11     2990415/861605540      25/32285        645/884994            28            32  1/10

            af12      677310/367870926       0/0             10/5466              24            32  1/10

            af13      154681/48657019        3/1634           4/307               20            32  1/10

 

        Class-map: class-default (match-any)

          60817716 packets, 19603759092 bytes

          5 minute offered rate 3363000 bps, drop rate 0000 bps

          Match: any

          Queueing

          queue limit 64 packets

          (queue depth/total drops/no-buffer drops) 0/4292/0

          (pkts output/bytes output) 60837828/19602980233

          bandwidth 6% (3900 kbps)

            Exp-weight-constant: 4 (1/16)

            Mean queue depth: 1 packets

            class       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark

                    pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

 

            0         3474437/1110326227    269/276936       742/851882            16            32  1/10

            1               0/0               0/0              0/0                 18            32  1/10

            2               0/0               0/0              0/0                 20            32  1/10

            3               0/0               0/0              0/0                 22            32  1/10

            4               0/0               0/0              0/0                 24            32  1/10

            5               0/0               0/0              0/0                 26            32  1/10

            6               0/0               0/0              0/0                 28            32  1/10

            7               0/0               0/0              0/0                 30            32  1/10

Everyone's tags (1)
26 REPLIES
New Member

i think i understand why

i think i understand why there are drops. Looks like RED is working fine and do some drops in order to avoid congestion

 

 

PS. Nope, still see that Tail Drops counter is growing

New Member

Hi, Is that fixed?

Hi, Is that fixed?

New Member

i still see a little bit

i still see a little bit dropped packets, but not so much as at first. i tuned a little parameters, acording to Akash advices.

New Member

Have you tried to remove

Have you tried to remove random-detect ?

New Member

no, i need random detect on

no, i need random detect on this queue. But, random drops has their own counter in this output (Random drop  pkts/bytes). And i`m confused about tail drops

New Member

So tried that increasing

So tried that increasing threshold min and max right? 

  1. queue limit

  2. Exp-weight-constant

http://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/quality-of-service-qos/white_paper_c11-481499.html 

 we had exact similar issue, but looks like need to consider what traffic going through the class, if like replication/file copying which using extreme burst packets ... then looks like that random-detect is checking utilization dropping some packets randomly and never gets the maximum bandwidth ... so may be looking for fair queue and remove that random-detect ... after that our link and that class is working 100%.

New Member

Thank you!yes, there is some

Thank you!

yes, there is some burst traffic... i`ll try you suggestion!

Super Bronze

DisclaimerThe Author of this

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Why do you need RED?

As the other posters have already noted, WRED (default) parameters are likely insufficient for 65 Mbps.  (Often Cisco defaults appears to be what you would want for a "typical" T1/E serial WAN link.)

WRED can be surprisingly difficult to configure correctly.

WRED tail drops happen when the average queue length exceeds the max threshold.  I.e. very much like tail drops on a normal FIFO queue.

However, ideally, you should not see WRED tail drops, just random early drops.  When you see WRED tail drops, you have an issue.

New Member

When you see WRED tail drops,

When you see WRED tail drops, you have an issue.

Yes, on most of my links i didn`t see any tail drops, just random drops. This link - is one that have such issue. Strange thing is that - all my links have very simular traffic (web,ftp,rdp) and none of them has tail drops on such small utilization.

 

for my understanding, please correct me if i wrong:

All parameters for RED configuration i see for this situation - is thresholds. So, we can increase max threshold in order to avoid tail drops. But if we will increase it to much, we will have a lot of non-actual traffic that waiting in a queue?

For my situation, we see small utilization of bandwith, so if i see tail drops, it mean that i have such bursty traffic, that tockens i get in one time is not enough. To avoid it - i can increase Bc. It helps a little.

If i turn off RED - there will be no mechanism for congestion avoidance on my link, and aggressive non-critical traffic will take a lot of bandwith, that`s why i want RED on class-default. Am i wrong?

 

Super Bronze

DisclaimerThe Author of this

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Are your other links are also using a shaper for the same bandwidth too?  If so, then yes, for some reason your traffic appears to be more bursty on this interface.

Yes, anytime you increase queue depths you often also increase latency (from more packets waiting in the queue to transmit).

Also yes, increasing Bc will help allow transient bursts to pass but then you're going "overrate" for increased time intervals.  That might be fine, but I assume there's a reason your shaping in the first place.  If you're overrate, your WAN vendor might drop excess traffic or queue it.  The problem with the former is obvious (although loss of what gets drop first might not be), but the latter also means you lose control over traffic prioritization.

Personally, if it's supported, I generally recommend FQ over WRED.  If there's transient congestion within the class, FQ will insure one monster flow isn't adverse to other flows within the same class.  Also, FQ should tail drop packets from the congestion causing flow(s) before the non-congested flows in that class.

BTW, for WRED to do its magic, the traffic has to be the kind that slows its transmission rate when a drop is detected, e.g. TCP.

New Member

all important UDP traffic

all important UDP traffic (VoIP) goes to critical queue, so if any UDP goes to class-default (and WRED) -> int`s not so bad. Main traffic in class-default is TCP, so it`s ok.

why do you prefer FQ instead of WRED? how it can replace WRED?

 

Thank you for help!

Super Bronze

DisclaimerThe Author of this

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Why do I prefer FQ?  Already explained, I thought.

How do enable FQ?  Remove WRED statements from class and add fair-queue.  (Note, you many need to adjust queue-length parameter.)

New Member

But in case of FQ, you must

But in case of FQ, you must describe all traffic very carefully. I mean:

In case of WRED, in perfect world, i will not have monster flow - one random drop and our growing tcp flow decrease it speed. So all flow in one class will have they band in most cases and there will be not so much fight for bandwith

In case of FQ, if i will have such big flow - it will influence on all flow in a class. Only tail drop will decrease this flow speed. If in one class will be 2 big flows, they will fight for bandwith and tail droped. So bandwith utilization will be not so effective

am i wrong?

Super Bronze

DisclaimerThe Author of this

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Are you wrong?  Yes and no.

First, with WRED, if there's a monster flow, along with other flows, the packet being dropped is random, i.e. it might not be from the monster flow.

Second, with WRED, the packet being randomly dropped is based on a moving average, so actual queue size can be much larger than the drop value trigger.  (Interesting, because it's a moving average, packets can be also dropped when the queue is small [after shrinking]).

Third, normal WRED drops packets as they are being added to the queue, so that, along with WRED's moving average, can delay a flow "knowing" about the drop.  I.e. A lot more packets can slam into your queue before the flow "knows" to slow.

Lastly, with WRED, it's one queue, so while all the above is happening, other flows are being impacted.

Yes, with FQ the monster flow will be tailed dropped, but other concurrent flows should not be.  Often, the monster flow, because there's no moving average delay, may slow faster than when using WRED.

With FQ, other flows are dequeued concurrently, so the monster flow has little impact on their queuing latencies (again unlike WRED).  Also again, other flow packets shouldn't be "mistakenly" dropped, as WRED can do.

In theory, a better WRED is Cisco's FRED, but its not widely provided across their products.

Interesting here's what Van Jacob has to say about RED: "there are not one, but two bugs in classic RED."

Research RED and you find lots and lots of research and variants to try to make it work better, including Dr. Floyd's (the creator of RED) revision - that should be a clue that it's not as simple as it appears.

 

PS:

Some later TCP implementation carefully monitor RTT, if they detect a jump in latency, they slow their flows, so for those you don't want to drop them at all.

Also note, I spent over 10 years working different QoS methodologies, in a large scale international (production) network.  My experience, WRED works better with high speed links and many flows.  It doesn't work so well at the slow end (i.e. T1/E1) with few flows.  At the slow end, latency is the bigger issue, and FQ better manages that.

New Member

Joseph, thanks a lot for your

Joseph, thanks a lot for your help!

I have only "QoS-FE:Implemeniting..." course as a background for QoS methods, and it was clear for me. But after your comments, i`ve understand, that sky is not so clean as i thought :) May be you can advice some helpfull material for deeper understanding (i`m now reading "IP Quality of Service [Srinivas Raju Vegesna]", but may be you can advice some thing more)?

Super Bronze

DisclaimerThe Author of this

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Unfortunately, I've never come across any really, really good source material that explained how to effectively use QoS in a production environment with real traffic.

Even simple "gotchas" like Cisco interface FIFO queue depth (tx-ring-limit) may need to be adjusted to meet QoS goals, but that's often overlooked.

I'll admit, when I first encountered WRED, I thought it was such a great idea (and in someways it still is), but nothing I initially read mentioned all the issues with it.  (Oh, and again, don't misunderstand, it can be useful, but it's a tricky technology to really obtain its best.)

Or, on WRED, besides the most obvious adjustment of min and max thresholds, when should the moving average parameter be adjusted.  Or, is 10% really the ideal value for the random drop percentage?

Did your course discuss those?  (If not, that's not a knock against your course, it's just determining the correct values for all RED parameters is difficult.)

Anyway, all I can suggest, if you really want to understand QoS, is read a lot.  Take any one source with a grain of salt.  Read research about "improved" versions of QoS approaches, especially taking note of what's the problem or issue trying to be solved (sometimes more interesting then their proposed gee-wiz solution).

Monitor you results and don't be afraid to adjust, but also be prepared for making things worse.  ;)

 

New Member

yep, we discuss it :)

yep, we discuss it :) actually - a lot of interesting information about qos tuning was on this course. But not so much practice on real WAN in my case. But, as for me - it`s very interesting to work on network improvement!

One more time - thank you for your help, time and good advices!

Have a nice day! :)

Cisco Employee

Hi, In this case i see two

Hi,

 

In this case i see two issues and few config tweaking can help us here. In case of shaping, packet gets queued only when traffic is crossing conform rate(packets (bits) arrived in Tc interval is more than Bc defined). Shaping rate of 65 mb for 1 sec is too high and we have very less traffic on the link but to check whether traffic conforming traffic contract or not, shaping tool uses Token bucket mechanism and in every Tc time interval it replenishes Bc amount of token. Here Bc is set very low which is just 260000 bits and if you calculate Tc it is 4 ms.

Bc = 260000 bits
CIR = 65000000 bits/sec
Tc = Bc/CIR = 260000/65000000
Tc = 4 ms

So every 4 ms, bucket will be filled with 260000 tokens. If within 4 ms router gets 21 packets (1500 bytes) so all tokens will be used and next packet will be queued in the respective queue. So for bursty traffic we may see queue getting filled for short duration and WRED is getting triggered. Probably it is happening for class BUSINESS and class-default. 

Even for latency/jitter sensitive traffic Tc is recommended as 10ms or by default it gets calculated as 25ms. You can try below values of Bc and Be for respective Tc.

 

Tc = 25 ms ---> Bc = 1625000 , Be = 1625000

Tc = 10 ms ---> Bc = 650000 , Be = 650000

 

Second thing queue-limit for each class is 64 packets, but WRED minimum threshold is set as 16 and maximum as 32. If changing Bc,Be does not help then WRED threshold needs to be increased.

 

Please do not forget to rate helpful post.

-Akash

New Member

Thank you Akash! for your

Thank you Akash!

 

for your first suggestion - i belive this could happen, if we have a lot of burst traffic. But on monitoring systems, i didn`t see any splashes (30s check rate). May be this burst traffic lasts not too long and i didn`t see it on monitoring...

I`ll try to set 10ms Tc, and will see what happened. 

 

second suggestion - i think you right. That was my confusion, couse i forget the way how WRED works. 

 

Thank you for your help!

New Member

still  see some tail drops in

still  see some tail drops in class-default (current Bc is 1625000)

 

Service-policy output: SHAPING

    Class-map: class-default (match-any)
      621651 packets, 242393853 bytes
      5 minute offered rate 4862000 bps, drop rate 0000 bps
      Match: any
      Queueing
      queue limit 270 packets
      (queue depth/total drops/no-buffer drops) 0/159/0
      (pkts output/bytes output) 621463/242224044
      shape (average) cir 65000000, bc 1625000, be 1625000
      target shape rate 65000000

      Service-policy : SP-QOS

        queue stats for all priority classes:
          Queueing
          queue limit 512 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 26105/5613542

        Class-map: CRITICAL (match-all)
          26105 packets, 5613542 bytes
          5 minute offered rate 116000 bps, drop rate 0000 bps
          Match:  precedence 5
          Priority: 70% (45500 kbps), burst bytes 1137500, b/w exceed drops: 0


        Class-map: IMPORTANT (match-any)
          17897 packets, 1842729 bytes
          5 minute offered rate 41000 bps, drop rate 0000 bps
          Match:  precedence 3
          Match:  precedence 6
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 17881/1841225
          bandwidth 12% (7800 kbps)

        Class-map: BUSSINESS (match-all)
          204242 packets, 55251009 bytes
          5 minute offered rate 1112000 bps, drop rate 0000 bps
          Match:  precedence 1
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 204242/55251009
          bandwidth 12% (7800 kbps)
            Exp-weight-constant: 4 (1/16)
            Mean queue depth: 1 packets
            dscp       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark
                    pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

            af11      171891/43885889        0/0              0/0                 28            32  1/10
            af12       24896/6077339         0/0              0/0                 24            32  1/10
            af13        7455/5287781         0/0              0/0                 20            32  1/10

        Class-map: class-default (match-any)
          373394 packets, 179683059 bytes
          5 minute offered rate 3607000 bps, drop rate 0000 bps
          Match: any
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/159/0
          (pkts output/bytes output) 373235/179518268
          bandwidth 6% (3900 kbps)
            Exp-weight-constant: 4 (1/16)
            Mean queue depth: 1 packets
            class       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark
                    pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

            0          373227/179516812      47/51277        112/113514            16            32  1/10
            1               0/0               0/0              0/0                 18            32  1/10
            2               0/0               0/0              0/0                 20            32  1/10
            3               0/0               0/0              0/0                 22            32  1/10
            4               8/1456            0/0              0/0                 24            32  1/10
            5               0/0               0/0              0/0                 26            32  1/10
            6               0/0               0/0              0/0                 28            32  1/10
            7               0/0               0/0              0/0                 30            32  1/10

Cisco Employee

 Hi, As I mentioned before,

 

Hi,

 

As I mentioned before, please try to increase WRED minimum,maximum threshold. right now it is 16 32, you may increase it to 32 48.

New Member

Done, cleared statistic - get

Done, cleared statistic - get TailDrop hit in a first few seconds sad

 

       Class-map: class-default (match-any)
          60351 packets, 22005334 bytes
          5 minute offered rate 559000 bps, drop rate 0000 bps
          Match: any
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/4/0
          (pkts output/bytes output) 60347/22003490
          bandwidth 6% (3900 kbps)
            Exp-weight-constant: 4 (1/16)
            Mean queue depth: 1 packets
            class       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark
                    pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

            0           60345/22003126        3/1774           1/70                32            48  1/10
            1               0/0               0/0              0/0                 18            32  1/10
            2               0/0               0/0              0/0                 20            32  1/10
            3               0/0               0/0              0/0                 22            32  1/10
            4               2/364             0/0              0/0                 24            32  1/10
            5               0/0               0/0              0/0                 26            32  1/10
            6               0/0               0/0              0/0                 28            32  1/10
            7               0/0               0/0              0/0                 30            32  1/10

Cisco Employee

 If you see number of drops

 

If you see number of drops have been reduced significantly. Earlier there were 159 drops (random + tail) in total 373227 transmitted packets which means 1 drop out of 2347 packets. Now it is 1 drop out of 15086 packets. So drop rate has been reduced by 6 times. If you want to reduce drop further, need to increase WRED threshold more.  If you have bursty traffic in network, you need to have larger queue to hold the bursty traffic.

New Member

i`ve checked my first post

i`ve checked my first post statistic for Tail Drop - 

            class       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark

                                     pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

          0         3474437/1110326227    269/276936       742/851882            16            32  1/10

 

So it about 1 of 12916 packet drop rate

today statistic

            class       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark
                    pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

            0        32108690/9746181825    895/1117314     3940/4774446           32            48  1/10

 

So it about 1 of 8149 packet is Tail droped.

I`ll try to set thresh a little bit larger, but don`t look like it helpes a lot.

Maybe set Bc larger?

Cisco Employee

 Hi,Definitely you can try

 

Hi,

Definitely you can try with increasing Bc. However would like to share below info related to WRED max threshold in HQF IOS.

The IOS you are using, i guess HQF is implemented and in HQF queue-limit of each class takes precedence over max-threshold of the WRED unlike in pre-HQF images. By default class queue-limit is 64 so WRED threshold can not be exceed more than the queue-limit(64). If you want to increase WRED max threshold more than 64 then queue-limit also needs to be increased. You can try to increase queue-limit to 1000 and WRED also as min-600 ,max-1000. Just for testing.

 

New Member

thanks, i`ll try!

thanks, i`ll try!

1135
Views
35
Helpful
26
Replies
CreatePlease to create content