WRED Tail Drops

[Craig Budrodeen]

We have a POS 155 Mbps circuit.

There are tail drops occurring on the default class which is allocated 25% (38.75 Mbps). There is some voice traffic but that is priority queued and is not a problem. Other control classes also OK.

But when I look at the overall utilization of the physical interface (155 Mbps) it is not congested. It peaks at 40% during business hours - most of which is default class.

It was my understanding that WRED should not "kick in" unless there is congestion - as defined by the WRED algorithm.

Does "congestion" here apply to the physical interface or does it apply the the 25% allocated to defualt traffic?

If the tail drops are occurring because of "congestion" on the default class, what parameters should I tweak - queue lengths or the bandwidth allocated to default?

policy-map eth-service-policy
  class voice
  priority percent 35
  class voice-control
  bandwidth percent 2
  class network-control
  bandwidth percent 5
  class class-default
  bandwidth percent 25
  random-detect

policy-map frame-relay-shape-policy
  class class-default
  shape average 154000000
  service-policy eth-service-policy

interface POS1/0
 description "..."
 no ip address
 encapsulation frame-relay
 no keepalive
 no arp frame-relay
 no frame-relay inverse-arp
 service-policy output frame-relay-shape-policy
 hold-queue 1000 out

show policy-map interface POS1/0

 POS1/0

  Service-policy output: frame-relay-shape-policy

    Class-map: class-default (match-any)
      1802709602 packets, 720560400569 bytes
      5 minute offered rate 26898000 bps, drop rate 0 bps
      Match: any
      Traffic Shaping
           Target/Average   Byte   Sustain   Excess    Interval  Increment
             Rate           Limit  bits/int  bits/int  (ms)      (bytes) 
        154000000/154000000 962500 3850000   3850000   25        481250  

        Adapt  Queue     Packets   Bytes     Packets   Bytes     Shaping
        Active Depth                         Delayed   Delayed   Active
        -      0         1802673602 3259356776 79971     71426633  no

      Service-policy : eth-service-policy

<omitted>

Class-map: class-default (match-any)
          1405375819 packets, 644742811187 bytes
          5 minute offered rate 24889000 bps, drop rate 0 bps
          Match: any
          Queueing
            Output Queue: Conversation 267
            Bandwidth 25 (%)
            Bandwidth 38500 (kbps)
            (pkts matched/bytes matched) 94915/108733008
        (depth/total drops/no-buffer drops) 0/36015/0
             exponential weight: 9
             mean queue depth: 0

  class    Transmitted      Random drop      Tail drop    Minimum Maximum  Mark
           pkts/bytes       pkts/bytes       pkts/bytes    thresh  thresh  prob
      0 1405364393/644719714663   4024/4793657    31991/36714190    20      40  1/10
      1   25378/1116693         0/0              0/0           22      40  1/10
      2     175/7736            0/0              0/0           24      40  1/10
      3       0/0               0/0              0/0           26      40  1/10
      4       0/0               0/0              0/0           28      40  1/10
      5       0/0               0/0              0/0           30      40  1/10
      6       0/0               0/0              0/0           32      40  1/10
      7       0/0               0/0              0/0           34      40  1/10
   rsvp       0/0               0/0              0/0           36      40  1/10

The drop packet counters increment when I check them overnight, and we have graphs that show random and tail drops.

Thank you (in anticipation).

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

But when I look at the overall utilization of the physical interface (155 Mbps) it is not congested. It peaks at 40% during business hours - most of which is default class.

You shouldn't be getting drops if there's no congestion.  % utilization doesn't often really indicate whether you have congestion or not.  (NB: there's congestion whenever a packet is queued.  If ingress bandwidth to this device allows for more than egress bandwidth to your PoS, you'll likely have packets being queued.)

It was my understanding that WRED should not "kick in" unless there is congestion - as defined by the WRED algorithm.

Correct.

Does "congestion" here apply to the physical interface or does it apply the the 25% allocated to defualt traffic?

Neither in this case.  It applies when traffic is being queue in the default class and that is determined by your shaper.  (Incidentally, are you sure you need a shaper?  I.e. is your POS is physically also 155 Mbps?)

If the tail drops are occurring because of "congestion" on the default class, what parameters should I tweak - queue lengths or the bandwidth allocated to default?

Depends on what you what to accomplish; i.e. your service level goals.  When managing congestion, when you push here, something else often pops out there.  I.e. it's all about trade-offs.

That noted, assuming you'll have about 39 Mbps for this class, the WRED defaults are probably too shallow (as Cisco's common defaults seem to have be selected for "typical" WAN T1s).

IMO, WRED is difficult to tune unless you're an expert in QoS.  I would recommend you remove it altogether.  You might consider using FQ, though.

[Vasilii Mikhailovskii · Accepted solution]

Hello, Craig.

The issue here is min and max thresholds you have on class-default WRED. As you run pretty fast link with a lot of users, traffic must be bursty, and mean queue of 20/40 packets is not enough!

For 100M link I would suggest to have tresholds at least 200  and 300 packets (or even more) respectively; queue size about 400-500.

Best regards.

PS: queue length constraints comes from queuing delay and buffers available; I guess in your case only queuing delay would come into play... so 300 packets*500 bytes*10/150M = 1/100 sec = 10 ms (in worst case).

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

As I noted in my origianl post, "the WRED defaults are probably too shallow", but I didn't suggest new values because as I also noted "WRED is difficult to tune unless you're an expert in QoS".

Vasilii is suggesting an initial threshold of 200 to 300 packet (or more) and a total queue size of 400 to 500 packets.  He notes "so 300 packets*500 bytes*10/150M = 1/100 sec = 10 ms (in worst case)", but is this really true?

He assumes your average packet size is about 500 bytes, but for "worst case" calculations it might be better to assume we'll have to deal with maximum size packets, so lets assume 1500.  If we do, though, we just tripled the delay.

Next, the 300 was actually suggested as the initial drop point, not max queue size, which was suggested as 400 to 500.  So, 500 packets*1500*10/150Mbps = 50ms.  That's not too bad, but WRED uses averages, so a much larger fast burst could go beyond 500 packets while WRED sees the average queue depth as much less.

How much larger?  I don't know, as that would depend on ingress to this device and WRED settings for how fast it adjusts its moving average.  But if it could go much higher resulting in additional delay.

Vasilii also assumes you have 150 Mbps but you've only defined 25% for class-default.  So, again assuming worst case, i.e. you've only guaranteed 1/4 the bandwidth, your latency would increase by a factor of 4.  In other words, not even accounting for WRED's moving average, allowing total queue depth beyond defined max, wost case latency for this class might be 200ms.

So, I wouldn't count on your worst case delay only being 10 ms.

However, even if Vasilii is correct about worst case delay being only 10 ms, increasing the min max thresholds as he suggests, might result in an increase in drops!

This is counter to what we normally expect, but with later TCP clients with very large RWINs, running across LFNs, slow start can create some very large bursts which can result in massive tail drops.

One of the reasons for WRED is to drop packets before they burst into a huge tail drop situation; remember WRED tail drop is just as bad as ordinary FIFO tail drop.  Ideally we want WRED's early drop to avoid us hitting any tail drop.  Getting this right has additional issues, one of the most serious is Cisco's WRED early drops as packets are added to the egress queue, which can mean a delay before the sender realizes there's a drop.  There's also the issue not all flows types slow when there are individual drops.  Also Cisco's WRED might drop packets from flows not even causing the congestion, while the congestion causing flow packets are not dropped (NB: Cisco FRED, addresses this, but few Cisco platforms support it).

I could go on (and on) but I'm hoping the above will how show that using WRED, to obtain it's full potential, can be a little more involved than just increasing its min and max settings.  Just doing that may, or may not, decrease the drops you're seeing, but QoS is really about meeting certain service objectives.  Often it's thought "no drops" is the ideal, but managed drops might better meet certain service objectives.

In my original post, I suggested using FQ (if supported), because it's not nearly as complicated to leverage properly.  It helps insures light bandwidth using flows aren't as adversely impacted by heavy bandwidth using flows, and it also tends to drop packets first from the flows causing the congestion.

[Vasilii Mikhailovskii]

Hello.

You might have missed, that I suggest to run min/max thresholds about 200/300 and queue size of 400-500, so class-default class queue would never grow larger than 400-500 packets.

It's hard to believe, that all the packets will be 1500 bytes size, but in theory it's definitely might be a case. 

From the diagram we see that traffic being dropped even when link utilization is below 100M; that is why I wrote about WRED thresholds. Author also stated that:

But when I look at the overall utilization of the physical interface (155 Mbps) it is not congested. It peaks at 40% during business hours - most of which is default class.

So, what is the point to calculate only 25% of 150M for class-default and to talk about queuing delay of 200ms?

PS: if you run fair-queue, you will face a limitation of flow limits router would track... I guess you might track 2000 flows, but it's unreal to have this low flow count on 150M link (unless it's a backup traffic only).

PS2: 200/300 thresholds are the values we run on POS 150M links, they are not just hypothetical.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You might have missed, that I suggest to run min/max thresholds about 200/300 and queue size of 400-500, so class-default class queue would never grow larger than 400-500 packets.

No, didn't miss, but my understanding is otherwise.  Max queue size, with WRED, is based on current average not actual queue depth.  I.e. tail drops will happen when average queue depth exceeds max; actual max can be more or even less!  (I.e. you can have tail drops, in theory, even when the actual current queue depth is zero.)

It's hard to believe, that all the packets will be 1500 bytes size, but in theory it's definitely might be a case. 

Agreed, but if you're going to plan of worst case, plan for worst case.  Server to server bulk transfers, like various kinds of replications, tend toward this.

From the diagram we see that traffic being dropped even when link utilization is below 100M; that is why I wrote about WRED thresholds. Author also stated that:

"But when I look at the overall utilization of the physical interface (155 Mbps) it is not congested. It peaks at 40% during business hours - most of which is default class."

So, what is the point to calculate only 25% of 150M for class-default and to talk about queuing delay of 200ms?

Again, planning for worst case, but I "simplified" the calculation.  Craig also wrote "allocated 25% (38.75 Mbps)", (1/4 of 155), but that's also incorrect as Craig didn't actually allocate 100%, and as 35% of his allocation is for LLQ, the class-default really is guaranteed (about) 43.9% (25/(25+5+2)*65) of the shaped 154.  I.e. not quite as dire as 200ms, but if he did keep the 25% for class-default and allocate the remaining portion of the 100%, it could be.

Craig also wondered how you can have drops when peak is only 40%, i.e. believing "it is not congested".  Many, I believe, don't fully appreciate that this is 40% of bandwidth capacity over the measured time interval, often 5 minutes.  It may, or may not, provide any clues about what might be happening down at the ms level.  You can have cases of high drops with 1% utilization and zero drops at 100% utilization.  Cisco did (still?) have a technology to give visibility into bandwidth needs, see http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_bndwth/configuration/15-mt/qos-bndwth-15-mt-book/qos-bndwth.pdf.

PS: if you run fair-queue, you will face a limitation of flow limits router would track... I guess you might track 2000 flows, but it's unreal to have this low flow count on 150M link (unless it's a backup traffic only).

You sure do, but a) many flows are short lived, and b) routers actually hash them into (relatively few) flow queues.

PS2: 200/300 thresholds are the values we run on POS 150M links, they are not just hypothetical.

I don't doubt it.  But a) you're really, really sure they're optimal for you, and b) how do you know they will be optimal for Craig?  Basically, I don't think we know enough to suggest what would be appropriate settings for Craig ditto for me to suggest any revision for your current settings.  For both you and Craig, don't know your QoS service goals, don't know anything else about your topology; heck don't even know Craig's BDP.

For a long time on these forums, I've recommended against using WRED, as I believe it's much more complex to use to full advantage then it at first appears.  (Oh, and like your 200/300 thresholds, my experience isn't just hypothetical.)

Recently I was reading this Wiki article, http://en.wikipedia.org/wiki/CoDel, and noticed this "One of the weaknesses in the RED algorithm (according to Jacobson) is that it is too difficult to configure (and too difficult to configure correctly, especially in an environment with dynamic link rates).", so I'm no longer alone in my opinion.

Any way, that's IMNSHO (in my not so humble opinion )

[Craig Budrodeen]

Thank you both Vasilii and Joseph.

These are really thoughtful answers and was just what I was after.

Cheers,

Craig

[TM13]

Hi Craig,

   Is that problem solved? if yes what exactly you did?

[Vasilii Mikhailovskii]

Hello, Joseph.

You wrote:

Max queue size, with WRED, is based on current average not actual queue depth.  I.e. tail drops will happen when average queue depth exceeds max; actual max can be more or even less!  (I.e. you can have tail drops, in theory, even when the actual current queue depth is zero.)

You also wrote that you don't account yourself as QoS expert, so please provide a proof-link that "MAX queue link is based on current average" and it's not configurable.

I don't doubt it.  But a) you're really, really sure they're optimal for you, and b) how do you know they will be optimal for Craig?

Yes, I'm sure they are optimal for us, as they were designed by QoS Experts. Not sure for Craig, but I suggested the values to try, as, per my understanding, they are more suitable for 150M link than 20 packets.

In original post we see:

Class-map: class-default (match-any)
      1802709602 packets, 720560400569 bytes        Adapt  Queue     Packets   Bytes     Packets   Bytes     Shaping

        Active Depth                         Delayed   Delayed   Active
        -      0         1802673602 3259356776 79971     71426633  no

Packets delayed are really small value comparing to overall packet number (this means that shaping is almost not involved - really no congestions on the link). And drops under class-default are primarily "tail drops". So, drops are definitely caused by bursts!

Also, there is a dangerous behavior (some IOS) in FQ assigned to class-default: it might starve as all the other classes (defined with "priority" and "bandwidth") may borrow all the bandwidth leaving nothing to class-default!

PS: garanteed bandwidth for class-default in this scenario is = 25% /(25+2+5 %)*65% = 51% => 75M.

PS: one more flaw in configuration - class "voice" may allocate more than 35% if the link is not congested (as it has no policer configured) and, I would guess, it's not an expected behavior.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You also wrote that you don't account yourself as QoS expert, so please provide a proof-link that "MAX queue link is based on current average" and it's not configurable.

Where did I write I don't consider myself an QoS expert?  I may have, though, as I don't consider myself an expert at anything.

I don't believe I wrote max isn't configurable.

It might take some time to find a "proof' link that you would accept, but why don't you view a WRED interface, with congestion in action.  Note the difference between actual queue depth and WRED computed queue depth.  It uses the latter for deciding when to apply the min/max thresholds.  This because, one of the design features of WRED is to allow transient bursts.  I.e. actual queue size can exceed max threshold for a short while.

Yes, I'm sure they are optimal for us, as they were designed by QoS Experts. Not sure for Craig, but I suggested the values to try, as, per my understanding, they are more suitable for 150M link than 20 packets.

I never questioned whether your settings are optimal for you, just I don't know.  [edit, in reflection, guess I did when I asked were you sure they were optimal.]  But since QoS Experts set yours, then I guess they must be optimal.  However, until the same QoS Experts review Craig's needs and environment, I wouldn't assume your settings will be optimal for Craig.  (IMNOHO, if your QoS Experts believe one-size always fits all, I would suspect how expert they are.  Optimal queue settings can very much be impacted by BDP, i.e. same bandwidth but different latency, changes the value.)

Packets delayed are really small value comparing to overall packet number (this means that shaping is almost not involved - really no congestions on the link). And drops under class-default are primarily "tail drops". So, drops are definitely caused by bursts!

Most likely.  However, that alone doesn't warrant increasing min/max settings, as how you drop and why you drop can be an important component for meeting your QoS policy.  One could even argue, the overall drops are so low, you need not change anything.

Also, there is a dangerous behavior (some IOS) in FQ assigned to class-default: it might starve as all the other classes (defined with "priority" and "bandwidth") may borrow all the bandwidth leaving nothing to class-default!

Yes, you're partially correct.  The behavior applies to pre-HQF.  It does not apply to LLQ.  It also doesn't take all bandwidth.  (What actually happens, FQ, in pre-HQF, active flow queues are not limited by just the class bandwidth reservation.)

PS: one more flaw in configuration - class "voice" may allocate more than 35% if the link is not congested (as it has no policer configured) and, I would guess, it's not an expected behavior.

That's also partially correct.  Implicit LLQ policing doesn't engage until packets would queue.  However, if there's congestion, i.e. queuing, the LLQ policer engages.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You also wrote that you don't account yourself as QoS expert, so please provide a proof-link that "MAX queue link is based on current average" and it's not configurable.

Found http://www.cisco.com/c/en/us/support/docs/routers/12000-series-routers/18841-mdrr-wred-18841.html?referring_site=smartnavRD#topic2

Although for a 1200, as far as I know, basic operation is similar across platforms.

Much of this, I believe, confirms what I've already written.

For example:

When the weighted queue average is below the minimum threshold, no packets are dropped. When the weighted queue average is above the maximum queue threshold, all packets are dropped until the average drops below the maximum threshold. When the average is between the minimum and the maximum thresholds, the probability that the packet will be dropped can be calculated by a straight line from the minimum threshold (probability of drop will be 0) to the maximum threshold (probability of drop is equal to the 1/mark probability denominator).

It is a complex challenge to tune WRED parameters to manage the queue depth, and depends on many factors, which include:

• Offered traffic load and profile.

• Ratio of load to available capacity.

• Behavior of traffic in the presence of congestion.

These factors vary network by network and, in turn, depend on the offered services and on the customers who use those services. Thus, we cannot make recommendations that apply to specific customer environments. However, table 4 describes generally recommended values based on the bandwidth of the link. In that case, we do not differentiate the dropping characteristics between the different classes of service.

Table 4 – Recommended Values Based on the Bandwidth of the Link

Bandwidth	Theoretical BW (kbps)	Physical BW1 (kbps)	Minimum Threshold	Maximum Threshold
OC3	155000	149760	94	606
OC12	622000	599040	375	2423
OC48	2400000	239616	1498	9690
OC192	10000000	9584640	5991	38759

•  

There are several constraints that are taken into account to compute the above threshold values. For instance, the maximization of the link utilization while minimizing the average queue depth, the difference between the Maximum and Minimum must be a power of two (due to hardware limitation). Based on experience and simulation, maximum instantaneous depth of a queue controlled by RED is less than 2 MaxTh. For 0C48 and above, 1 MaxTh, and so on. However, the exact determination of these values is beyond the scope of this document.

In the above generic recommendation, the min and max are both smaller and larger than Vasilii's settings.  Again, though, optimal settings depend on your goals, traffic, and environment.

I will say, my personal experience has been max should be much more than 2x min, which appears to be the case with the above generic recommendation.

So, again, WRED to optimize, requires an expert.  That said, I will agree with Vasilii that increasing your min max settings will likely decrease the drops your seeing now.  You might try Vasilii's suggested values, or those that Cisco recommends, above.  Of the two, my general experience is the larger ratio between min and max, that Cisco is suggesting, works better.