CBWFQ/LLQ Bandwidth allocation question, please help

[blackladyJR]

Hello,

After reading multiple document, I have left with many question marks in my head. Can you please help to clarify?

I know by default total bandwidth is 75% for all the class defined in the policy map including voice and data (i.e including the bandwidth from LLQ). So here is the example:

Say we stay with the default so 25% belongs to default class and say we have a T1 serial so bandwidth is 1536 on the interface.

Now, 25% of 1536 is 384k. So we are left with 1152k to work with for all the voice and AFxx I assume. Say we have 128k for voice in LLQ. So that left with 1024k for all the AFxx class to use and I assume I just have one AF41 to use all the bandwidth

policy-map test

class EF

set ip dscp ef

priority 128

class AF41

set ip dscp af41

bandwidth 1024

class class-default

So i assume that will be the policy-map. If I want to define the exact above via bandwidth percent, will it be like this?

policy-map test

class EF

set ip dscp ef

priority 128

class AF41

set ip dscp af41

bandwidth percent 67

class class-default

So I am using bandwidth percent so it is the absolute percent to the interface so 1024k of 1536 will be 66.7%. So the default class still will be 25% with 384k, correct? The voice is 8.3% so when I add LLQ + AF41 + BE = 8.3+66.7+25=100% of 1536.

How about if i use bandwidth remaining percent? how would that change?

policy-map test

class EF

set ip dscp ef

priority 128

class AF41

set ip dscp af41

bandwidth remaining percent 73

class class-default

why 73? if the remaining percent is relative term to the bandwidth left after LLQ. Then we have 1536-128=1408 left for AF41 and Default class. So 1024k/1408k = 72.7% while the same 384k Default will be 27.2% in the bandwidth remaining percent perspective.

So is it correct that Default class is 25% in bandwidth percent (absolute) and it is 27.2% under bandwidth remaining percent with 128k LLQ?

So when I use bandwidth percent, when I add up the percent value of AF41 + Default is not equal to 100%. It will be 100% if I add LLQ + AF41 + Default.

but when I use bandwidth remaining percent, then the % value will be 100% when I add AF41 + Default because it is relative term of the remaining bandwidth after LLQ. So 73%+27% in AF41+BE under bandwidth remaining = 100% of 1408k.

[Joseph W. Doherty]

I'm afraid I've lost my "score card" with all the question 4 variations.

With 4d, whenever you use FQ within class-default on most platforms, it disturbs defined class bandwidth calculations because on many platforms every class-default flow competes for bandwidth with defined classes.  In other words, class af41 might not obtain its minimum bandwidth guarantee (although it may obtain more).

With your other #4 questions, try to keep in mind CBWFQ attempts to guarantee minimum bandwidth, not a fixed slice of bandwidth.  Again, that's why the class bandwidth sums don't have to hit 100% yet they can not allocate more than 100%.

Further, CBWFQ classes, by default, preclude explicitly reserving more than 75%, to allow some bandwidth for other stuff including class-default when it's bandwidth isn't explicitly defined.

In other words, given a T-1, instead of trying to insure each class gets its maximum share of bandwidth, you use CBWFQ to insure it gets the minimum it needs to work well.  If class af41 only needed 256 Kbps to work well, you define:

policy-map x

class ef

priority 128

class af41

bandwidth 256

class class-default

bandwidth 8 (think 8 is minimum parameter)

This policy could be used on any interface that supports the policy and it would guarantee class af41 should also obtain at least 256 Kbps.  You don't need to try to figure out how to slice off the maximum amount of bandwidth that might be available to the class.

If you're still trying to get a handle on possible bandwidth allocations, I can try to explain that better, but I'm trying to convey you might be missing how you actually want to use CBWFQ, and normal usage doesn't require such computations.

Further if you think in relative bandwidth ratios, e.g. I want class af41 to have a minimum of half the link's bandwidth, you just use bandwidth percentage statements (which also allows the same policy to be used on interfaces with different amounts of bandwidth).

[blackladyJR]

Joe,

Once again, thank you so much for your patience :)

Yes I do want to understand better on bandwidth allocation because I really need that for project QoS work that we need to try to figure out the "max" of each class we can have.

First of all, thank you for clarifying the main purpose of CBWFQ is to two things : 1) guarantee minimum bandwidth for each class as defined during congestion 2) NOT cap maximum bandwidth while every class can "burst" more than their "minimum" defined bandwidth when it's available. (while no congestion).

So now I understand that we don't need to define the bandwidth to add up 100%, just don't define more than 100%.

For my project, I need to understand how to slice my interface to full maximum so during congestion, I know what each class (include default) will guarantee as a minimum will get so will maximize each class to its full use. Having said that, here are my revised questions:

1. You have mentioned when Default class is in FQ on most platform, it will compete with the class in CBWFQ so the class in CBWFQ won't get their minimum guarantee bandwidth. What do you mean by this?

So if we use the same e.g. T1 = interface, 128k = voice. cisco default 75%

policy-map x

class ef

priority 128

class af41

bandwidth 1024

class class-default

fair-queue

With cisco default and not explicit define default class, that means default class with routing overhead will get 25% of T1, so that's 384k. But you said FQ in class-default will compete with CBWFQ, so that means if I have more than 384k worth of default class traffic, and I have more than 1024k worth of AF41 traffic "during congestion", what will happen? Will Default traffic gets "more than" 384k and AF41 gets less than "1024k" even when it is "bandwidth 1024" in CBWFQ for AF41? I mean any QoS is really not relevant during "no congestion", so I am interested to know what will happen during congestion when QoS is really in use when I have more than 1024k AF41 traffic and more than 384k default class traffic.

2. by configuring class-class default

fair-queue

or just class-class default, it makes no difference, correct? meaning whether I specify 'fair-queue' or not, I still have fair-queue on the default class, right?

3. So for my project where I really want to guarantee some minimum bandwidth for my default class while maximizing other AFxx class guaranteed bandwidth. Under this objective, that means I want to explicit define class-default with bandwidth command. So the class-default will fall under the 75% bandwidth during congestion. So in this case I would set max-reserved-bandwidth 95 so I can define all my AFxx + class-default within 95% of bandwidth to maximize it while leaving 5% for routing overhead. Is it true that the 5% will be used by routing overhead when there is routing overhead traffic passing?

4. If I don't explicit define class-default with bandwidth, then that means class-default traffic will goto the 5% during congestion if all 95% are allocated to AFxx max out, if that true?

5. I saw someone do max-reserved-bandwidth 100, so what does that mean? We don't set aside any bandwidth for routing overhead at all? (this is assuming we will explicit define bandwidth for class-default so it doesn't fall to the "set-aside" pile which in this case we don't even have anything to set aside. So with max-reserved-bandwidth as 100% and class-default do have explicit bandwidth command, so where will the routing overhead go? I mean is it bad to set 100 and not set aside anything? Will this work?

final question -- continue on next message

[blackladyJR]

continue from above---

6. Final question: my project has pvc CIR type of CIR guarantee to AF31 traffic. That's why I need to have the calculation matching that actual CIR value. I plan to use bandwidth remaining percent on all my locations to standardize the QoS configuration. So my final question is assuming i have max-reserved-bandwidth 95.

T1=1536k, voice = 128k, Explicit define bandwidth for class-default.

AF31 needs to be = 1024k. So what is the bandwidth remaining percent value I should use for AF31?

Will it be 1024/[(1536 x 0.95) - 128] = 77% ??

I don't know if this is correct because Maria says bandwidth remaining percent excludes both LLQ + Default class.

In my case,

T1= 1536

LLQ= 128

AF31=1024

max-reserved=95%

So my default (explicit) will be left with (1536x0.95) -128 -1024 = 307k which is fine.

But how do I calculate "bandwidth remaining percent" for AF31 which I need it be 1024k?

policy-map x

class ef

priority 128

class af31

bandwidth remaining percent ???

class class-default

bandwidth remaining percent ??? or what other bandwidth command I can use on default class if I already use "bandwidth remaining percent on AF31? I assume all classes including default class will need to "match" the method when assigning bandwidth to each class that we can't mix and match.(ie. we can't use absolute value bandwidth 307 on default class and then use bandwidth remaining percent on AF31).

So will 1024/1331 is the correct way to provide the value for bandwidth remaining percent for af31?

If so, then what would we put in default class. Well we know default class can have upto 307k minimum guarantee but it doesn't need to use it all. But just say it does, then will it be 307/1331 = 23%? so AF31 + default = 100%

this will contradict what Maria says when she says bandwidth remaining percent excludes not only LLQ, but also default class.

So I think this question is my biggest problem to understand how to calculate it properly in above scenario to assign to correct value.

thank you again very much for your time and patience.

Joyce

[Joseph W. Doherty]

#1

"Will Default traffic gets "more than" 384k and AF41 gets less than "1024k" even when it is "bandwidth 1024" in CBWFQ for AF41?"

Yes, that's what I believe happens on most platforms that schedule class-default FQ flows with class flows.

"2. by configuring class-class default

fair-queue

or just class-class default, it makes no difference, correct? meaning whether I specify 'fair-queue' or not, I still have fair-queue on the default class, right? "

That what's documented, although on some platforms and latter IOSs, I recall show policy-map stats look different.

#3 Yes, and there might be other traffic in the set aside bandwidth than just routing updates. In fact, routing updates might be part of class-default traffic.

Try to keep in mind, you don't need to try to allocate all the bandwidth. If you only allocated 1% for class AFxx, and nothing else wanted bandwidth, usually the class could have 100% of the link. The issue is how bandwidth is allocated when more than one class, or the sum of the classes exceed the link's capacity. (You haven't made clear what you're trying to accomplish.)

#4 No, if you don't explictly allocate bandwidth for class-default, and it defaults to FQ as we believe, and if it's like most platforms, FQ might take bandwidth form AFxx even though you've allocated 95%.

#5 Will it work? Probably. But will it work well, depends on if there is any other traffic that needs some bandwidth that doesn't fall into an explictly defined class. (If there was only one class, or one class LLQ and one non-LLQ, and no other traffic, you probably wouldn't need to define the non-LLQ class.

#6 If your working with some type of CIR, and if it less than the link's full capacity, and if you only want to use it for one special class, you've working on a very complex configuration.

To truly guarantee performance, you would likely need to either shape the whole policy to not exceed CIR and then sub divide the bandwidth, or you need to insure your AF31 will not exceed CIR and possibly mark all other traffic that it can be discarded. (E.g. DE set on all non-AF31 traffic if frame-relay.)

Again remember, normal class bandwidths define a minimum, but a CIR is a logical maximum (traffic above that rate is not within agreed bandwidth and is subject to discard).

If your really wanted to insure class AF31 had 1024 of 1536, you would do this:

policy-map x

class ef

priority 128

police 128000

class af31

bandwidth (any value because other classes will leave 1024 available)

class class-default

shape 384000

"But how do I calculate "bandwidth remaining percent" for AF31 which I need it be 1024k? "

You try not, just use 1024 if that's the minimum you need to reserve. Use percentages when ratios of one class to another is acceptable.

"So I think this question is my biggest problem to understand how to calculate it properly in above scenario to assign to correct value. "

Again, I suspect you're trying to compute a percentage before understanding what CBWFQ will deliver. Also again, if you could clarify what you're trying to accomplish, I might be able to describe a method to accomplish it, but it's possible your focus on defining bandwidth exactly based on max-reserve will not get you what you want (beyond perhaps a better understanding of calculating bandwidth values).

[blackladyJR]

Joe,

Thanks again very much. On #6, what I have is a LLQ, a AF31 = CIR queue, AF32 (burstable traffic but better than default), and then a default queue.

So for a T1 interface, I have 128k LLQ, 1024k CIR = AF31, some xx for AF32, and some xx for default (default will be everything else other than AF31 and 32 such as email, http, misc applications...etc).

I have many sites and they will have various interface bandwidth and various CIR value. But they all fall into same type.

So I want to use bandwidth remaining percent as a standarized method in the AF31/32/Default class. LLQ can just use priority 128 to make it simple and clear.

So for LLQ I will use priority 128.

For AF31, I need to assignment the bandwidth remaining percent that will give me a minimum of equivalent 1024k.

For Af32 and default, there is no absolute bandwidth, but it will be good to have some reasonable guarantee minimum for each of these 2 remaining class.

Since say we use max reserved 90 so give 10% on overhead. So 90% of 1536 will be 1331k. So I have only 1331k to allocate between AF31, AF32 and Default while I must have 1024k at the minimum to reserve for AF31. So 1331-1024=307k left. So I only have 307k left for AF32 and Default so will give it half and half like 150k to each. I know i can assign even just 8k for AF32 and Default but I will feel more comfortable that I try to allocate as much as i left (which is only 307k) to those two classes so that during congestion, I know at least I have 150k will pass thru default and 150k will pass thru AF32 while AF31 will pass thru 1024k which is CIR.

Our CIR is the guarantee minimum (not cap maximum), we can burst to full interface when network has no congestion, so CIR is not the cap, it's just the minimum guarantee bandwidth that network won't drop during congestion.

So with the above bandwidth value, I guess it's easiest to define CBWFQ for AF31/32/Default with bandwidth 1024, 150, 150 then I am done. But if I want to use bandwidth remaining percent, what will be the value to use to represent this?

1024/1331 = 77%

150/1331 = 11%

So that's why I wonder if this is correct to do :

policy-map x

class ef

priority 128

class af31

bandwidth remaining percent 77

class as32

bandwidth remaining percent 11

class class-default

bandwidth remaining percent 11

int s0/0

max-reserved-bandwidth 90

service policy output x

The above seems to me is logical but it seems to completely contradict what Maria says about bandwidth remaining percent does not apply to default class as it excludes both LLQ and default class. I didn't get the clarification of whether this excludes default class under which situation (ie. is it only true when default class does not have explicit bandwidth defined, so if that is the case, the AF31 + Af32 suppose "can" add up to 100% excluding default class when default class doesn't explicit define bandwidth), so when we do define explicit bandwidth (in our case, we use bandwidh remaining percent on default class, then when we do this explicit, then the bandwidth remaining percent is NOT excluded from default class anymore), not sure if this is true.

thanks,

Joyce

[Joseph W. Doherty · Accepted solution]

Got it.

From what you describe, I believe you would be much better using an explict bandwidth rather than percentages since this avoids both the original calculations and helps avoid confusion for anyone looking at it later. I.e., if CIR for AF31 is 1024 on a T1, then define its bandwidth as 1024 and insure either there's no significant volume of traffic in class-default FQ (on most platforms) or define an explict bandwidth for class-default.

I also advise, if possible, to avoid changing max-reserve since this type of the change often seems to lead to unpleasant surprises latter on.

With the cautions out of the way, I believe you calculate remaining percent as follows:

Total remaining percent is max-reserve % of link less LLQ allocations.

e.g.

using your 90% max, that allows (about) 1382 (1536 * .9) to be defined (Not "So 90% of 1536 will be 1331k." - unless my calculator is wrong). Subtract out all LLQ allocations, in your case 128, which leaves 1254. 1024 is (about) 82% of this, and that would be the value you want for AF31 remaining percent. The remaining 18% can be allocated as you chose, such a 9% for AF32 and class-default.

[blackladyJR]

THANKS!!! Since I know there are always a lot of default traffic such as internet traffic that is unpredicatable, so I think from your advice, I am safer to go with defining default class explicit rather than FQ.

Yes I forgot how 1331 comes from but you are right, it's 1382 :)

And I understand I don't necessary allocate 9% to each remaining class, I can put 5% on each so it doesn't have to add up to be 100%, correct? (when using bandwidth remaining percent), I wasn't sure where I read in the past that somehwere it says it needs to add up 100% when using this but I can't find it so my memory is certainly not reliable. I trust you :)

As you the QoS Expert, my final question is that how does "weight" determine in the router? This is CBWFQ, so each class has different weights and I assume weights means different priority when scheduling the packets from various queue onto the outgoing interface.

I wonder is it Cisco router already has some default behavior so AF4x has higher weight than AF3x and so on. So say I have 100 packets in AF3x queues waiting to be served, then at this moment, I have 20 packets coming into AF4x queue, will router then do "round robin" to serve ALL the classes (except LLQ which we know is highest priority and will cut line and will serve ALL its packets before back to the CBWFQ queues). So now the router will round robin to AF4x and will server perhaps 10 packets in AF4x, and serve 8 packets in AF3x, and 4 packets in AF2x, 1 packet in Default class and round robin again? So my question is is this the behavior so each CB classes get its turn but the higher class gets more serving as it has higher weights?

If the above is true, then how does weight determine? Does it has "anything" to do with the bandwidth value that we assign on each class? So if I assign 60% on AF31 and 30% on AF41 let's say, will router treat AF31 a 2:1 ratio higher weight in AF31 instead of AF41? Or will router totally ignore the bandwidth so even thought AF31 has 60%, the router will still have higher weight in AF41 just based on Dscp value and will serve more on AF41 than AF31 regardless of bandwidth value?

I promise this is my last question (for now :)

thanks again.

Joyce

[Joseph W. Doherty]

When allocating class bandwidths, you never need to allocate the maximum possible, but when allocating, you can't exceed the maximum reserved. (E.g. your remaining percent doesn't need to sum to 100, but sum can't exceed 100.)

Weight is what, I believe, the scheduler actually uses to determine how much bandwidth a queue will obtain. When we set bandwidth on classes, this defines a weight on that class's packets.

WFQ does use a packet's IP precedence to adjust a packet's weight. I believe on many platforms, fair-queue within CBWFQ class-default, is WFQ.

With regard to the QoS packet scheduler dequeuing packets, it's not just counting packets, but, I believe, counting bytes.

In your question about assiging AF31 60% and AF41 30%, the scheduler should attempt to dequeue twice the bytes for AF31 relative to AF41 if in unique CBWFQ class queues. If the AF31 and AF41 were in WFQ, you don't get to assign bandwidths but AF41 would get more that AF31. (For WFQ flows, it's possible to calculate what the bandwidth allocations would be - see Cisco documentation on WFQ - you might also want to read Cisco documentation on custom queuing.)

[blackladyJR]

Hi Joe,

Right, I recalled WFQ use IP Precedence so there is "priority" to serve higher class so TOS 4 has higher priority than TOS 3.

But here is from the congestion mgmt overview doc on cisco:

"For CBWFQ, the weight for a packet belonging to a specific class is derived from the bandwidth you assigned to the class when you configured it. Therefore, the bandwidth assigned to the packets of a class determines the order in which packets are sent. All packets are serviced fairly based on weight; no class of packets may be granted strict priority."

From these statement, it makes me think that when we do CBWFQ, the router does not care DSCP or TOF value at all when it comes to "dequeue" and "serve" the packets going out. It looks like the 'weight' is related to ' bandwidth' so if I have 60% AF31 and 30% AF41 in the bandwidth assignment, then the router will serve 2:1 ratio round robin to serve twice the packet/bytes in AF31 than AF41 at each round. So that makes AF41 does not have any "better" treatment than AF31 at all.

So that means if we have 30% AF31 and 70% Default let's say, then that means Default traffic gets serve more at each round than AF31.

It just doesn't seem to make sense because the router doesn't care the "priority" or importance of DSCP order while a regular WFQ will treat weight based on TOS.

What do you think?

I mean sometimes you have high priority application so you put them in AF31 let's say, but they can be very important but they may not need large bandwidth and rest may be default internet low priority traffic. But since the bandwidth in default is say 5 times the AF31, then the router dequeue 5 times more packets than the AF31? Just seem wrong.....

[Joseph W. Doherty]

The documentation is correct when referring to CBWFQ single FIFO queue classes. However, I believe "fair-queue" within most platforms class-default is WFQ, i.e. class-default FQ flows may have different weights assiged based on IP Precedence. (I've yet to figure out how those flow weights are assigned relative to other class queues. I've done some simple testing and I recall what I saw wasn't obvious.)

If you explictly assign class-default 70%, it's a FIFO queue like other class queues. If you've only assigned 30% to AF31 leaving 70% undefined, don't assume class-default FQ gets a weight for 70%. Again, the default 25% is reserved such that you can not explictly reserve that bandwidth, although class-default is able to use that bandwidth, but that bandwidth is not allocated just for class-default. In other words, don't equate the set aside bandwidth as being the same as class-default.

In your comment where if you consider AF31 5x more important than default Internet traffic, you need to classify both into classes such that the AF31 class has 5x the bandwidth allocated to it the Internet traffic class. (Also again, don't assume queues are dequeued by packet count, they should be dequeued based on packets' byte count. E.g. 5 packets dequeued sized 200 should count as equal to one packet dequeued sized 1,000.)

[blackladyJR]

so pretty much the router when dealing with CBWFQ does not care the DSCP or IP Precedence at all. AF41 does not mean to have any 'weights" to serve more packet bytes by the DSCP value. The only thing that controls the 'weight" is the 'bandwidth" value then. So if i have important application say in AF41 but even though it is "important", but it doesn't have many users for that application. While I have many users using other application in AF31 and default. So if I assign each class based on the bandwidth usage to give AF31 and default more bandwidth than AF41. Then AF41 althought the application is most criticial, it will not get any better treatment, in fact AF31 and Default will get to serve /dequeue more packets bytes than AF41 at each round.

got it!

Thank you very much once again.

Bye.

[Joseph W. Doherty]

Again, standard CBWFQ FIFO classes don't implicitly consider DSCP marking for dequeuing weight. Since you define the bandwidth for these classes, which indirectly assigns weights, you have precise control. Also keep in mind, strictly speaking, no AFx class is more imporant than another, they represent service classes. It's up to you to define the service for the AFx classes. It's not a question of one class of traffic is more critical than another, but rather whether defined classes obtain the resources they need to work as required. Recall, non AFx traffic often falls within best effort (BE), i.e. no guarantee beyond we'll try.

[milan.kulik]

Hi Joseph,

ad "how those flow weights are assigned relative to other class queues", I just found a very interesting article:

http://blog.internetworkexpert.com/2008/08/17/insights-on-cbwfq/

This seems to explain a lot!!!

BR,

Milan

[blackladyJR]

Hi Milan,

This is really a very useful document!!! I read it and it really explains a lot!

Thanks for sharing.

[Joseph W. Doherty]

Indeed, very interesting! However, keep in mind any time you start to use undocumented information, vendor could change how it works.

One thing missing, author doesn't make clear how multiple class-default FQ flows also compete for bandwidth since only one class-default flow is shown. So I don't agree with ". . . we may conclude that user-defined classes dominate dynamic flows almost all the time, unless they have small shares of bandwidth configured." except at a per flow level, not at the class level.

"If you want the scheduler to honor class-default traffic, assign it an explicit bandwidth value. ", this is something I often emphasis for most platforms, but not all platforms.

Still, again, interesting read.