CBWFQ/LLQ Bandwidth allocation question, please help

Answered Question
Jan 23rd, 2009

Hello,

After reading multiple document, I have left with many question marks in my head. Can you please help to clarify?

I know by default total bandwidth is 75% for all the class defined in the policy map including voice and data (i.e including the bandwidth from LLQ). So here is the example:

Say we stay with the default so 25% belongs to default class and say we have a T1 serial so bandwidth is 1536 on the interface.

Now, 25% of 1536 is 384k. So we are left with 1152k to work with for all the voice and AFxx I assume. Say we have 128k for voice in LLQ. So that left with 1024k for all the AFxx class to use and I assume I just have one AF41 to use all the bandwidth

policy-map test

class EF

set ip dscp ef

priority 128

class AF41

set ip dscp af41

bandwidth 1024

class class-default

So i assume that will be the policy-map. If I want to define the exact above via bandwidth percent, will it be like this?

policy-map test

class EF

set ip dscp ef

priority 128

class AF41

set ip dscp af41

bandwidth percent 67

class class-default

So I am using bandwidth percent so it is the absolute percent to the interface so 1024k of 1536 will be 66.7%. So the default class still will be 25% with 384k, correct? The voice is 8.3% so when I add LLQ + AF41 + BE = 8.3+66.7+25=100% of 1536.

How about if i use bandwidth remaining percent? how would that change?

policy-map test

class EF

set ip dscp ef

priority 128

class AF41

set ip dscp af41

bandwidth remaining percent 73

class class-default

why 73? if the remaining percent is relative term to the bandwidth left after LLQ. Then we have 1536-128=1408 left for AF41 and Default class. So 1024k/1408k = 72.7% while the same 384k Default will be 27.2% in the bandwidth remaining percent perspective.

So is it correct that Default class is 25% in bandwidth percent (absolute) and it is 27.2% under bandwidth remaining percent with 128k LLQ?

So when I use bandwidth percent, when I add up the percent value of AF41 + Default is not equal to 100%. It will be 100% if I add LLQ + AF41 + Default.

but when I use bandwidth remaining percent, then the % value will be 100% when I add AF41 + Default because it is relative term of the remaining bandwidth after LLQ. So 73%+27% in AF41+BE under bandwidth remaining = 100% of 1408k.

I have this problem too.
0 votes
Correct Answer by Joseph W. Doherty about 7 years 10 months ago

Got it.

From what you describe, I believe you would be much better using an explict bandwidth rather than percentages since this avoids both the original calculations and helps avoid confusion for anyone looking at it later. I.e., if CIR for AF31 is 1024 on a T1, then define its bandwidth as 1024 and insure either there's no significant volume of traffic in class-default FQ (on most platforms) or define an explict bandwidth for class-default.

I also advise, if possible, to avoid changing max-reserve since this type of the change often seems to lead to unpleasant surprises latter on.

With the cautions out of the way, I believe you calculate remaining percent as follows:

Total remaining percent is max-reserve % of link less LLQ allocations.

e.g.

using your 90% max, that allows (about) 1382 (1536 * .9) to be defined (Not "So 90% of 1536 will be 1331k." - unless my calculator is wrong). Subtract out all LLQ allocations, in your case 128, which leaves 1254. 1024 is (about) 82% of this, and that would be the value you want for AF31 remaining percent. The remaining 18% can be allocated as you chose, such a 9% for AF32 and class-default.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.7 (6 ratings)
Loading.
marikakis Fri, 01/23/2009 - 09:09

Hello,

The class-default gets 25% of configured bandwidth (using 'bandwidth' command) by default in both cases.

When you use the 'percent' option, all bandwidth values of all other classes can sum up to the 75% of configured bandwidth. The 75% includes any bandwidth assigned to any LLQ. So, to find out how much you can assign to non-LLQ classes you subtract the LLQ bandwidth from the 0.75*bandwidth, and what is left can be assigned to other classes. The percentages for those non-LLQ classes are expressed as a percent of the 'bandwidth' setting of the interface. But, because you subtract LLQ assignment, the percents configured can only sum up to 75% of total bandwidth only in the case where you have no LLQs.

Because the above described calculations are a bit of a headache, the 'remaining percent' can act as a pain killer. When you use the 'remaining percent' option you say 'OK, class-default you can get your default. And you LLQ, you can get your guarantee. Whetever remained I won't calculate it. I will just say what percent of the remaining bandwidth can be assigned to the rest of the classes'. This means that now we are considering percentages not of the configured bandwidth, but of the remaining, after we subtract class-default and LLQ. Because those are percentages of the remaining bandwidth, the percentage values can add up to 100%, but 100% of the remaining bandwidth (not the total bandwidth available).

This basically means that you should have also subtracted class-default (not only LLQ) to find out how much the remaining bandwidth actually is. Also, note that when we say that class-default gets 25% by default, we mean 25% of the total configured bandwidth.

Kind Regards,

M.

blackladyJR Fri, 01/23/2009 - 09:47

Hello,

Thank you for your reply. So if I understand you correctly, the remaining percent command does not include Default class value. So 100% is just all the AFxx.

To use my original example to clarify,

we have Interface at 1536k, using cisco default 75% rule. LLQ=128. 1024k=AF41, 384k=Default, my policy-map should be like this:

policy-map test

class EF

set ip dscp ef

priority 128

class AF41

set ip dscp af41

bandwidth remaining percent 100

class class-default

Is this correct? So the Default class does not participate in the bandwidth remaining percent value at all where 100 is correct to use instead of 73.

I can't find anywhere in the website to mention the remaining percent exclude the default class. It clearly states excluding the LLQ though.

thank again,

Joyce

marikakis Fri, 01/23/2009 - 10:42

Hello,

Yes, this is correct and has an inherent simplicity. I read this on the Cisco QOS exam certification guide while I was preparing to take the QOS exam. This book can help you (with the examples contained therin) clarify many QOS-related issues that are not explained very well in other places.

You are right that excluding class-default is not explicitly mentioned. It is rather implied. I mean, with the 'percent' option you subtract class-default from total bandwidth and the remaining 75% includes LLQ bandwidth. The 'remaining percent' option further excludes the LLQ (that's the new thing), so we are clean to go and calculate bandwidth assignments for other classes without requiring to think about LLQ and class-default.

Kind Regards,

M.

Joseph W. Doherty Fri, 01/23/2009 - 11:07

BTW:

I think a bigger advantage of using remaining percent, less chance the policy failing on an interface if someone changes max reserved.

Also, on newer IOSs, you can use a percentage for LLQ's priority statement. (Also newer IOSs support CBWFQ percentage based shapers/policiers within classes, too.)

blackladyJR Fri, 01/23/2009 - 11:22

Hi Maria,

Thank you very much. May I have the name of the book that you mentioned that has good QoS examples?

From our discussion, bandwidth remaining percent is only applicable to the AFxx class that excludes LLQ and Default. However, when I am configuring the default class, it has the bandwidth remaining percent "available" as the command to enter. So would this be odd then if this is not supposed to apply to the default class but yet the command is available?

marikakis Fri, 01/23/2009 - 11:32

Hi Joyce,

I am not sure. This might be a hardware/interface specific issue (QOS is sometimes very hardware specific despite the abstractions). I am thinking this because I recall reading somewhere that the max-reserved-bandwidth command (which is to blame for the 75%-25% rule) is not supported in some cases (I think it was frame-relay). What you mentioned would make sense in the case where no bandwidth is implicitly assigned to class-default.

Joseph, thanks for participating and good point you made. I hope you or somebody else can go further with this case, because I have to get prepared for my Networkers in Barcelona trip and I am running out of time :-)

Kind Regards,

Maria

[Edit] p.s. I forgot to say about the book:

"Cisco QOS Exam Certification Guide", Second Edition, by Wendel Odom and Michael J.Cavanaugh

Joseph W. Doherty Fri, 01/23/2009 - 11:49

The class-default is a special case. When you explicitly define bandwidth it behaves much as the other classes do.

Maria first post's statement "The class-default gets 25% of configured bandwidth (using 'bandwidth' command) by default in both cases. " is a bit misleading, it's a bit more complex. From Cisco (http://www.cisco.com/en/US/docs/ios/12_2/qos/configuration/guide/qcfconmg_ps1835_TSD_Products_Configuration_Guide_Chapter.html):

"The sum of all bandwidth allocation on an interface cannot exceed 75 percent of the total available interface bandwidth. The remaining 25 percent is used for other overhead, including Layer 2 overhead, routing traffic, and best-effort traffic. Bandwidth for the CBWFQ class-default class, for instance, is taken from the remaining 25 percent. However, under aggressive circumstances in which you want to configure more than 75 percent of the interface bandwidth to classes, you can override the 75 percent maximum sum allocated to all classes or flows using the max-reserved-bandwidth command. If you want to override the default 75 percent, exercise caution and ensure that you allow enough remaining bandwidth to support best-effort and control traffic, and Layer 2 overhead. "

Although it addresses ATM interfaces, you might also want to read this whitepaper: http://www.cisco.com/en/US/tech/tk39/tk48/technologies_tech_note09186a00800fe2c1.shtml

In the foregoing paper, take special note on how class-default FQ is scheduled on most platforms (which is why I believe using FQ with non-LLQ classes on those platforms doesn't preserve those class's minimum bandwidth guarantees).

blackladyJR Fri, 01/23/2009 - 12:50

Hi Joe,

Hm...now I am back to step one. In fact, I have read those two document prior posting the original question.

So I am now confused about whether I should include or exclude Default class in the bandwidth remaining percent of the 100% calculation.

So is 73 correct or 100 correct in this scenario:

Interface 1536, cisco default 75%, LLQ 128k, AF41 use all available which is 1024k, Default class 384k (which is 25% of 1536).

So in my policy-map, should the AF41 class be defined with bandwidth remaining percent 73 or 100?

And you have mentioned class-default is special case and different when it explicit define bandwidth. Can you explain the difference when it explicit define bandwidth vs not explicit define bandwidth using my scenario bandwidth value above?

thank you so much for your assistance.

Joyce

Joseph W. Doherty Fri, 01/23/2009 - 17:41

"So in my policy-map, should the AF41 class be defined with bandwidth remaining percent 73 or 100? "

100% (if using remaining)

"And you have mentioned class-default is special case and different when it explicit define bandwidth. Can you explain the difference when it explicit define bandwidth vs not explicit define bandwidth using my scenario bandwidth value above?"

If you define class-default with an explict bandwidth, you would treat it as aother class.

From other posts of mine (made today and tested on a 2811 running 12.4):

With default max reserved:

policies that should be "legal":

policy-map x

class x

bandwidth percent 75

or

policy-map x

class x

bandwidth percent 75

class class-default

or

policy-map x

class x

bandwidth percent 75

class class-default

fair-queue

or

policy-map x

class x

bandwidth percent 74

class class-default

bandwidth percent 1

or

policy-map x

class class-default

bandwidth percent 75

or

policy-map x

class x

bandwidth percent 1

class class-default

bandwidth percent 74

The prior all use percentages, but hopefully they show how the bandwidth can be partitioned between classes especially when using class-default in different ways. Remaining percents (not tested) should, I believe, all classes can sum to 100 rather then max reserved (default 75%). Router should not accept policy that oversubscribes bandwidth reservation regardless of whether they're explict in Kbps, or percentages.

blackladyJR Mon, 01/26/2009 - 10:45

Hi Joe,

Thank you for your help. I still have a little bit not sure.

For bandwidth remaining percent, the bandwidth remaining needs to add up to be 100% (excluding both LLQ and Default).

So Scenario A:

policy-map x

class ef

priority 128

class af41

bandwidth remaining percent 100

class class-default

So is it correct that my default class has 384kbps in above configuration? i.e. 1536k interface value, 128k LLQ, 25% of 1536k = 384k.

So My LLQ = 128k, my AF41 = 1024k, my Default = 384k

Question:

1. is above correct?

2. If somehow I want to explicit define default class bandwidth, what should I use?

choice: a) bandwidth 384?

b) bandwidth percent 25?

c) bandwidth remaining percent (not valid)?

Is 2(a) and 2(b) are both correct? Or will both are not possible that we can't mix and match that we already use "remaining" in AF41 so we can't use 2(a) or 2(b) in the default class? Is 2(c) an invalid choice (i.e. default class should never configured with bandwidth remaining percent since it is excluded from this command calculation?

Scenario B:

If I want to use bandwidth percent, should it look like this?

policy-map x

class ef

priority 128

class af41

bandwidth percent 66

class class-default

OR

policy-map x

class ef

priority 128

class af41

bandwidth percent 66

class class-default

bandwidth percent 25

Will these two policies produce the same kbps? (ie. af41 = 1024k, default = 384k)?

From your above one of the policy:

policy-map x

class x

bandwidth percent 74

class class-default

bandwidth percent 1

I don't understand this one, as if we defined default class with 1 percent. Then where is the 25% go? will that means the default class gets 26% (25+1)? This policy map assumes there is no LLQ, just one class and one default class.

thank you so much again for all your help.

Joyce

Joseph W. Doherty Mon, 01/26/2009 - 11:15

"For bandwidth remaining percent, the bandwidth remaining needs to add up to be 100% (excluding both LLQ and Default)."

Needs, no, but the sum of all classes using remaining percent can not exceed 100%.

(Scenario A) "So is it correct that my default class has 384kbps in above configuration? i.e. 1536k interface value, 128k LLQ, 25% of 1536k = 384k."

I don't believe that's correct for two reasons. First, class-default without an explict bandwidth statement counts on the bandwidth that's been set aside by max-reserved (default 25%). Unsure, and perhaps unlikely, this means the class class-default is guaranteed that bandwidth. Second, FQ within class-default precludes, I believe, class bandwidth guarantees except for LLQ.

(Scenario B) "Will these two policies produce the same kbps? (ie. af41 = 1024k, default = 384k)? "

No, they are very different I beleive. First, when you use an explicit bandwidth statement with class-default it becomes FIFO. Second, I believe it now allocates its bandwidth statement within the same context as other explicit class bandwidth assignments.

"I don't understand this one, as if we defined default class with 1 percent. Then where is the 25% go? will that means the default class gets 26% (25+1)? This policy map assumes there is no LLQ, just one class and one default class. "

The (default) set aside 25% bandwidth is available to both class x and class class-default, and, I believe, other CBWFQ internal queues. (If you look at queue conversation numbers for CBWFQ, you'll find not all are assigned to user defined classes or, I believe, class-default FQ queues.)

blackladyJR Fri, 01/30/2009 - 15:09

Joe,

First of all, very appreaciate all your help and I will sure rate on your post :)

Sorry for taking so long to reply because I was so confused and trying to digest your answers and went to read more before reply.

I think I have a wrong understanding all along, now that after reading your most recent post, here are my questions to see if these are correct?

I try to post all my questions and it runs our of character, so I split the message in two:

Under Cisco standard 75% rule as our assumption:

1. If I don't explicit define "bandwidth" in default class, then there will be 25% of the interface reserved for routing overhead + "default class" to use.

So if we use the same e.g. T1 = interface, 128k = voice.

policy-map x

class ef

priority 128

class af41

bandwidth 1024

class class-default

fair-queue

So for not explicit define bandwidth in default class, then we should have 25% x 1536 = 384k for default class.

so that's how we leave with 1024k for AF41 to use. (1536k - 384k - 128k = 1024k).

Is this correct?

2. Now if I explicit define bandwidth for default class, does that mean now I no longer have that 25% for default class anymore. Still staying at cisco default 75%, does that mean now if I want to explicit define bandwidth for default class, then I need to use part of the 1024k that was previously all for AF41, now it needs to share with default class?

policy-map x

class ef

priority 128

class af41

bandwidth 640

class class-default

bandwidth 384

So that means my AF41 only has 640k left if I still want to have default class to have 384k. That means there is that 25% hidden 384k is now used for routing overhead ONLY. The default class no longer can tap into that 25% to use but instead tap into the AF41 bandwidth to split it.

Is this correct?

3. If #2 is correct, that mean if I ever want to explicit define default class, then I should change the cisco default and change it to like 95%, so just leave 5% of those routing overhead. So in our e.g , now it becomes:

1536 x 0.95 = 1460

voice = 128

so AF41 + Default = 1460 - 128 = 1332

So if I still give 384k to Default, then my AF41 now has 948k instead of 640k in case #2 and instead of 1024k in case #1.

Is this correct?

Question 4 is a separate message....

Joseph W. Doherty Fri, 01/30/2009 - 18:59

#1 yes (or at least that's my understanding)

#2 yes, class af41 and class-default would split the reservable 75% (1024), and 25% is still set aside, but, no, it's not just for routing overhead.  The 25% could be used by class af41 or class-default if not otherwise being used.  (Normal class bandwidths insure a minimum they don't cap.)

#3 no, because again, other classes can usually use available bandwidth.  For instance given:

policy-map x

class ef

priority 128

class af41

bandwidth 8

class class-default

bandwidth 8

And still assuming a T-1, any class could use the whole link's bandwidth if it's available.

blackladyJR Fri, 01/30/2009 - 15:10

Second half of the message starting question 4....

4. If #3 is correct, then how do I configure the policy to reflect to above value in CASE #3 e.g. by trying all 3 methods?

4a) policy-map x

class ef

priority 128

class af41

bandwidth 948

class class-default

bandwidth 384

4b) policy-map x

class ef

priority 128

class af41

bandwidth percent 62

class class-default

bandwidth percent 25

(Note: 62% = 948/1536 and 128k voice is equivalent to 8% 128/1536. So when we add them together we get 8%+62%+25% = 95% which is the max reserved bandwidth) so 5% is the overhead that we don't see in the policy.

4c) policy-map x

class ef

priority 128

class af41

bandwidth remaining percent 71

class class-default

bandwidth remaining percent 29

(note: Case #4 is when we want to explicit define bandwidth for default class and also changes max reserved to 95%.

I remembered Maria says "bandwidth remaining percent exludes LLQ and default", will this statement only true when we don't explicit define bandwidth for default class?

that means if I am explicit to define default class to give 384k to that class, then will 4C be correct as 384k/1332k = 29%.

1332k = (1536k X 0.95) - LLQ 128 = 1332k

So 1332k is the remaining bandwidth for both AF41 and Default to use when we "explicit" define default class.

So in this case AF41 + Default = 100% if I want 948k AF41 and 384k Default so 71% and 29%.

is this correct?

4d) Now if we don't explicit define default, and if I have the max reserve still set to 95%, then that means my default now will NOT be 384k anymore, it will be that 5% along with routing overhead.

So that means the entire 1332k is available for AF41 now.

so the policy will be:

policy-map x

class ef

priority 128

class af41

bandwidth remaining percent 100

class class-default

fair-queue

So this 4d), if I define this way with max reserved 95%, will I yield 1332k for AF41 and 5% of 1536 for default class + routing overhead?

I think I have all possible scenario to try to understand the way to configure via all 3 methods and also whether we explicit or not explicit define bandwidth for default class.

I hope all the above is correct but if you find anything that is wrong, please let me know.

Again,

Very appreciated all your time to spend with me.

Thanks a lot,

Joyce

Joseph W. Doherty Fri, 01/30/2009 - 19:28

I'm afraid I've lost my "score card" with all the question 4 variations.

With 4d, whenever you use FQ within class-default on most platforms, it disturbs defined class bandwidth calculations because on many platforms every class-default flow competes for bandwidth with defined classes.  In other words, class af41 might not obtain its minimum bandwidth guarantee (although it may obtain more).

With your other #4 questions, try to keep in mind CBWFQ attempts to guarantee minimum bandwidth, not a fixed slice of bandwidth.  Again, that's why the class bandwidth sums don't have to hit 100% yet they can not allocate more than 100%.

Further, CBWFQ classes, by default, preclude explicitly reserving more than 75%, to allow some bandwidth for other stuff including class-default when it's bandwidth isn't explicitly defined.

In other words, given a T-1, instead of trying to insure each class gets its maximum share of bandwidth, you use CBWFQ to insure it gets the minimum it needs to work well.  If class af41 only needed 256 Kbps to work well, you define:

policy-map x

class ef

priority 128

class af41

bandwidth 256

class class-default

bandwidth 8 (think 8 is minimum parameter)

This policy could be used on any interface that supports the policy and it would guarantee class af41 should also obtain at least 256 Kbps.  You don't need to try to figure out how to slice off the maximum amount of bandwidth that might be available to the class.

If you're still trying to get a handle on possible bandwidth allocations, I can try to explain that better, but I'm trying to convey you might be missing how you actually want to use CBWFQ, and normal usage doesn't require such computations.

Further if you think in relative bandwidth ratios, e.g. I want class af41 to have a minimum of half the link's bandwidth, you just use bandwidth percentage statements (which also allows the same policy to be used on interfaces with different amounts of bandwidth).

blackladyJR Mon, 02/02/2009 - 09:59

Joe,

Once again, thank you so much for your patience :)

Yes I do want to understand better on bandwidth allocation because I really need that for project QoS work that we need to try to figure out the "max" of each class we can have.

First of all, thank you for clarifying the main purpose of CBWFQ is to two things : 1) guarantee minimum bandwidth for each class as defined during congestion 2) NOT cap maximum bandwidth while every class can "burst" more than their "minimum" defined bandwidth when it's available. (while no congestion).

So now I understand that we don't need to define the bandwidth to add up 100%, just don't define more than 100%.

For my project, I need to understand how to slice my interface to full maximum so during congestion, I know what each class (include default) will guarantee as a minimum will get so will maximize each class to its full use. Having said that, here are my revised questions:

1. You have mentioned when Default class is in FQ on most platform, it will compete with the class in CBWFQ so the class in CBWFQ won't get their minimum guarantee bandwidth. What do you mean by this?

So if we use the same e.g. T1 = interface, 128k = voice. cisco default 75%

policy-map x

class ef

priority 128

class af41

bandwidth 1024

class class-default

fair-queue

With cisco default and not explicit define default class, that means default class with routing overhead will get 25% of T1, so that's 384k. But you said FQ in class-default will compete with CBWFQ, so that means if I have more than 384k worth of default class traffic, and I have more than 1024k worth of AF41 traffic "during congestion", what will happen? Will Default traffic gets "more than" 384k and AF41 gets less than "1024k" even when it is "bandwidth 1024" in CBWFQ for AF41? I mean any QoS is really not relevant during "no congestion", so I am interested to know what will happen during congestion when QoS is really in use when I have more than 1024k AF41 traffic and more than 384k default class traffic.

2. by configuring class-class default

fair-queue

or just class-class default, it makes no difference, correct? meaning whether I specify 'fair-queue' or not, I still have fair-queue on the default class, right?

3. So for my project where I really want to guarantee some minimum bandwidth for my default class while maximizing other AFxx class guaranteed bandwidth. Under this objective, that means I want to explicit define class-default with bandwidth command. So the class-default will fall under the 75% bandwidth during congestion. So in this case I would set max-reserved-bandwidth 95 so I can define all my AFxx + class-default within 95% of bandwidth to maximize it while leaving 5% for routing overhead. Is it true that the 5% will be used by routing overhead when there is routing overhead traffic passing?

4. If I don't explicit define class-default with bandwidth, then that means class-default traffic will goto the 5% during congestion if all 95% are allocated to AFxx max out, if that true?

5. I saw someone do max-reserved-bandwidth 100, so what does that mean? We don't set aside any bandwidth for routing overhead at all? (this is assuming we will explicit define bandwidth for class-default so it doesn't fall to the "set-aside" pile which in this case we don't even have anything to set aside. So with max-reserved-bandwidth as 100% and class-default do have explicit bandwidth command, so where will the routing overhead go? I mean is it bad to set 100 and not set aside anything? Will this work?

final question -- continue on next message

blackladyJR Mon, 02/02/2009 - 09:59

continue from above---

6. Final question: my project has pvc CIR type of CIR guarantee to AF31 traffic. That's why I need to have the calculation matching that actual CIR value. I plan to use bandwidth remaining percent on all my locations to standardize the QoS configuration. So my final question is assuming i have max-reserved-bandwidth 95.

T1=1536k, voice = 128k, Explicit define bandwidth for class-default.

AF31 needs to be = 1024k. So what is the bandwidth remaining percent value I should use for AF31?

Will it be 1024/[(1536 x 0.95) - 128] = 77% ??

I don't know if this is correct because Maria says bandwidth remaining percent excludes both LLQ + Default class.

In my case,

T1= 1536

LLQ= 128

AF31=1024

max-reserved=95%

So my default (explicit) will be left with (1536x0.95) -128 -1024 = 307k which is fine.

But how do I calculate "bandwidth remaining percent" for AF31 which I need it be 1024k?

policy-map x

class ef

priority 128

class af31

bandwidth remaining percent ???

class class-default

bandwidth remaining percent ??? or what other bandwidth command I can use on default class if I already use "bandwidth remaining percent on AF31? I assume all classes including default class will need to "match" the method when assigning bandwidth to each class that we can't mix and match.(ie. we can't use absolute value bandwidth 307 on default class and then use bandwidth remaining percent on AF31).

So will 1024/1331 is the correct way to provide the value for bandwidth remaining percent for af31?

If so, then what would we put in default class. Well we know default class can have upto 307k minimum guarantee but it doesn't need to use it all. But just say it does, then will it be 307/1331 = 23%? so AF31 + default = 100%

this will contradict what Maria says when she says bandwidth remaining percent excludes not only LLQ, but also default class.

So I think this question is my biggest problem to understand how to calculate it properly in above scenario to assign to correct value.

thank you again very much for your time and patience.

Joyce

Joseph W. Doherty Mon, 02/02/2009 - 16:49

#1

"Will Default traffic gets "more than" 384k and AF41 gets less than "1024k" even when it is "bandwidth 1024" in CBWFQ for AF41?"

Yes, that's what I believe happens on most platforms that schedule class-default FQ flows with class flows.

"2. by configuring class-class default

fair-queue

or just class-class default, it makes no difference, correct? meaning whether I specify 'fair-queue' or not, I still have fair-queue on the default class, right? "

That what's documented, although on some platforms and latter IOSs, I recall show policy-map stats look different.

#3 Yes, and there might be other traffic in the set aside bandwidth than just routing updates. In fact, routing updates might be part of class-default traffic.

Try to keep in mind, you don't need to try to allocate all the bandwidth. If you only allocated 1% for class AFxx, and nothing else wanted bandwidth, usually the class could have 100% of the link. The issue is how bandwidth is allocated when more than one class, or the sum of the classes exceed the link's capacity. (You haven't made clear what you're trying to accomplish.)

#4 No, if you don't explictly allocate bandwidth for class-default, and it defaults to FQ as we believe, and if it's like most platforms, FQ might take bandwidth form AFxx even though you've allocated 95%.

#5 Will it work? Probably. But will it work well, depends on if there is any other traffic that needs some bandwidth that doesn't fall into an explictly defined class. (If there was only one class, or one class LLQ and one non-LLQ, and no other traffic, you probably wouldn't need to define the non-LLQ class.

#6 If your working with some type of CIR, and if it less than the link's full capacity, and if you only want to use it for one special class, you've working on a very complex configuration.

To truly guarantee performance, you would likely need to either shape the whole policy to not exceed CIR and then sub divide the bandwidth, or you need to insure your AF31 will not exceed CIR and possibly mark all other traffic that it can be discarded. (E.g. DE set on all non-AF31 traffic if frame-relay.)

Again remember, normal class bandwidths define a minimum, but a CIR is a logical maximum (traffic above that rate is not within agreed bandwidth and is subject to discard).

If your really wanted to insure class AF31 had 1024 of 1536, you would do this:

policy-map x

class ef

priority 128

police 128000

class af31

bandwidth (any value because other classes will leave 1024 available)

class class-default

shape 384000

"But how do I calculate "bandwidth remaining percent" for AF31 which I need it be 1024k? "

You try not, just use 1024 if that's the minimum you need to reserve. Use percentages when ratios of one class to another is acceptable.

"So I think this question is my biggest problem to understand how to calculate it properly in above scenario to assign to correct value. "

Again, I suspect you're trying to compute a percentage before understanding what CBWFQ will deliver. Also again, if you could clarify what you're trying to accomplish, I might be able to describe a method to accomplish it, but it's possible your focus on defining bandwidth exactly based on max-reserve will not get you what you want (beyond perhaps a better understanding of calculating bandwidth values).

blackladyJR Mon, 02/02/2009 - 19:37

Joe,

Thanks again very much. On #6, what I have is a LLQ, a AF31 = CIR queue, AF32 (burstable traffic but better than default), and then a default queue.

So for a T1 interface, I have 128k LLQ, 1024k CIR = AF31, some xx for AF32, and some xx for default (default will be everything else other than AF31 and 32 such as email, http, misc applications...etc).

I have many sites and they will have various interface bandwidth and various CIR value. But they all fall into same type.

So I want to use bandwidth remaining percent as a standarized method in the AF31/32/Default class. LLQ can just use priority 128 to make it simple and clear.

So for LLQ I will use priority 128.

For AF31, I need to assignment the bandwidth remaining percent that will give me a minimum of equivalent 1024k.

For Af32 and default, there is no absolute bandwidth, but it will be good to have some reasonable guarantee minimum for each of these 2 remaining class.

Since say we use max reserved 90 so give 10% on overhead. So 90% of 1536 will be 1331k. So I have only 1331k to allocate between AF31, AF32 and Default while I must have 1024k at the minimum to reserve for AF31. So 1331-1024=307k left. So I only have 307k left for AF32 and Default so will give it half and half like 150k to each. I know i can assign even just 8k for AF32 and Default but I will feel more comfortable that I try to allocate as much as i left (which is only 307k) to those two classes so that during congestion, I know at least I have 150k will pass thru default and 150k will pass thru AF32 while AF31 will pass thru 1024k which is CIR.

Our CIR is the guarantee minimum (not cap maximum), we can burst to full interface when network has no congestion, so CIR is not the cap, it's just the minimum guarantee bandwidth that network won't drop during congestion.

So with the above bandwidth value, I guess it's easiest to define CBWFQ for AF31/32/Default with bandwidth 1024, 150, 150 then I am done. But if I want to use bandwidth remaining percent, what will be the value to use to represent this?

1024/1331 = 77%

150/1331 = 11%

So that's why I wonder if this is correct to do :

policy-map x

class ef

priority 128

class af31

bandwidth remaining percent 77

class as32

bandwidth remaining percent 11

class class-default

bandwidth remaining percent 11

int s0/0

max-reserved-bandwidth 90

service policy output x

The above seems to me is logical but it seems to completely contradict what Maria says about bandwidth remaining percent does not apply to default class as it excludes both LLQ and default class. I didn't get the clarification of whether this excludes default class under which situation (ie. is it only true when default class does not have explicit bandwidth defined, so if that is the case, the AF31 + Af32 suppose "can" add up to 100% excluding default class when default class doesn't explicit define bandwidth), so when we do define explicit bandwidth (in our case, we use bandwidh remaining percent on default class, then when we do this explicit, then the bandwidth remaining percent is NOT excluded from default class anymore), not sure if this is true.

thanks,

Joyce

Correct Answer
Joseph W. Doherty Tue, 02/03/2009 - 05:18

Got it.

From what you describe, I believe you would be much better using an explict bandwidth rather than percentages since this avoids both the original calculations and helps avoid confusion for anyone looking at it later. I.e., if CIR for AF31 is 1024 on a T1, then define its bandwidth as 1024 and insure either there's no significant volume of traffic in class-default FQ (on most platforms) or define an explict bandwidth for class-default.

I also advise, if possible, to avoid changing max-reserve since this type of the change often seems to lead to unpleasant surprises latter on.

With the cautions out of the way, I believe you calculate remaining percent as follows:

Total remaining percent is max-reserve % of link less LLQ allocations.

e.g.

using your 90% max, that allows (about) 1382 (1536 * .9) to be defined (Not "So 90% of 1536 will be 1331k." - unless my calculator is wrong). Subtract out all LLQ allocations, in your case 128, which leaves 1254. 1024 is (about) 82% of this, and that would be the value you want for AF31 remaining percent. The remaining 18% can be allocated as you chose, such a 9% for AF32 and class-default.

blackladyJR Tue, 02/03/2009 - 05:41

THANKS!!! Since I know there are always a lot of default traffic such as internet traffic that is unpredicatable, so I think from your advice, I am safer to go with defining default class explicit rather than FQ.

Yes I forgot how 1331 comes from but you are right, it's 1382 :)

And I understand I don't necessary allocate 9% to each remaining class, I can put 5% on each so it doesn't have to add up to be 100%, correct? (when using bandwidth remaining percent), I wasn't sure where I read in the past that somehwere it says it needs to add up 100% when using this but I can't find it so my memory is certainly not reliable. I trust you :)

As you the QoS Expert, my final question is that how does "weight" determine in the router? This is CBWFQ, so each class has different weights and I assume weights means different priority when scheduling the packets from various queue onto the outgoing interface.

I wonder is it Cisco router already has some default behavior so AF4x has higher weight than AF3x and so on. So say I have 100 packets in AF3x queues waiting to be served, then at this moment, I have 20 packets coming into AF4x queue, will router then do "round robin" to serve ALL the classes (except LLQ which we know is highest priority and will cut line and will serve ALL its packets before back to the CBWFQ queues). So now the router will round robin to AF4x and will server perhaps 10 packets in AF4x, and serve 8 packets in AF3x, and 4 packets in AF2x, 1 packet in Default class and round robin again? So my question is is this the behavior so each CB classes get its turn but the higher class gets more serving as it has higher weights?

If the above is true, then how does weight determine? Does it has "anything" to do with the bandwidth value that we assign on each class? So if I assign 60% on AF31 and 30% on AF41 let's say, will router treat AF31 a 2:1 ratio higher weight in AF31 instead of AF41? Or will router totally ignore the bandwidth so even thought AF31 has 60%, the router will still have higher weight in AF41 just based on Dscp value and will serve more on AF41 than AF31 regardless of bandwidth value?

I promise this is my last question (for now :)

thanks again.

Joyce

Joseph W. Doherty Tue, 02/03/2009 - 06:18

When allocating class bandwidths, you never need to allocate the maximum possible, but when allocating, you can't exceed the maximum reserved. (E.g. your remaining percent doesn't need to sum to 100, but sum can't exceed 100.)

Weight is what, I believe, the scheduler actually uses to determine how much bandwidth a queue will obtain. When we set bandwidth on classes, this defines a weight on that class's packets.

WFQ does use a packet's IP precedence to adjust a packet's weight. I believe on many platforms, fair-queue within CBWFQ class-default, is WFQ.

With regard to the QoS packet scheduler dequeuing packets, it's not just counting packets, but, I believe, counting bytes.

In your question about assiging AF31 60% and AF41 30%, the scheduler should attempt to dequeue twice the bytes for AF31 relative to AF41 if in unique CBWFQ class queues. If the AF31 and AF41 were in WFQ, you don't get to assign bandwidths but AF41 would get more that AF31. (For WFQ flows, it's possible to calculate what the bandwidth allocations would be - see Cisco documentation on WFQ - you might also want to read Cisco documentation on custom queuing.)

blackladyJR Tue, 02/03/2009 - 06:41

Hi Joe,

Right, I recalled WFQ use IP Precedence so there is "priority" to serve higher class so TOS 4 has higher priority than TOS 3.

But here is from the congestion mgmt overview doc on cisco:

"For CBWFQ, the weight for a packet belonging to a specific class is derived from the bandwidth you assigned to the class when you configured it. Therefore, the bandwidth assigned to the packets of a class determines the order in which packets are sent. All packets are serviced fairly based on weight; no class of packets may be granted strict priority."

From these statement, it makes me think that when we do CBWFQ, the router does not care DSCP or TOF value at all when it comes to "dequeue" and "serve" the packets going out. It looks like the 'weight' is related to ' bandwidth' so if I have 60% AF31 and 30% AF41 in the bandwidth assignment, then the router will serve 2:1 ratio round robin to serve twice the packet/bytes in AF31 than AF41 at each round. So that makes AF41 does not have any "better" treatment than AF31 at all.

So that means if we have 30% AF31 and 70% Default let's say, then that means Default traffic gets serve more at each round than AF31.

It just doesn't seem to make sense because the router doesn't care the "priority" or importance of DSCP order while a regular WFQ will treat weight based on TOS.

What do you think?

I mean sometimes you have high priority application so you put them in AF31 let's say, but they can be very important but they may not need large bandwidth and rest may be default internet low priority traffic. But since the bandwidth in default is say 5 times the AF31, then the router dequeue 5 times more packets than the AF31? Just seem wrong.....

Joseph W. Doherty Tue, 02/03/2009 - 12:26

The documentation is correct when referring to CBWFQ single FIFO queue classes. However, I believe "fair-queue" within most platforms class-default is WFQ, i.e. class-default FQ flows may have different weights assiged based on IP Precedence. (I've yet to figure out how those flow weights are assigned relative to other class queues. I've done some simple testing and I recall what I saw wasn't obvious.)

If you explictly assign class-default 70%, it's a FIFO queue like other class queues. If you've only assigned 30% to AF31 leaving 70% undefined, don't assume class-default FQ gets a weight for 70%. Again, the default 25% is reserved such that you can not explictly reserve that bandwidth, although class-default is able to use that bandwidth, but that bandwidth is not allocated just for class-default. In other words, don't equate the set aside bandwidth as being the same as class-default.

In your comment where if you consider AF31 5x more important than default Internet traffic, you need to classify both into classes such that the AF31 class has 5x the bandwidth allocated to it the Internet traffic class. (Also again, don't assume queues are dequeued by packet count, they should be dequeued based on packets' byte count. E.g. 5 packets dequeued sized 200 should count as equal to one packet dequeued sized 1,000.)

blackladyJR Tue, 02/03/2009 - 12:36

so pretty much the router when dealing with CBWFQ does not care the DSCP or IP Precedence at all. AF41 does not mean to have any 'weights" to serve more packet bytes by the DSCP value. The only thing that controls the 'weight" is the 'bandwidth" value then. So if i have important application say in AF41 but even though it is "important", but it doesn't have many users for that application. While I have many users using other application in AF31 and default. So if I assign each class based on the bandwidth usage to give AF31 and default more bandwidth than AF41. Then AF41 althought the application is most criticial, it will not get any better treatment, in fact AF31 and Default will get to serve /dequeue more packets bytes than AF41 at each round.

got it!

Thank you very much once again.

Bye.

Joseph W. Doherty Tue, 02/03/2009 - 13:02

Again, standard CBWFQ FIFO classes don't implicitly consider DSCP marking for dequeuing weight. Since you define the bandwidth for these classes, which indirectly assigns weights, you have precise control. Also keep in mind, strictly speaking, no AFx class is more imporant than another, they represent service classes. It's up to you to define the service for the AFx classes. It's not a question of one class of traffic is more critical than another, but rather whether defined classes obtain the resources they need to work as required. Recall, non AFx traffic often falls within best effort (BE), i.e. no guarantee beyond we'll try.

blackladyJR Mon, 02/16/2009 - 08:28

Hi Milan,

This is really a very useful document!!! I read it and it really explains a lot!

Thanks for sharing.

Joseph W. Doherty Mon, 02/16/2009 - 15:57

Indeed, very interesting! However, keep in mind any time you start to use undocumented information, vendor could change how it works.

One thing missing, author doesn't make clear how multiple class-default FQ flows also compete for bandwidth since only one class-default flow is shown. So I don't agree with ". . . we may conclude that user-defined classes dominate dynamic flows almost all the time, unless they have small shares of bandwidth configured." except at a per flow level, not at the class level.

"If you want the scheduler to honor class-default traffic, assign it an explicit bandwidth value. ", this is something I often emphasis for most platforms, but not all platforms.

Still, again, interesting read.

milan.kulik Tue, 02/17/2009 - 06:02

Hi Joseph,

ad "how multiple class-default FQ flows also compete for bandwidth")

If you read the Questions/Answers below the article, #8 confirms Sequence Numbers (described in Cisco WFQ documentation) are assigned not only to FQ packets, but also to packets coming to other (User-defined) Classes.

This is exactly what I was missing in all Cisco documents and books :-((

(Cisco trying to keep their know-how possibly?)

I'm not sure if assigning an explicit bandwidth value to the class-default is an optimal solution. You are degrading it to a simple FIFO then, but IMHO, the internal SN algorithm probably keeps running even among less flows (one per a class).

BR,

Milan

Joseph W. Doherty Tue, 02/17/2009 - 07:00

Milan, oh I agree making class-default FIFO isn't optimal (depending on what you're trying to optimize), and yes things may run smoothly between class flows, except, I believe, when one class, class-default, has multiple flows within the class which are competing for bandwidth against single class flows.

The better solution is seen on some platforms such as the 7500, 6500/7600 FlexWAN and SIP-200, which limit the aggregate bandwidth of multiple FQ with class-default. (They also permit FQ within other classes.)

Take a look as section Understand Platform Differences within http://www.cisco.com/en/US/tech/tk39/tk48/technologies_tech_note09186a00800fe2c1.shtml#platform, although the document is about ATM interfaces, believe scheduling holds true for other interface types.

On platforms that allow class-default FQ flows to schedule against defined classes, the only way I see you can easily guarantee your bandwidth allocations for defined classes is insuring class-default doesn't use FQ.

I'm sure you understand, as number of flows with FQ increase, each individual flow receives proprotional less bandwidth guarantee. Again, the problem is, this would also hold true for other class flows on those platforms which don't constrain class-default FQ.

Given all the variables and if we know the total allowed FQ flows and if we allow for FQ IP Precedence impact, we can predict any individual flow's bandwidth guarantee. A lot of work though, and we hope Cisco doesn't change the internal rules as they might on maximum number of flows permitted or as they done with base weight (4096 vs. 32384).

So, again, your reference is extremely interesting, but you may place your network functioning as intended at risk using undocumented information, and you make a lot of work for yourself (and others to support) if you need to calculate each flow's bandwidth guarantee when using FQ.

Don't misunderstand, I'm not suggesting not to use FQ, just depending on what you're trying to accomplish, the QoS impact might be consirable depending on platform and whether it's active or not.

blackladyJR Tue, 02/17/2009 - 15:25

Here is an interesting page from cisco about IOS 12.2.20T.

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6558/white_paper_c11-481499.html

It talks about default class with FQ no longer use IP Precedence to determine weight but it serves equally. Maybe I just read this wrong.

Also it mentions we no longer need to assign max-reserved-bandwidth as we can use all 100% (with at least 1% for default class) so the 75% rule is gone in this release.

I am not familar with this HQF of what it means. Does that mean starting this IOS version, all the changes in this link will apply to general CBWFQ?

Here is a link by Andy with simulation of exact same config with an version prior to 12.2.20T and then with exact config with 12.2.20T. The result is dramatic different.

http://cisconinja.wordpress.com/2009/01/22/class-based-weighted-fair-queueing-and-low-latency-queueing-tests/#comment-17

Just look at the very end of his simulation for the two results with two IOS versions.

Joseph W. Doherty Tue, 02/17/2009 - 17:01

Actually, Cisco promised all (router?) QoS would eventually work much as it did on the 7500 some time back. Looks like with 12.4(20)T they have finally done so. I've worked with a couple of 7200s with -G2 processors running 12.4(20)T and had noticed some of the differences but not all of them. I hadn't read the release notes nor the white paper you've reference. (Thanks!)

milan.kulik Wed, 02/18/2009 - 05:37

Hi Joseph,

I made some math excercises on formulas given within the article I found with following conclusions:

The IP Precence can twist the bandwidth guarantee considerably.

I played with maximum 256 flows in class-default. In worst case (256 flows with IP Prec=7) I noticed only 50% of bandwidth reserved by the bandwidth command used for user-defined classes was provided in fact.

But if only 128 Flows presented with IP Prec 7, then the bandwidth provided to the user-defined classes was OK.

When IP Prec 0 was set to all 256 flows in the class-default, the bandwidth provided to user-defined classes was always higher than the reserved values.

I'm not sure if the formulas are correct, I can't see how were derived from the SN formula used by the CBWFQ scheduler (packet lenght was ignored so far).

And it's also possible CBWFQ is calculating with IP Prec also for user-defind classes.

But my feeling here is:

If you set IP Precedence to 0 for the class-default flows (and that's possible while marking incoming packets), you can rely on CBWFQ to provide at least the bandwidth reserved for your user-defined classes even when your class-default is running FQ.

IMHO, Cisco should publish the algorithm to stop all these guessing discussions :-((

BR,

Milan

Actions

This Discussion