LLQ understanding

Answered Question
May 6th, 2010
User Badges:
  • Silver, 250 points or more

Hello all,


I have an issue with theory and practice with LLQ. I understand that LLQ allow you to prioritize packets (e.g. voice) up to a certain limit (bandwidth) when available bandwidth is fully utilized. When there is enough bandwidth free, than this LLQ limit can overcome.


Let me give you an example to understand better and be able to help me with an advice.


Let's assume that I have a 4Mbit available bandwidth on a Serial line with Frame-Relay encapsulation and I apply the following configuration:


class-map voice-ef
match dscp 46


policy-map po2-s1-out
class voice-ef
  priority 423
class class-default
  bandwidth 3489


policy-map po1-s1-out
  class class-default
  shape average 3912000 39120 0
  service-policy po2-s1-out


map-class frame-relay frm-s1-out
frame-relay fragment 1600
service-policy output po1-s1-out


int s1.100
frame-relay interface-dlci 100 IETF  
  class frm-s1-out


I reserved 10% from the bandwith in LLQ and the rest goes to default class (the packets are marked correctly at network edge). The rest of the QoS is necessary because I have to shape the bandwidth and apply it in a FR interface.


Question, in this scenario, if the bandwitdh is not occupied, can voice calls overcome the 10% bandwidth (from 4Mbit aprox 423kbit). If I use the codec g711 which has about 100k / call , that mean that I can do 4 parallel calls in this 423kbit. What if I make 6,7 calls when the bandwidth if free?


My findings are that if I overcome the 423kbit, then I have packet loss and choopy voice calls. It's like LLQ is not "borrowing" bandwidth from other classes.


Thank you!

Correct Answer by Jon Marshall about 7 years 3 weeks ago

chiorean.calin wrote:


Hello all,


I have an issue with theory and practice with LLQ. I understand that LLQ allow you to prioritize packets (e.g. voice) up to a certain limit (bandwidth) when available bandwidth is fully utilized. When there is enough bandwidth free, than this LLQ limit can overcome.


Let me give you an example to understand better and be able to help me with an advice.



I reserved 10% from the bandwith in LLQ and the rest goes to default class (the packets are marked correctly at network edge). The rest of the QoS is necessary because I have to shape the bandwidth and apply it in a FR interface.


Question, in this scenario, if the bandwitdh is not occupied, can voice calls overcome the 10% bandwidth (from 4Mbit aprox 423kbit). If I use the codec g711 which has about 100k / call , that mean that I can do 4 parallel calls in this 423kbit. What if I make 6,7 calls when the bandwidth if free?


My findings are that if I overcome the 423kbit, then I have packet loss and choopy voice calls. It's like LLQ is not "borrowing" bandwidth from other classes.


Thank you!


If there is no congestion then yes voice calls can indeed use spare bandwidth from other classes. But the key thing to understand is that any spare bandwidth used is not prioritised so you will simply get best effort which means your VOIP calls could become very choppy. It's not that LLQ is not "borrowing" bandwidth, it's that the additional bandwidth being used is not prioritised by LLQ.


So you need to allocate enough bandwidth to the priority queue for the amount of calls you expect to be made.


Jon

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.8 (6 ratings)
Loading.
Correct Answer
Jon Marshall Thu, 05/06/2010 - 04:02
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

chiorean.calin wrote:


Hello all,


I have an issue with theory and practice with LLQ. I understand that LLQ allow you to prioritize packets (e.g. voice) up to a certain limit (bandwidth) when available bandwidth is fully utilized. When there is enough bandwidth free, than this LLQ limit can overcome.


Let me give you an example to understand better and be able to help me with an advice.



I reserved 10% from the bandwith in LLQ and the rest goes to default class (the packets are marked correctly at network edge). The rest of the QoS is necessary because I have to shape the bandwidth and apply it in a FR interface.


Question, in this scenario, if the bandwitdh is not occupied, can voice calls overcome the 10% bandwidth (from 4Mbit aprox 423kbit). If I use the codec g711 which has about 100k / call , that mean that I can do 4 parallel calls in this 423kbit. What if I make 6,7 calls when the bandwidth if free?


My findings are that if I overcome the 423kbit, then I have packet loss and choopy voice calls. It's like LLQ is not "borrowing" bandwidth from other classes.


Thank you!


If there is no congestion then yes voice calls can indeed use spare bandwidth from other classes. But the key thing to understand is that any spare bandwidth used is not prioritised so you will simply get best effort which means your VOIP calls could become very choppy. It's not that LLQ is not "borrowing" bandwidth, it's that the additional bandwidth being used is not prioritised by LLQ.


So you need to allocate enough bandwidth to the priority queue for the amount of calls you expect to be made.


Jon

Calin Chiorean Thu, 05/06/2010 - 04:48
User Badges:
  • Silver, 250 points or more

Hi Jon,


Thanks for pointing this out, was very helpful. In this scenario, when LLQ queue is taking bandwidth from other classes, it is normal that it's showing drops?


Class-map: voice-ef (match-all)
          21023000 packets, 4121654150 bytes
          30 second offered rate 37000 bps, drop rate 0 bps
          Match:  dscp ef (46)
          Queueing
            Strict Priority
            Output Queue: Conversation 136
            Bandwidth 1024 (kbps) Burst 25600 (Bytes)
            (pkts matched/bytes matched) 21023000/4121654150
            (total drops/bytes drops) 68775/13847017


Thanks!

Jon Marshall Thu, 05/06/2010 - 04:54
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

chiorean.calin wrote:


Hi Jon,


Thanks for pointing this out, was very helpful. In this scenario, when LLQ queue is taking bandwidth from other classes, it is normal that it's showing drops?


Class-map: voice-ef (match-all)
          21023000 packets, 4121654150 bytes
          30 second offered rate 37000 bps, drop rate 0 bps
          Match:  dscp ef (46)
          Queueing
            Strict Priority
            Output Queue: Conversation 136
            Bandwidth 1024 (kbps) Burst 25600 (Bytes)
            (pkts matched/bytes matched) 21023000/4121654150
            (total drops/bytes drops) 68775/13847017


Thanks!


Very interesting question and the short answer is i don't know. I would have though a drop is a drop in that it is not then "passed" on to the other available bandwidth but i'm not sure.


Perhaps if i get the time i'll lab something up to test it.


Jon

Calin Chiorean Thu, 05/06/2010 - 05:06
User Badges:
  • Silver, 250 points or more

Fair answer


To go into more details and to tell you why I have doubts about LLQ taking bandwidth from another class, when there is enough free bandwidth.


I have setup a scenario:


PC1 -> Router1 - > Router2 -> PC2


On R1 I have reserved 400kbit for LLQ and the rest in the default-class (up to 100Mbps). Then with no traffic on the line, I have tested with IPerf with 4 parallel streams, each stream 100k (like a phone call with G711) and everything was fine, IPerf showed no packet loss on those 4 streams.


The I tried with 5,6,7 and so on parallel streams of 100k each, and already I had packet loss and drops on the router. It was like LLQ was not taking that additional 100k (e.g. 5 parallel streams) from the class-default queue but instead put all 5 x 100k into the 400k reservation and drop packets.


I'm confuse a little bit, so please if you have time and find something new, add it here. Maybe it will help others as well.


Thanks!

Calin Chiorean Mon, 05/10/2010 - 02:59
User Badges:
  • Silver, 250 points or more

If anybody is still interested in this subject:


I did some testing in the LAB, 2 routers, back2back connection over Serial interface (diff encaps, HDLC, PPP, FR) and basic QoS config, with LLQ priority up to 400k and burst value calculated automatically.


The I started to push traffic from R1 to R2

- 4 parallel streams, each stream 100k, result no packet drops

- 6 parallel streams, each stream 100k, results random, sometimes not packet loss, sometimes yes (I believe depending on the burst value and serialization delay)

-10 parallel streams, each stream 100k, alway packet drops.


Then I increase the burst value to maximum (I guess value 2000000). With this burst on 10 parallel streams (100k/stream) there was no packet drops. And I have repeated the test couple of times.


It is possible that LLQ borrow bandwidth from other classes (class-default in my case) only up to the limit of the burst value?


I'm still somehow confuse about this topic. I'll read more in the meantime and post here if I find something useful.

milan.kulik Wed, 05/26/2010 - 02:09
User Badges:
  • Red, 2250 points or more

Hi,


isn't the shaper shown in your original message still active?

Or some policer?


Could you provide the complete

sh policy-map int ... out

output?


BR,

Milan

d.serra Tue, 05/25/2010 - 07:53
User Badges:

Very very interesting.  Just as a side note, run a sniffer to see how big the packets are that you are generating as I have noticed that an LLQ will drop packets bigger then the MTU.  You can test this by using the 'IP MTU {value}' command on your outgoing interface where service-policy is applied.  The lower the MTU, the more packet loss you will have or in your case, the bigger the packets, the more drops you get.


Maybe this has something to do with it?


Also, this issue is not seen in a standard CBWFQ.  Only on LLQ.


Dave

Calin Chiorean Wed, 05/26/2010 - 02:41
User Badges:
  • Silver, 250 points or more

Thank you all!


Below you have all my findings in regard to LLQ. Some of this were confirmed by Cisco guys also.


1. The amount of bandwidth reserved for LLQ has to be calculated in such manner to accommodate most of the matched traffic

- in other words if you have a bandwidth of 10Mbps, and you know that you need for voice traffic 5Mbps, you don't configure LLQ just for 1Mbps in the idea that it will borrow as much as it want if the 10Mbps are not used; this behavior is more specific to PQ


2.  You reserve an amount of bandwidth for LLQ for the case in which bandwidth is fully occupied and when it's empty

- I mean that if you know you need to reserver 5Mbps, do it; don't rely on the fact that it will borrow;


3. LLQ cannot borrow the full bandwidth

- I have observed this in my test, but maybe I didn't had all the necessary tools to have a more accurate result, but somehow it's normal this behavior; you cannot expect to reserver some Kbps for LLQ and then this to go up to Gbps if you have this amount of bandwitdh


4. LLQ borrowing depend on the burst value

- LLQ class can borrow bandwidth in relation to the burst value; if this is higher, LLQ can borrow more bandwidth, but be aware that if you have set manually the burst value and this is too high, it can influence your Tc value, and this will have negative effect on the voice


5. Last but not least, I have read the LLQ is using the method "tokens in the bucket" to borrow bandwidth from other classes

- LLQ can borrow bandwidth for a limited time, equal to the the tokens that it have in the bucket; when tokens are exhausted, the LLQ will be limited to the amount of bandwidth specified in it own class and not more; when the LLQ uses less that the bandwidth amount specified, the tokens are refilled in the bucket; how is refilled and how it is spending tokens...this I didn't found; maybe it's a Cisco secret



I hope this will help others. Final word, configure your LLQ as accurate as you can and don't rely on it's capacity of borrowing bandwidth. This protocol was develop to reserve amount of bandwidth for certain traffic and not to borrow it.


If you find useful, please rate!


Calin

milan.kulik Wed, 05/26/2010 - 05:57
User Badges:
  • Red, 2250 points or more

Hi,


IMHO, all what you say is correct.

BUT in a time of congestion.

See http://www.cisco.com/en/US/customer/docs/ios/qos/configuration/guide/congstion_mgmt_oview_ps6350_TSD_Products_Configuration_Guide_Chapter.html#wp1003189

It says:

"Priority traffic metering has the following qualities:

•It is much like the rate-limiting feature of CAR, except that priority traffic metering is only performed under congestion conditions. When the device is not congested, the priority class traffic is allowed to exceed its allocated bandwidth. When the device is congested, the priority class traffic above the allocated bandwidth is discarded.

•It is performed on a per-packet basis, and tokens are replenished as packets are sent. If not enough tokens are available to send the packet, it is dropped.

•It restrains priority traffic to its allocated bandwidth to ensure that nonpriority traffic, such as routing packets and other data, is not starved."


Generally, Cisco documentation is foggy regarding clear statements how QoS works in deep details. Unfortunately even the expensive trainings don't answer questions like this one.

I'm afraid it's a part of Cisco know-how strategy.


BR,

Milan

Calin Chiorean Wed, 05/26/2010 - 06:20
User Badges:
  • Silver, 250 points or more

I totally agree with you:


"When the device is not congested, the priority class traffic is allowed to exceed its allocated bandwidth."


But it's not saying until which limit it can exceed the it's allocated bandwidth. So, back to my example, if you have 10Mbps bandwidth and you set LLQ to 3Mbps, this does not mean that when it's not congestion the LLQ WILL reach 10Mbps. Maybe...depending on the burst and tokens in the bucket system.


At least this is how I see things.


Thanks again!

d.serra Wed, 05/26/2010 - 05:30
User Badges:

"3. LLQ cannot borrow the full bandwidth

- I have observed this in my test, but maybe I didn't had all the necessary tools to have a more accurate result, but somehow it's normal this behavior; you cannot expect to reserver some Kbps for LLQ and then this to go up to Gbps if you have this amount of bandwitdh"


This is not correct.  Below you can see that I have a 5 minute average of 30000 bps and yet the LLQ was set to 16k (priority 16).  It is clear that an LLQ can handle more bandwidth then what was allocated.  Below is how I tested this:


Sw1 -> R1 -> R2


From Sw1 I would ping R2 using 1000 packets with 1400 bytes.  I also set the dscp bits in the packets to af41 to ensure they match the LLQ. 


R1 and R2 are connected via a Serial.  


R1#show policy-map interface ser0/0
Serial0/0

  Service-policy output: DSCP-41

    Class-map: DSCP-41 (match-all)
      1000 packets, 1404000 bytes
      5 minute offered rate 30000 bps, drop rate 1000 bps
      Match:  dscp af41 (34)
      Queueing
        Strict Priority
        Output Queue: Conversation 264
       Bandwidth 16 (kbps) Burst 400 (Bytes)
        (pkts matched/bytes matched) 1/1404
        (total drops/bytes drops) 1/1404

    Class-map: class-default (match-any)
      8 packets, 500 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any


Configuration for R1 is below...please note that I had also configured NAT just so I would not have to configure routing.  The NAT configuration has no impact on the CBWFQ configuration:


R1#show run
Building configuration...

Current configuration : 1114 bytes
!
version 12.4
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname R1
!
boot-start-marker
boot-end-marker
!
enable password cisco
!
no aaa new-model
no network-clock-participate slot 1
no network-clock-participate wic 0
ip cef
!
!
!
!
no ip domain lookup
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
class-map match-all DSCP-41
match  dscp af41
!
!
policy-map DSCP-41
class DSCP-41
  priority 16
!
!
!
!
!
!
interface FastEthernet0/0
ip address 172.25.0.14 255.255.255.0
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
!
interface Serial0/0
ip address 2.2.2.1 255.255.255.0
ip nat inside
ip virtual-reassembly
service-module t1 timeslots 1-24
service-policy output DSCP-41
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
ip nat inside source static 2.2.2.2 172.25.0.22
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
exec-timeout 0 0
no login
transport input telnet ssh
!
!
end

R1#

Calin Chiorean Wed, 05/26/2010 - 06:31
User Badges:
  • Silver, 250 points or more

Thank you for your observation and time spent on this topic.


I still think that I'm somehow right when what I say "LLQ cannot borrow the full bandwidth"

You have set LLQ to 16k and you have 30000bps (which is about 30k) and you already have a drop of 1000bps. If LLQ is allowed to borrow as much bandwidth as it want, why there is packet loss?

I think you know why. Because of the burst limitation and tokens in the bucket system.

If I'm not wrong the bandwidth for a Serial interface default to 1544kbps (in case that bandwidth command is not specified). Now, with your 16k on LLQ try with some tool like IPerf to saturate this 1544kbps with traffic that is matched on the LLQ class, and let me know what is the amount of packet loss that you will have on this class. I think it will be quite high.


" It is clear that an LLQ can handle more bandwidth then what was allocated "


You're right, but I never said that LLQ cannot go above the limit, but how much and for how many time, this depend of the protocol.

milan.kulik Wed, 05/26/2010 - 07:08
User Badges:
  • Red, 2250 points or more

Hi,


iI need to refresh my knowledge of the counter meanings here, but:

R1#show policy-map interface ser0/0
Serial0/0

  Service-policy output: DSCP-41

    Class-map: DSCP-41 (match-all)
      1000 packets, 1404000 bytes
      5 minute offered rate 30000 bps, drop rate 1000 bps
      Match:  dscp af41 (34)
      Queueing
        Strict Priority
        Output Queue: Conversation 264
       Bandwidth 16 (kbps) Burst 400 (Bytes)
        (pkts matched/bytes matched) 1/1404
        (total drops/bytes drops) 1/1404

    Class-map: class-default (match-any)
      8 packets, 500 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any


IMHO, this might mean:

1000 packets matched the DSCP-41 class, but only 1 packet was placed into the output queue (as the output buffer was full at the moment that packet should be sent out, possibly by other large packets sent out from the serial interface).

And this single packet was dropped - because a 1404-Byte packet coming from LAN might exceed 16 kbps speed and 400Byte burst easily.


IMHO, if you make a test on FastEthernet ports with the same policy, you'll see the priority traffic (matching DSCP-41 class)  would take much higher bandwidth. The drops  would occur only in a case of congestion (i.e., ouput port buffer full) and then the token bucket, etc., would apply.

Edit: But this would never happen on a FastEthernet port.


BR,

Milan


Message was edited by: milan.kulik

d.serra Wed, 05/26/2010 - 07:40
User Badges:

"


I still think that I'm somehow right when what I say "LLQ cannot borrow the full bandwidth"

You have set LLQ to 16k and you have 30000bps (which is about 30k) and you already have a drop of 1000bps. If LLQ is allowed to borrow as much bandwidth as it want, why there is packet loss?

I think you know why. Because of the burst limitation and tokens in the bucket system.

If I'm not wrong the bandwidth for a Serial interface default to 1544kbps (in case that bandwidth command is not specified). Now, with your 16k on LLQ try with some tool like IPerf to saturate this 1544kbps with traffic that is matched on the LLQ class, and let me know what is the amount of packet loss that you will have on this class. I think it will be quite high."


Speculation...Speculation.  LLQ dropping 1 out of 1000 packets is hardly a significant number.  The point of what this is to illustrate is that LLQ is not setting a hard limit to 16k and if there is available bandwidth it will use it. The Burst is set to 400k and yet it transfered all but 1 packet each having a size of 1400k.  Burst value is not in play here.

milan.kulik Thu, 05/27/2010 - 03:27
User Badges:
  • Red, 2250 points or more

Hi,


priority 16
configured with no options means:

Burst = 400 Bytes

As "The default burst size is calculated based on a 200-millisecond interval and the bandwidth configured for low latency queueing (LLQ)."

(So 16*1024 bps /5 = 3276,8 bits in 200ms = 409,6 Bytes in 200 ms)

See http://www.cisco.com/en/US/docs/ios/12_1t/12_1t3/feature/guide/dtcfgbst.html for details.


Now some serialisation delay maths:

If I suppose 100Mbps LAN interface, one 1400Bytes packet can come in 0,0001 s in theory.

In a case of 10Mbps LAN, it's 0,001s.

To send the same packet out of the 1544 kbps serial line would take 0,007 s.

In a case of a continuous packet flow, the serial line output buffer might become full, and the burst size can play a role.

(Don't forget LLQ was designed for voice originally, so small packets are expected by default.)


But generally, I agree the LLQ should be able to get all the bandwidth available in a case of no congestion.

As I said already, if you repeat the same test on Ethernet interfaces instead of serial one, you might be able to get the full bandwidth.


BR,

Milan

Calin Chiorean Thu, 05/27/2010 - 04:41
User Badges:
  • Silver, 250 points or more

I will try to find time to repeat the tests on Ethernet line with some scenario like


PC (Iperf) - R1 - R2 - PC (Iperf)


and see how much I can go over the reserved bandwidth for LLQ. Of course I'll have no congestion on the line.


Once it's completed I will post the results here.

d.serra Thu, 05/27/2010 - 13:12
User Badges:

Milan,


I think you are dead on but you math may be a bit off.  Tc is 250 ms not 200ms.  The reason is Bc is 400k and bps is 16k.  So 16000 divided by 400 is 4 meaning that there are 4 250ms time slots transmitting 400bps (Bc).  Add the 4 slots together and you get 16k bps per 1 second.


To be honest, I've read the same docs where it describes the LLQ as having some sort of CAR function but I believe there are changes to the newer code such as the 12.4 I used to create this test that allows LLQ to use excess burst (Be) though that too is speculation.


Calin, looking forward to seeing the results of you test.


Dave

Calin Chiorean Fri, 05/28/2010 - 02:08
User Badges:
  • Silver, 250 points or more

Hello again,


So, I took some time to test, and indeed it seems that I was wrong assuming that LLQ cannot borrow full bandwidth.

The testing scenario is very simple


PC-SRC(IPerf) -> R1 -> R2 -> PC-DST(IPerf).


Bandwidth: 10Mbps


On R1, I have applied the following configuration:


####### Voice packet marking with EF #######################################################



access-list 101 permit udp any any range 16384 32767
access-list 101 permit udp any range 16384 32767 any
access-list 101 permit udp any any range 5060 5061
access-list 101 permit udp any range 5060 5061 any



class-map match-all VOIP
match access-group 101



policy-map ASTERISK
class VOIP
  set dscp ef


int gi0/0
service-policy input ASTERISK


######## LLQ configuration with 1Mbps reservation ########################################################


class-map voice-ef
match dscp 46


policy-map Queue-out
class voice-ef
  priority 1000
class class-default


interface gi0/1
service-policy output Queue-out




Now, some traffic


10 parallel calls, each call 100kbps that should be aprox. 1Mbps

Result: 0% packet loss


20 parallel calls, each call 100kbps that should be aprox. 1Mbps

Result: 0% packet loss


50 parallel calls, each call 100kbps that should be aprox. 1Mbps

Result: 0% packet loss


You can see a screencast of the test here: bit.ly/aIsI1B


Now I have too see why there were issues in the first test. Maybe due to frame-relay, nested policy-maps or shaping.

I will let you know if with FR everything is fine once I'll complete that test

Actions

This Discussion

Related Content