cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
13262
Views
24
Helpful
18
Replies

LLQ understanding

Calin C.
Level 5
Level 5

Hello all,

I have an issue with theory and practice with LLQ. I understand that LLQ allow you to prioritize packets (e.g. voice) up to a certain limit (bandwidth) when available bandwidth is fully utilized. When there is enough bandwidth free, than this LLQ limit can overcome.

Let me give you an example to understand better and be able to help me with an advice.

Let's assume that I have a 4Mbit available bandwidth on a Serial line with Frame-Relay encapsulation and I apply the following configuration:

class-map voice-ef
match dscp 46

policy-map po2-s1-out
class voice-ef
  priority 423
class class-default
  bandwidth 3489

policy-map po1-s1-out
  class class-default
  shape average 3912000 39120 0
  service-policy po2-s1-out

map-class frame-relay frm-s1-out
frame-relay fragment 1600
service-policy output po1-s1-out

int s1.100
frame-relay interface-dlci 100 IETF  
  class frm-s1-out

I reserved 10% from the bandwith in LLQ and the rest goes to default class (the packets are marked correctly at network edge). The rest of the QoS is necessary because I have to shape the bandwidth and apply it in a FR interface.

Question, in this scenario, if the bandwitdh is not occupied, can voice calls overcome the 10% bandwidth (from 4Mbit aprox 423kbit). If I use the codec g711 which has about 100k / call , that mean that I can do 4 parallel calls in this 423kbit. What if I make 6,7 calls when the bandwidth if free?

My findings are that if I overcome the 423kbit, then I have packet loss and choopy voice calls. It's like LLQ is not "borrowing" bandwidth from other classes.

Thank you!

1 Accepted Solution

Accepted Solutions

Jon Marshall
Hall of Fame
Hall of Fame

chiorean.calin wrote:

Hello all,

I have an issue with theory and practice with LLQ. I understand that LLQ allow you to prioritize packets (e.g. voice) up to a certain limit (bandwidth) when available bandwidth is fully utilized. When there is enough bandwidth free, than this LLQ limit can overcome.

Let me give you an example to understand better and be able to help me with an advice.

I reserved 10% from the bandwith in LLQ and the rest goes to default class (the packets are marked correctly at network edge). The rest of the QoS is necessary because I have to shape the bandwidth and apply it in a FR interface.

Question, in this scenario, if the bandwitdh is not occupied, can voice calls overcome the 10% bandwidth (from 4Mbit aprox 423kbit). If I use the codec g711 which has about 100k / call , that mean that I can do 4 parallel calls in this 423kbit. What if I make 6,7 calls when the bandwidth if free?

My findings are that if I overcome the 423kbit, then I have packet loss and choopy voice calls. It's like LLQ is not "borrowing" bandwidth from other classes.

Thank you!

If there is no congestion then yes voice calls can indeed use spare bandwidth from other classes. But the key thing to understand is that any spare bandwidth used is not prioritised so you will simply get best effort which means your VOIP calls could become very choppy. It's not that LLQ is not "borrowing" bandwidth, it's that the additional bandwidth being used is not prioritised by LLQ.

So you need to allocate enough bandwidth to the priority queue for the amount of calls you expect to be made.

Jon

View solution in original post

18 Replies 18

Jon Marshall
Hall of Fame
Hall of Fame

chiorean.calin wrote:

Hello all,

I have an issue with theory and practice with LLQ. I understand that LLQ allow you to prioritize packets (e.g. voice) up to a certain limit (bandwidth) when available bandwidth is fully utilized. When there is enough bandwidth free, than this LLQ limit can overcome.

Let me give you an example to understand better and be able to help me with an advice.

I reserved 10% from the bandwith in LLQ and the rest goes to default class (the packets are marked correctly at network edge). The rest of the QoS is necessary because I have to shape the bandwidth and apply it in a FR interface.

Question, in this scenario, if the bandwitdh is not occupied, can voice calls overcome the 10% bandwidth (from 4Mbit aprox 423kbit). If I use the codec g711 which has about 100k / call , that mean that I can do 4 parallel calls in this 423kbit. What if I make 6,7 calls when the bandwidth if free?

My findings are that if I overcome the 423kbit, then I have packet loss and choopy voice calls. It's like LLQ is not "borrowing" bandwidth from other classes.

Thank you!

If there is no congestion then yes voice calls can indeed use spare bandwidth from other classes. But the key thing to understand is that any spare bandwidth used is not prioritised so you will simply get best effort which means your VOIP calls could become very choppy. It's not that LLQ is not "borrowing" bandwidth, it's that the additional bandwidth being used is not prioritised by LLQ.

So you need to allocate enough bandwidth to the priority queue for the amount of calls you expect to be made.

Jon

Hi Jon,

Thanks for pointing this out, was very helpful. In this scenario, when LLQ queue is taking bandwidth from other classes, it is normal that it's showing drops?

Class-map: voice-ef (match-all)
          21023000 packets, 4121654150 bytes
          30 second offered rate 37000 bps, drop rate 0 bps
          Match:  dscp ef (46)
          Queueing
            Strict Priority
            Output Queue: Conversation 136
            Bandwidth 1024 (kbps) Burst 25600 (Bytes)
            (pkts matched/bytes matched) 21023000/4121654150
            (total drops/bytes drops) 68775/13847017

Thanks!

chiorean.calin wrote:

Hi Jon,

Thanks for pointing this out, was very helpful. In this scenario, when LLQ queue is taking bandwidth from other classes, it is normal that it's showing drops?

Class-map: voice-ef (match-all)
          21023000 packets, 4121654150 bytes
          30 second offered rate 37000 bps, drop rate 0 bps
          Match:  dscp ef (46)
          Queueing
            Strict Priority
            Output Queue: Conversation 136
            Bandwidth 1024 (kbps) Burst 25600 (Bytes)
            (pkts matched/bytes matched) 21023000/4121654150
            (total drops/bytes drops) 68775/13847017

Thanks!

Very interesting question and the short answer is i don't know. I would have though a drop is a drop in that it is not then "passed" on to the other available bandwidth but i'm not sure.

Perhaps if i get the time i'll lab something up to test it.

Jon

Fair answer

To go into more details and to tell you why I have doubts about LLQ taking bandwidth from another class, when there is enough free bandwidth.

I have setup a scenario:

PC1 -> Router1 - > Router2 -> PC2

On R1 I have reserved 400kbit for LLQ and the rest in the default-class (up to 100Mbps). Then with no traffic on the line, I have tested with IPerf with 4 parallel streams, each stream 100k (like a phone call with G711) and everything was fine, IPerf showed no packet loss on those 4 streams.

The I tried with 5,6,7 and so on parallel streams of 100k each, and already I had packet loss and drops on the router. It was like LLQ was not taking that additional 100k (e.g. 5 parallel streams) from the class-default queue but instead put all 5 x 100k into the 400k reservation and drop packets.

I'm confuse a little bit, so please if you have time and find something new, add it here. Maybe it will help others as well.

Thanks!

If anybody is still interested in this subject:

I did some testing in the LAB, 2 routers, back2back connection over Serial interface (diff encaps, HDLC, PPP, FR) and basic QoS config, with LLQ priority up to 400k and burst value calculated automatically.

The I started to push traffic from R1 to R2

- 4 parallel streams, each stream 100k, result no packet drops

- 6 parallel streams, each stream 100k, results random, sometimes not packet loss, sometimes yes (I believe depending on the burst value and serialization delay)

-10 parallel streams, each stream 100k, alway packet drops.

Then I increase the burst value to maximum (I guess value 2000000). With this burst on 10 parallel streams (100k/stream) there was no packet drops. And I have repeated the test couple of times.

It is possible that LLQ borrow bandwidth from other classes (class-default in my case) only up to the limit of the burst value?

I'm still somehow confuse about this topic. I'll read more in the meantime and post here if I find something useful.

Hi,

isn't the shaper shown in your original message still active?

Or some policer?

Could you provide the complete

sh policy-map int ... out

output?

BR,

Milan

d.serra
Level 1
Level 1

Very very interesting.  Just as a side note, run a sniffer to see how big the packets are that you are generating as I have noticed that an LLQ will drop packets bigger then the MTU.  You can test this by using the 'IP MTU {value}' command on your outgoing interface where service-policy is applied.  The lower the MTU, the more packet loss you will have or in your case, the bigger the packets, the more drops you get.

Maybe this has something to do with it?

Also, this issue is not seen in a standard CBWFQ.  Only on LLQ.

Dave

Thank you all!

Below you have all my findings in regard to LLQ. Some of this were confirmed by Cisco guys also.

1. The amount of bandwidth reserved for LLQ has to be calculated in such manner to accommodate most of the matched traffic

- in other words if you have a bandwidth of 10Mbps, and you know that you need for voice traffic 5Mbps, you don't configure LLQ just for 1Mbps in the idea that it will borrow as much as it want if the 10Mbps are not used; this behavior is more specific to PQ

2.  You reserve an amount of bandwidth for LLQ for the case in which bandwidth is fully occupied and when it's empty

- I mean that if you know you need to reserver 5Mbps, do it; don't rely on the fact that it will borrow;

3. LLQ cannot borrow the full bandwidth

- I have observed this in my test, but maybe I didn't had all the necessary tools to have a more accurate result, but somehow it's normal this behavior; you cannot expect to reserver some Kbps for LLQ and then this to go up to Gbps if you have this amount of bandwitdh

4. LLQ borrowing depend on the burst value

- LLQ class can borrow bandwidth in relation to the burst value; if this is higher, LLQ can borrow more bandwidth, but be aware that if you have set manually the burst value and this is too high, it can influence your Tc value, and this will have negative effect on the voice

5. Last but not least, I have read the LLQ is using the method "tokens in the bucket" to borrow bandwidth from other classes

- LLQ can borrow bandwidth for a limited time, equal to the the tokens that it have in the bucket; when tokens are exhausted, the LLQ will be limited to the amount of bandwidth specified in it own class and not more; when the LLQ uses less that the bandwidth amount specified, the tokens are refilled in the bucket; how is refilled and how it is spending tokens...this I didn't found; maybe it's a Cisco secret

I hope this will help others. Final word, configure your LLQ as accurate as you can and don't rely on it's capacity of borrowing bandwidth. This protocol was develop to reserve amount of bandwidth for certain traffic and not to borrow it.

If you find useful, please rate!

Calin

Hi,

IMHO, all what you say is correct.

BUT in a time of congestion.

See http://www.cisco.com/en/US/customer/docs/ios/qos/configuration/guide/congstion_mgmt_oview_ps6350_TSD_Products_Configuration_Guide_Chapter.html#wp1003189

It says:

"Priority traffic metering has the following qualities:

•It is much like the rate-limiting feature of CAR, except that priority traffic metering is only performed under congestion conditions. When the device is not congested, the priority class traffic is allowed to exceed its allocated bandwidth. When the device is congested, the priority class traffic above the allocated bandwidth is discarded.

•It is performed on a per-packet basis, and tokens are replenished as packets are sent. If not enough tokens are available to send the packet, it is dropped.

•It restrains priority traffic to its allocated bandwidth to ensure that nonpriority traffic, such as routing packets and other data, is not starved."

Generally, Cisco documentation is foggy regarding clear statements how QoS works in deep details. Unfortunately even the expensive trainings don't answer questions like this one.

I'm afraid it's a part of Cisco know-how strategy.

BR,

Milan

I totally agree with you:

"When the device is not congested, the priority class traffic is allowed to exceed its allocated bandwidth."

But it's not saying until which limit it can exceed the it's allocated bandwidth. So, back to my example, if you have 10Mbps bandwidth and you set LLQ to 3Mbps, this does not mean that when it's not congestion the LLQ WILL reach 10Mbps. Maybe...depending on the burst and tokens in the bucket system.

At least this is how I see things.

Thanks again!

d.serra
Level 1
Level 1

"3. LLQ cannot borrow the full bandwidth

- I have observed this in my test, but maybe I didn't had all the necessary tools to have a more accurate result, but somehow it's normal this behavior; you cannot expect to reserver some Kbps for LLQ and then this to go up to Gbps if you have this amount of bandwitdh"

This is not correct.  Below you can see that I have a 5 minute average of 30000 bps and yet the LLQ was set to 16k (priority 16).  It is clear that an LLQ can handle more bandwidth then what was allocated.  Below is how I tested this:

Sw1 -> R1 -> R2

From Sw1 I would ping R2 using 1000 packets with 1400 bytes.  I also set the dscp bits in the packets to af41 to ensure they match the LLQ. 

R1 and R2 are connected via a Serial.  

R1#show policy-map interface ser0/0
Serial0/0

  Service-policy output: DSCP-41

    Class-map: DSCP-41 (match-all)
      1000 packets, 1404000 bytes
      5 minute offered rate 30000 bps, drop rate 1000 bps
      Match:  dscp af41 (34)
      Queueing
        Strict Priority
        Output Queue: Conversation 264
       Bandwidth 16 (kbps) Burst 400 (Bytes)
        (pkts matched/bytes matched) 1/1404
        (total drops/bytes drops) 1/1404

    Class-map: class-default (match-any)
      8 packets, 500 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

Configuration for R1 is below...please note that I had also configured NAT just so I would not have to configure routing.  The NAT configuration has no impact on the CBWFQ configuration:

R1#show run
Building configuration...

Current configuration : 1114 bytes
!
version 12.4
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname R1
!
boot-start-marker
boot-end-marker
!
enable password cisco
!
no aaa new-model
no network-clock-participate slot 1
no network-clock-participate wic 0
ip cef
!
!
!
!
no ip domain lookup
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
class-map match-all DSCP-41
match  dscp af41
!
!
policy-map DSCP-41
class DSCP-41
  priority 16
!
!
!
!
!
!
interface FastEthernet0/0
ip address 172.25.0.14 255.255.255.0
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
!
interface Serial0/0
ip address 2.2.2.1 255.255.255.0
ip nat inside
ip virtual-reassembly
service-module t1 timeslots 1-24
service-policy output DSCP-41
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
ip nat inside source static 2.2.2.2 172.25.0.22
!
!
!
!
control-plane
!
!
!
!
!
!
!
!
!
!
line con 0
exec-timeout 0 0
logging synchronous
line aux 0
line vty 0 4
exec-timeout 0 0
no login
transport input telnet ssh
!
!
end

R1#

Thank you for your observation and time spent on this topic.

I still think that I'm somehow right when what I say "LLQ cannot borrow the full bandwidth"

You have set LLQ to 16k and you have 30000bps (which is about 30k) and you already have a drop of 1000bps. If LLQ is allowed to borrow as much bandwidth as it want, why there is packet loss?

I think you know why. Because of the burst limitation and tokens in the bucket system.

If I'm not wrong the bandwidth for a Serial interface default to 1544kbps (in case that bandwidth command is not specified). Now, with your 16k on LLQ try with some tool like IPerf to saturate this 1544kbps with traffic that is matched on the LLQ class, and let me know what is the amount of packet loss that you will have on this class. I think it will be quite high.

" It is clear that an LLQ can handle more bandwidth then what was allocated "

You're right, but I never said that LLQ cannot go above the limit, but how much and for how many time, this depend of the protocol.

Hi,

iI need to refresh my knowledge of the counter meanings here, but:

R1#show policy-map interface ser0/0
Serial0/0

  Service-policy output: DSCP-41

    Class-map: DSCP-41 (match-all)
      1000 packets, 1404000 bytes
      5 minute offered rate 30000 bps, drop rate 1000 bps
      Match:  dscp af41 (34)
      Queueing
        Strict Priority
        Output Queue: Conversation 264
       Bandwidth 16 (kbps) Burst 400 (Bytes)
        (pkts matched/bytes matched) 1/1404
        (total drops/bytes drops) 1/1404

    Class-map: class-default (match-any)
      8 packets, 500 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

IMHO, this might mean:

1000 packets matched the DSCP-41 class, but only 1 packet was placed into the output queue (as the output buffer was full at the moment that packet should be sent out, possibly by other large packets sent out from the serial interface).

And this single packet was dropped - because a 1404-Byte packet coming from LAN might exceed 16 kbps speed and 400Byte burst easily.

IMHO, if you make a test on FastEthernet ports with the same policy, you'll see the priority traffic (matching DSCP-41 class)  would take much higher bandwidth. The drops  would occur only in a case of congestion (i.e., ouput port buffer full) and then the token bucket, etc., would apply.

Edit: But this would never happen on a FastEthernet port.

BR,

Milan

Message was edited by: milan.kulik

d.serra
Level 1
Level 1

"

I still think that I'm somehow right when what I say "LLQ cannot borrow the full bandwidth"

You have set LLQ to 16k and you have 30000bps (which is about 30k) and you already have a drop of 1000bps. If LLQ is allowed to borrow as much bandwidth as it want, why there is packet loss?

I think you know why. Because of the burst limitation and tokens in the bucket system.

If I'm not wrong the bandwidth for a Serial interface default to 1544kbps (in case that bandwidth command is not specified). Now, with your 16k on LLQ try with some tool like IPerf to saturate this 1544kbps with traffic that is matched on the LLQ class, and let me know what is the amount of packet loss that you will have on this class. I think it will be quite high."

Speculation...Speculation.  LLQ dropping 1 out of 1000 packets is hardly a significant number.  The point of what this is to illustrate is that LLQ is not setting a hard limit to 16k and if there is available bandwidth it will use it. The Burst is set to 400k and yet it transfered all but 1 packet each having a size of 1400k.  Burst value is not in play here.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco