Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Community Member

QoS without congestion on ASR 9K

I am seeing behavior in the lab which I don't understand.  I hope someone can help.

 

I have three ASR 9010 running XR 4.2.3.  All have RSP440, MOD-80, and A9K-MPA-4X10GE cards.  Node A has a 10 gigabit ethernet link to node B, and node B has a 10 gigabit ethernet link to node C.  I have test systems connected to A and C, again with 10 gigabit ethernet links.

 

If I configure all the ports with input service policies that only remark packets, and no output service policies, I can pass traffic loads from 5 to 9 gigabits, at various small packet sizes, without losing any datagrams.  If I add an output service policy to the port on C linked to a test system:

 

policy-map test
 class VOICE
  priority level 1 
  police rate percent 5 
   conform-action transmit
   exceed-action drop
  !
 !
 class BETTER
  bandwidth percent 35 
 !
 class GOOD
  bandwidth percent 45 
 !
 class class-default
  random-detect discard-class 2 4096 bytes 8192 bytes 
  random-detect discard-class 1 256 bytes 8192 bytes 
  queue-limit 8192 bytes 
 !
 end-policy-map

 

then I get packet drops, both from RED and from tail drops.

 

This may seem perfectly fine to you, until you remember that I'm testing the interfaces at less than 100%.  Since there is no congestion, why is the output service policy having any impact at all?

Bitwrangler for FirstLight Fiber Corp
1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

ah that helps ERM!ok so here

ah that helps ERM!

ok so here is the deal, by default WRED kicks in at the queue limit as the "max" value and the min is 1/2* the queue limit. If the queue limit is not defined on a class, 100msec as default is taken for that value.

Now you ended up with a queue limit of only 8k, which is very small obviously. And as you can see in the output, the instant queue length is 30 (that is instant read out of the hardware that moment), so there were packets in the queue. That means that the WRED has kicked in and there will be a drop probability assigned to the packet and it might get wacked, and that is what you are seeing here.

Things you can do here is adjust your WRED curves to get less likely drops, adjust the queue limit of the default class (which will also help reducing the drop probability), fact of the matter is your WRED current config is very "tight" which means that if there is the slightest amount of buffering going on, WRED will immediately kick in.

cheers!

xander

Xander Thuijs CCIE #6775 Principal Engineer ASR9000, CRS, NCS6000 & IOS-XR
5 REPLIES
Cisco Employee

hi evan,can you show us the

hi evan,

can you show us the show policy-map interface <name> out from the C device to the tester?

also show me a show qos int te<if of C to tester> out

this way I can see what the hw programmed (with the 2nd command) and what the policy-map is doing.

regards!

xander

Xander Thuijs CCIE #6775 Principal Engineer ASR9000, CRS, NCS6000 & IOS-XR
Community Member

As you requested: RP/0/RSP0

As you requested:

 

RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#show int te0/7/0/3
TenGigE0/7/0/3 is up, line protocol is up 
  Interface state transitions: 3
  Hardware is TenGigE, address is d867.d94d.e0f3 (bia d867.d94d.e0f3)
  Layer 1 Transport Mode is LAN
  Internet address is 64.17.123.5/30
  MTU 9100 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
     reliability 255/255, txload 196/255, rxload 196/255
  Encapsulation ARPA,
  Full-duplex, 10000Mb/s, link type is force-up
  output flow control is off, input flow control is off
  loopback not set,
  ARP type ARPA, ARP timeout 04:00:00
  Last input 00:00:00, output 00:00:00
  Last clearing of "show interface" counters 00:03:19
  5 minute input rate 7698307000 bits/sec, 6776681 packets/sec
  5 minute output rate 7697799000 bits/sec, 6776233 packets/sec
     1346430692 packets input, 191193158118 bytes, 0 total input drops
     0 drops for unrecognized upper-level protocol
     Received 0 broadcast packets, 0 multicast packets
              0 runts, 0 giants, 0 throttles, 0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     1346259548 packets output, 191168855600 bytes, 0 total output drops
     Output 0 broadcast packets, 0 multicast packets
     0 output errors, 0 underruns, 0 applique, 0 resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions


RP/0/RSP0/CPU0:NodeC#show qos inter te0/7/0/3 output
Interface: TenGigE0_7_0_3 output 
Bandwidth configured: 10000000 kbps Bandwidth programed: 10000000 kbps
ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps
Port Shaper programed in HW: 0 kbps 
Policy: core-link-10g-intrapop Total number of classes: 4
----------------------------------------------------------------------
Level: 0 Policy: core-link-10g-intrapop Class: qos5dscpEF
QueueID: 196648 (Priority 1)
Queue Limit: 6272 kbytes Abs-Index: 111 Template: 0 Curve: 6
Shape CIR Profile: INVALID
Policer Profile: 47 (Single)
Conform: 500000 kbps (5 percent) Burst: 6250000 bytes (0 Default)
Child Policer Conform: TX
Child Policer Exceed: DROP
Child Policer Violate: DROP
----------------------------------------------------------------------
Level: 0 Policy: core-link-10g-intrapop Class: qos3dscpBetter
QueueID: 196650 (Priority Normal)
Queue Limit: 52224 kbytes Abs-Index: 158 Template: 0 Curve: 0
Shape CIR Profile: INVALID
WFQ Profile: 3/204 Committed Weight: 877 Excess Weight: 877
Bandwidth: 3500000 kbps, BW sum for Level 0: 8000000 kbps, Excess Ratio: 1
----------------------------------------------------------------------
Level: 0 Policy: core-link-10g-intrapop Class: qos2dscpGood
QueueID: 196651 (Priority Normal)
Queue Limit: 65536 kbytes Abs-Index: 163 Template: 0 Curve: 0
Shape CIR Profile: INVALID
WFQ Profile: 3/219 Committed Weight: 1119 Excess Weight: 1119
Bandwidth: 4500000 kbps, BW sum for Level 0: 8000000 kbps, Excess Ratio: 1
----------------------------------------------------------------------
Level: 0 Policy: core-link-10g-intrapop Class: class-default
QueueID: 196652 (Priority Normal)
Queue Limit: 8 kbytes (8192 bytes) Abs-Index: 6 Template: 0 Curve: 0
Shape CIR Profile: INVALID
WFQ Profile: 3/0 Committed Weight: 1 Excess Weight: 1
Bandwidth: 0 kbps, BW sum for Level 0: 8000000 kbps, Excess Ratio: 1
WRED Type: Discard-class based Table: 0 Curves: 3
Default RED Curve Profile: 6/0/0 Thresholds Min : 7 (8) kbytes Max: 8 (8) kbytes
WRED Curve: 1 Profile: 6/7/4 Thresholds Min : 3 (4) kbytes Max: 8 (8) kbytes
 Match: 2 
WRED Curve: 2 Profile: 6/15/3 Thresholds Min : 0 (0) kbytes Max: 8 (8) kbytes
 Match: 1 
----------------------------------------------------------------------
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#show policy-map int te0/7/0/3

TenGigE0/7/0/3 input: input-classify

Class exp1
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class exp2
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class exp3
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class exp4
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class exp5
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class exp6
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class exp7
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class dscpEF
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                 194/29458                0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class dscpScavenger
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class dscpGood
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class dscpBetter
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         : N/A 
    Total Dropped       : N/A 
Class class-default
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :       1735410808563/282710291507630      7923363
    Transmitted         : N/A 
    Total Dropped       : N/A 

TenGigE0/7/0/3 output: core-link-10g-intrapop

Class qos5dscpEF
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         :                   0/0                    0
    Total Dropped       :                   0/0                    0
  Policing statistics                (packets/bytes)     (rate - kbps) 
    Policed(conform)    :                   0/0                    0
    Policed(exceed)     :                   0/0                    0
    Policed(violate)    :                   0/0                    0
    Policed and dropped :                   0/0                  
  Queueing statistics
    Queue ID                             : 196648 
    High watermark  (Unknown)            
    Inst-queue-len  (packets)            : 0
    Avg-queue-len   (Unknown)            
    Taildropped(packets/bytes)           : 0/0
    Queue(conform)      :                   0/0                    0
    Queue(exceed)       :                   0/0                    0
    RED random drops(packets/bytes)      : 0/0

Class qos3dscpBetter
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         :            21893601/3196465746           0
    Total Dropped       :                   0/0                    0
  Queueing statistics
    Queue ID                             : 196650 
    High watermark  (Unknown)            
    Inst-queue-len  (packets)            : 0
    Avg-queue-len   (Unknown)            
    Taildropped(packets/bytes)           : 0/0
    Queue(conform)      :            21893601/3196465746           0
    Queue(exceed)       :                   0/0                    0
    RED random drops(packets/bytes)      : 0/0

Class qos2dscpGood
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                   0/0                    0
    Transmitted         :                   0/0                    0
    Total Dropped       :                   0/0                    0
  Queueing statistics
    Queue ID                             : 196651 
    High watermark  (Unknown)            
    Inst-queue-len  (packets)            : 0
    Avg-queue-len   (Unknown)            
    Taildropped(packets/bytes)           : 0/0
    Queue(conform)      :                   0/0                    0
    Queue(exceed)       :                   0/0                    0
    RED random drops(packets/bytes)      : 0/0

Class class-default
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :          1475073517/215360733700         7919974
    Transmitted         :          1470089735/214633101528         7918540
    Total Dropped       :              271052/39573592             1519
  Queueing statistics
    Queue ID                             : 196652 
    High watermark  (Unknown)            
    Inst-queue-len  (packets)            : 30
    Avg-queue-len   (Unknown)            
    Taildropped(packets/bytes)           : 0/0
    Queue(conform)      :          1470089735/214633101528         7918540
    Queue(exceed)       :                   0/0                    0
    RED random drops(packets/bytes)      : 271052/39573592

    WRED profile for WRED Curve 1
    RED Transmitted (packets/bytes)      : N/A
    RED random drops(packets/bytes)      : 271052/39573592
    RED maxthreshold drops(packets/bytes): N/A 
    WRED profile for WRED Curve 2
    RED Transmitted (packets/bytes)      : N/A
    RED random drops(packets/bytes)      : 0/0
    RED maxthreshold drops(packets/bytes): N/A 
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#

 

In this particular run, taildrops haven't yet occurred, but RED drops have.  If the interface was actually congested, this would make complete sense.  As the show interface output indicates, I'm not even at 80% load, so I don't understand why the service-policy is active.

 

I'll also add that these are JDSU test sets generating constant load traffic on a single flow, so I don't think it should be a "microburst" situation.

 

Thanks for taking a look.

 

ERM

Bitwrangler for FirstLight Fiber Corp
Cisco Employee

ah that helps ERM!ok so here

ah that helps ERM!

ok so here is the deal, by default WRED kicks in at the queue limit as the "max" value and the min is 1/2* the queue limit. If the queue limit is not defined on a class, 100msec as default is taken for that value.

Now you ended up with a queue limit of only 8k, which is very small obviously. And as you can see in the output, the instant queue length is 30 (that is instant read out of the hardware that moment), so there were packets in the queue. That means that the WRED has kicked in and there will be a drop probability assigned to the packet and it might get wacked, and that is what you are seeing here.

Things you can do here is adjust your WRED curves to get less likely drops, adjust the queue limit of the default class (which will also help reducing the drop probability), fact of the matter is your WRED current config is very "tight" which means that if there is the slightest amount of buffering going on, WRED will immediately kick in.

cheers!

xander

Xander Thuijs CCIE #6775 Principal Engineer ASR9000, CRS, NCS6000 & IOS-XR
Community Member

TAC just responded to my case

TAC just responded to my case (Mark Sanchez for the win).  He reminds me that of course RED is dropping packets when the port is less than 100%; that's what RED is meant to do.

Sometimes you just need someone to tell you what's already in front of your eyes.

 

ERM

 

Bitwrangler for FirstLight Fiber Corp
Cisco Employee

yup yup indeed! :) there is a

yup yup indeed! :) there is a great Dutch saying: "double screwed is fixed for sure"

applies here also :) glad it is resolved ERM!

cheers

xander

Xander Thuijs CCIE #6775 Principal Engineer ASR9000, CRS, NCS6000 & IOS-XR
290
Views
5
Helpful
5
Replies
CreatePlease to create content