Output drops due to QoS on 2960/3560/3750 switches

Blog

Apr 8, 2011 2:15 AM
Apr 8th, 2011

Do you see incrementing output drops on some interfaces after configuring QoS on your 2960/3560/3750 switch?

Common Scenarios

Some of the interfaces start experiencing output drops once QoS is configured on the switch.

Specific applications may experience degraded performance after configuring QoS on the switch. Say IP phones start experiencing choppy calls.

Possible Reason

Once you enable QoS on the switch, some traffic may start getting lesser resources than before (bandwidth or buffer) and hence may get dropped on the switch.



Troubleshooting steps


Step1> Identify the interfaces which carry outgoing data for the affected application or are seeing incrementing output drops. Compare the interface output rate and the interface speed and ensure that the drops are not due to overutilization of the link.


Switch#sh int gi1/0/1

<some output ommitted>

GigabitEthernet0/1 is up, line protocol is up (connected)

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX


!interface speed is 1000 mbps


  input flow-control is off, output flow-control is unsupported

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1089  <<---


!ensure these drops are incrementing


  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 4000 bits/sec, 6 packets/sec

  5 minute output rate 3009880 bits/sec, 963 packets/sec


!output rate is about 3 mbps while the interface speed is 1000 mbps.

Step2> Ensure that QoS is enabled on the switch. If it is not enabled, output drops are not related to QoS and hence further steps mentioned here are irrelevant.


Switch#sh mls qos

QoS is enabled  <<----

QoS ip packet dscp rewrite is enabled


Step3> Identify the marking of the outgoing traffic that is getting dropped on the interface.


Switch#sh mls qos int gi1/0/1 statistics


GigabitEthernet1/0/1 (All statistics are in packets)


  dscp: incoming

-------------------------------


0 -  4 :           0            0            0            0            0

5 -  9 :           0            0            0            0            0

10 - 14 :           0            0            0            0            0

15 - 19 :           0            0            0            0            0

20 - 24 :           0            0            0            0            0

25 - 29 :           0            0            0            0            0

30 - 34 :           0            0            0            0            0

35 - 39 :           0            0            0            0            0

40 - 44 :           0            0            0            0            0

45 - 49 :           0       198910            0            0            0

50 - 54 :           0            0            0            0            0

55 - 59 :           0            0            0            0            0

60 - 64 :           0            0            0            0

  dscp: outgoing

-------------------------------


0 -  4 :           0            0            0            0            0

5 -  9 :           0            0            0            0            0

10 - 14 :           0            0            0            0            0

15 - 19 :           0            0            0            0            0

20 - 24 :           0            0            0            0            0

25 - 29 :           0            0            0            0            0

30 - 34 :           0            0            0            0            0

35 - 39 :           0            0            0            0            0

40 - 44 :           0            0            0            0            0

45 - 49 :           0      248484            0            0            0

50 - 54 :           0            0            0            0            0

55 - 59 :           0            0            0            0            0

60 - 64 :           0            0            0            0

  cos: incoming

-------------------------------


  0 -  4 :           2            0            0            0            0

  5 -  7 :           0            0            0

  cos: outgoing

-------------------------------


  0 -  4 :           0            0            0            0            0

  5 -  7 :           0            0            0

  output queues enqueued:

queue:    threshold1   threshold2   threshold3

-----------------------------------------------

queue 0:           248484      0           0

queue 1:           0           0           0

queue 2:           0           0           0

queue 3:           0           0           0


  output queues dropped:

queue:    threshold1   threshold2   threshold3

-----------------------------------------------

queue 0:       1089           0           0

queue 1:           0           0           0

queue 2:           0           0           0

queue 3:           0           0           0


Policer: Inprofile:            0 OutofProfile:            0


Note: Though you see queue 0-threshold 1 dropping packets, this actually will be queue 1 in further troubleshooting as queue numbering is 1 to 4 in further outputs.



Step4> Check the marking to output-q map on the switch to determine which queue-threshold pair

maps to the marking getting dropped.


In this scenario, queue1-threshold1 is mapped to dscp 46, which is getting dropped on the interface. This means that dscp 46 traffic is being sent to queue1 and is getting dropped because that queue has insufficient buffer or lesser CPU cycles.


Switch#sh mls qos maps dscp-output-q


   Dscp-outputq-threshold map:

     d1 :d2    0     1     2     3     4     5     6     7     8     9

     ------------------------------------------------------------

      0 :    02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01

      1 :    02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01

      2 :    03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01

      3 :    03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01

      4 :    01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01

      5 :    04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01

      6 :    04-01 04-01 04-01 04-01



Step5> There are two ways to tackle these drops. First method is by changing the buffer and threshold values for the queue dropping packets. second method is to configure the scheduler so that the queue dropping packets is serviced more often than rest of the queues.

First let us see how we can change the buffer and threshold for the affected queues. Let us check the buffer and threshold values associated with the queue identified in step 4.

Note: Each queue set has the option to configure the buffer size and threshold value for the four egress queues. Then, you can apply any one of the queue sets to any of the ports. By default, all interfaces use queue-set 1 for output queues unless explicitly configured to use queue-set 2.


In this scenario queue 1 in queue-set 1 has 25% of the total buffer space and threshold 1 is set to 100%


Switch#sh mls qos queue-set

Queueset: 1

Queue     :       1       2       3       4

----------------------------------------------

buffers   :      25      25      25      25

threshold1:     100     200     100     100

threshold2:     100     200     100     100

reserved  :      50      50      50      50

maximum   :     400     400     400     400

Queueset: 2

Queue     :       1       2       3       4

----------------------------------------------

buffers   :      25      25      25      25

threshold1:     100     200     100     100

threshold2:     100     200     100     100

reserved  :      50      50      50      50

maximum   :     400     400     400     400



Step6> If you wish to change the buffer and threshold values just for the affected interface, please change queue-set 2 and configure the affected interface to use queue-set 2.


Note: You can change queue-set 1 also but as all the interfaces by default use queue-set 1, the change will be reflected to all the interfaces.


In the next step I am changing queue-set 2 so that queue 1 gets 70% of total buffer.

Switch(config)#mls qos queue-set output 2 buffers 70 10 10 10


In next step i am changing queue-set 2, queue 1 thresholds. Both Threshold 1 and Threshold 2 are mapped to 3100 so that they can pull buffer from the reserved pool if required.

Switch(config)#mls qos queue-set output 2 threshold 1 3100 3100 100 3200

Step7> Check if the changes reflect under correct queue and queue-set.


Switch#sh mls qos queue-set

Queueset: 1

Queue     :       1       2       3       4

----------------------------------------------

buffers   :      25      25      25      25

threshold1:     100     200     100     100

threshold2:     100     200     100     100

reserved  :      50      50      50      50

maximum   :     400     400     400     400

Queueset: 2

Queue     :       1       2       3       4

----------------------------------------------

buffers   :      70      10      10      10

threshold1:    3100     100     100     100

threshold2:    3100     100     100     100

reserved  :     100      50      50      50

maximum   :    3200     400     400     400



Step8> Make the affected interface use queue-set 2 so that the changes take effect on this interface.


Switch(config)#int gi1/0/1

Switch(config-if)#queue-set 2

Switch(config-if)#end


Confirm if the interface is mapped to queue-set 2

Switch#sh run int gi1/0/1

interface GigabitEthernet1/0/1

switchport mode access

mls qos trust dscp

queue-set 2

end

Check if the interface is still dropping packets.

Step9> We can also configure the scheduler to increase the rate at which queue 1 will be serviced using the share and shape options. In this example Queue 1 alone will get 50% of the total CPU cycles and rest three queues will collectively get 50% of the CPU cycles.

Switch(config-if)#srr-queue bandwidth share 1 75 25 5


Switch(config-if)#srr-queue bandwidth shape  2  0  0  0

Check if the interface is still dropping packets.

Step10> If the packets are still getting dropped, as a last resort we can enable priority queue on this interface. This will ensure that all the traffic in the priority queue to gets processed before any other queue.

Note:Priority queue is serviced until empty before the other queues are serviced.By default on 2960/3560/3750 switches, queue 1 is the priority queue.

Switch(config)#int gi1/0/1

Switch(config-if)#priority-queue out

Switch(config-if)#end

The marking getting dropped on the interface can be mapped so that it goes to queue 1, which is now the priority queue . In this way we ensure that traffic with this marking always gets processed before anything else.

Switch(config)#mls qos srr-queue output dscp-map queue 1 threshold 1 ?

<0-63>  dscp values separated by spaces (up to 8 values total)


For detailed configuration examples and command syntax please visit
http://tinyurl.com/7zvf9o

**************************************************************************************************

COMMENTS? QUESTIONS? SUGGESTIONS?


Please sign in, and enter your feedback below!  Thank you!


Average Rating: 5 (2 ratings)

Actions

Login or Register to take actions

This Blog

Posted April 8, 2011 at 2:15 AM
Stats:
Comments:0 Avg. Rating:5
Views:17483   
Shares:0
Tags: switching
+

Related Content

Blogs Leaderboard

Rank Username Points
1 5
Rank Username Points
5