3560E output drops without QOS enabled

rejackson · ‎08-27-2013

I have 3560E 48 1gb port switches. No port is able to get over 450 mb and they show millions of output drops. MLS QOS is not enabled. These switches have file servers on them. This is just one port. The drops go up 100s every few seconds.

Cab1.1.SW1#sh int gi1/1

GigabitEthernet1/1 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is c464.132b.04b1 (bia c464.132b.04b1)

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 36/255, rxload 45/255

Encapsulation ARPA, loopback not set

Keepalive not set

Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters 6w2d

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 96129877

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 178329000 bits/sec, 24382 packets/sec

5 minute output rate 142493000 bits/sec, 22821 packets/sec

82123761686 packets input, 52635032674394 bytes, 0 no buffer

Received 106896520 broadcasts (67802176 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 67802176 multicast, 0 pause input

0 input packets with dribble condition detected

98589773767 packets output, 115123919773878 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Cab1.1.SW1#sh platform port-asic stat drop gi1/1

Interface Gi1/1 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 136
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 227790796
    Queue 4
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 5
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 6
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 7
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

Cab1.1.SW1#sh platform port-asic stat enqu gi1/1

Interface Gi1/1 TxQueue Enqueue Statistics
    Queue 0
        Weight 0 Frames 0
        Weight 1 Frames 0
        Weight 2 Frames 0
    Queue 1
        Weight 0 Frames 187615
        Weight 1 Frames 5905942
        Weight 2 Frames 11185165
    Queue 2
        Weight 0 Frames 0
        Weight 1 Frames 0
        Weight 2 Frames 0
    Queue 3
        Weight 0 Frames 0
        Weight 1 Frames 0
        Weight 2 Frames 3446387190
    Queue 4
        Weight 0 Frames 0
        Weight 1 Frames 0
        Weight 2 Frames 0
    Queue 5
        Weight 0 Frames 0
        Weight 1 Frames 0
        Weight 2 Frames 0
    Queue 6
        Weight 0 Frames 0
        Weight 1 Frames 0
        Weight 2 Frames 0
    Queue 7
        Weight 0 Frames 0
        Weight 1 Frames 0
        Weight 2 Frames 0

I see lots of posts about drops with QOS enabled. Nothing where is it not. It is apparent that classification and queueing is still going on. It must be COS. It looks like Queue 3 is hitting a threshhold. I dont want to play with multiple queues I just want it turned off and all the buffers in one queue like they say it is when you turn off MLS.

Joseph W. Doherty · ‎08-27-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Are different queues ASIC stats incrementing with QoS disabled? Reason I ask, unlike the interface counters, I believe the ASIC counters are only cleared with a reboot.

What IOS version are you running?

rejackson · ‎08-27-2013

Thanks for the interest Joseph,

Yes they are still going up

Cab1.1.SW1#sh platform port-asic stat drop gi1/1

Interface Gi1/1 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 136
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 227823774
    Queue 4
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

Cab1.1.SW1# sh ver

Cisco IOS Software, C3560E Software (C3560E-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)

Cab1.1.SW1# sh mls qos

QoS is disabled

QoS ip packet dscp rewrite is enabled

Cab1.1.SW1#sh mls qos interface gi1/1
GigabitEthernet1/1
QoS is disabled. When QoS is enabled, following settings will be applied
trust state: not trusted
trust mode: not trusted
trust enabled flag: ena
COS override: dis
default COS: 0
DSCP Mutation Map: Default DSCP Mutation Map
Trust device: none
qos mode: port-based

This is from a doc on 3750 switches but it really sounds like what is happening.

Queue Thresholds

The Cisco Catalyst 3750 Series Switch supports Weighted Tail Drop (WTD) with three thresholds. Two of the

thresholds can be configured by the user; the third is set to 100 percent. Packets that exceed the specified thresholds

are dropped.

Packets are queued-based on the quality of service (QoS) label. For example, in Figure 2 below, packets with class of

service (CoS) values of 0 to 4 are assigned to a 30-percent threshold, packets with a CoS value of 5 are assigned to a

100-percent threshold, and packets with CoS values of 6 or 7 are assigned to a 70-percent threshold. When packets

fill 30 percent of the queue, packets with CoS values of 0 to 4 are dropped. When packets fill 70 percent of the queue, packets with CoS values of 6 or 7 are dropped. Packets with a CoS value of 5 are not dropped unless the queue is

full. Typically, voice packets are assigned a CoS value of 5. Control traffic and video traffic are assigned CoS values

of 6 or 7, and low-priority data traffic is assigned CoS values 0 to 4.

Leo Laohoo · ‎08-27-2013

12.2(58)SE2

Wow! That's one of the few IOS I would avoid at all cost.

Unless there are specific features you need, try 12.2(55)SE8.

Joseph W. Doherty · ‎08-28-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

As Leo notes, you may want to change your IOS version. The one Leo recommends is considered very solid. (Unsure it's available for a 3560E, but you might also consider 15.0(2)SE or later.)

My understanding is, with QoS disabled all queuing resources (i.e. buffers) should be available. If this is correct (i.e. assuming your not dealing with some bug), enabling QoS and buffer tuning, as Dmytro suggests, I wouldn't expect to a decrease in drops. However, Dmytro is correct in that all 3560/3750 are somewhat lacking on buffer resources and if your traffic is bursty enough, you'll see drops. (NB: Cisco recently published a note about buffer usage on 3750s (specific mention of the 3750X - applies to -E too?, where they noted the uplinks ports have more RAM than the ordinary ports. If you have a really busy port, and you have some uplinks ports that are available, you might try a copper SFP on one of those ports.)

PS:

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_qanda_item09186a0080c13273.shtml

Dmytro Skotnikov · ‎08-27-2013

Output drops you are observing on the switch is most likely due to a bursty traffic nature, the average traffic rate statistics on the interface won't allow you to catch the time of burst and you can evaluate traffic capture on the port in order to verify if it's due to a burst.

Another point to be aware is small egress buffers on 3560E box which can be a limitation in case of bursty traffic and are not sufficient to accomodate traffic spikes. There is no difference when QoS is disabled or enabled, since the egress logic does not change, so basically we just setup default setting for egress queues and got less number of queues.

I would suggest you to enable "mls qos" and tune egress buffers for a particular interface in order to reduce drop count, but be aware egress buffers are derived from ASICs buffer and are shared between ports.

There is a good writeup to take a look for a better qos understanding on C3K units:

http://www.cisco.com/en/US/partner/products/hw/switches/ps5023/products_tech_note09186a0080883f9e.shtml

--
Best regards,
Dmitry Skotnikov

-- Best regards, Dmitry Skotnikov

rejackson · ‎08-28-2013

I watch both sides of the connection and while packets are dropping neither side shows more than 27% utilization. I cant believe the bursty traffic scenario. I know it can happen but it is not causing this constant packet loss.

Do these complaints about the IOS version have anything to do with the problems I am describing?

Joseph W. Doherty · ‎08-28-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

27% utilization measured over what time interval?

Bursting, especially microbursting, can overflow queues during fractional seconds.

The 3750X is documented as having 2 Mbps per 24 copper ports. (Let's assume 3750E is similar.)

If just two gig ports were sending to single gig port, it would only take (roughly) less than .2 sec to fill 2 MB of buffers (assuming my math is correct).

Leo Laohoo · ‎08-28-2013

Do these complaints about the IOS version have anything to do with the problems I am describing?

Yes and no. For the sake of stability in your network, YES. In regards to the issues you are facing, maybe. Maybe because it could potentially be.

NOTE: IOS version 15.2(1)E is out TODAY, 28 August 2013. Don't be tempted to use this as no one has tested this as of yet.