Unexplained output drop on Cat3750

[jfcourteau]

Hello,

I have been trying to troubleshoot an output drop issue on my Cat3750G-24TS switch and can't figure out what is causing output drops.

Any input from you guys would be very appreciated!

Here is some information from my switch:

Switch#show interfaces counters errors

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards

Gi1/0/1             0           1           0           1          0            0

Gi1/0/2             0           1           0           1          0         9203

Gi1/0/3             0           0           0           0          0            0

Gi1/0/4             0           0           0           0          0       194431

Gi1/0/5             0           0           0           0          0        29443

Gi1/0/6             0           0           0           0          0        51055

Gi1/0/7             0           1           0           1          0       263160

Gi1/0/8             0           0           0           0          0            0

Gi1/0/9             0           0           0           0          0       932177

Gi1/0/10            0           0           0           0          0            0

Gi1/0/11            0           0           0           0          0            0

Gi1/0/12            0           0           0           0          0            0

Gi1/0/13            0           0           0           0          0        80123

Gi1/0/14            0           0           0           0          0         8541

Gi1/0/15            0           0           0           0          0         5011

Gi1/0/16            0           0           0           0          0            0

...

Port G1/0/13 is the most problematic interface, output drops counter keep going up at a rapid pace everyting time I do a show int (around 1% of packets are dropped) :

GigabitEthernet1/0/13 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 0024.9856.688d (bia 0024.9856.688d)

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 131/255, rxload 23/255

  Encapsulation ARPA, loopback not set

  Keepalive not set

  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output never, output hang never

  Last clearing of "show interface" counters 00:33:20

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 86920

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  30 second input rate 9317000 bits/sec, 2860 packets/sec

  30 second output rate 51572000 bits/sec, 5940 packets/sec

     5963933 packets input, 2489026935 bytes, 0 no buffer

     Received 3 broadcasts (3 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 3 multicast, 0 pause input

     0 input packets with dribble condition detected

     12710888 packets output, 13628047061 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

The interface never goes overs 60% utilization so I don't see why I would get any drops:

switch# show controllers g1/0/13 utilization

Receive Bandwidth Percentage Utilization   : 9

Transmit Bandwidth Percentage Utilization  : 46

I've already tried upgrading the IOS to 12.2.55SE4 and i'm still getting those.

I'm sure im missing something...

Regards,

John

[Leo Laohoo]

reliability 255/255, txload 131/255, rxload 23/255
Transmit Bandwidth Percentage Utilization  : 46

That's about right.  51.37%.

You may want to enable QoS on the interface.  Whatever is connected to Gi1/0/13 can't take the amount of data the switch is pushing (at 100 Mbps), hence the switch is dropping them.

[nkarpysh]

Agree with Leo.

50% rate is just an average for 30 seconds. Traffic is never stable - so that can indeed go down in a certain moment - then that would mean that it can spike in the other one to sustain the rate of 50%. Those spike force the port to queue the triffic in the output buffer and if that is full to drop eventually. Those drops you see os out-discards.

QoS can help here as it not only allow you to spread traffic among the queues and give particular one a priority - but that laso allow you to fine tune your buffers depth also permitting those to take additional buffer from common pull.

Good Egress QoS  explanation can be found here:

https://supportforums.cisco.com/docs/DOC-8093

Nik

[ninoroygaleos]

Hi Jean-Francois Courteau,

1. your keepalive status should be set.
2. media type should be match with the other end, for instance, you may have 100MB/s on your SWs' port the other end should also be 100MB/s, if 1000MB/s then the other end shouls also be 1000MB/s. and don't try to use the autonegotiate just set it to a specific speed and should be at the full-duplex to both ends. That's what i have found from CISCO, I've been encountered same issue you have had when I was on the network before.

[jfcourteau]

Hello again,

Thanks for the replies!

I did some more testing on my switch and found out that disabling QOS on it stopped the output drops from hapenning completely.

I do run QOS with priority queuing on some interfaces that have VOIP traffic passing through.

Do you guys think I should increase the buffers?

I found this exemple in the link provided by Nik:

Switch#config term

Switch(config)#mls qos queue-set output 1 threshold 1 1000 1000 50 1000

Switch(config)#mls qos queue-set output 1 threshold 2 1000 1000 50 1000

Switch(config)#mls qos queue-set output 1 threshold 3 1000 1000 50 1000

Switch(config)#mls qos queue-set output 1 threshold 4 1000 1000 50 1000

Not sure what effect it will have on my 3750 switch?

Thanks!

John

[paolo bevilacqua]

You do not need any QoS on LAN switches, even if your run VoIP, and no matter what people of small experience will tell you.

As you noticed yourself, QoS only causes drops. Then tuning these values is another source of trouble.

Disable QoS, and you will be fine.

[ninoroygaleos]

Hi Jean-Francois Courteau,

Yes, you may disable your Qos on the LAN. I believe the best practices when applying Qos is by internet level or external level not on the LAN.

-onin

[nkarpysh]

Nice comment from Paolo

Though I need to say that i my practise I've seen both. Somietime QoS enabled cause the drops with buffers/queues not being properly tuned. And I agree fine tuning might be a pain and not clear at all. However default performance and buffer capacity of the 3750 switch is quite low and I saw many cases when the default parameters affected throughput heavily due to exausted buffers.

So buffer tuning can make a good trick.

John you may check

show mls qos int gi1/0/13

and

show queueing int gi1/0/13 to see whcih queue is dropping most.

Then you can try disabling QoS globally to see if that is giving improvement. If not or if  you still need to have some particular traffic prioritized then you can enable QoS back and tune the buffers.

the values you found above for buffer tuning are possibly too much for begining as it will also cause the commong bufer to be exausted more quickly thus other ports may not be able to get from it. So just increase that slightly for the queue which drop and monitor.

nik