Cisco 3560E Output drops

[chris-gibbs]

Hey all,

We are receiving a large amount of output drops on 4 interfaces that connect to microwave wireless.

This is playing up with our Management system. Saw a bug report but that was for 3750, not the IOS version  but was a similar issue.

Switch Ports Model              SW Version            SW Image

------ ----- -----              ----------            ----------

*    1 30    WS-C3560E-24TD     12.2(46)SE            C3560E-UNIVERSALK9-M

The output errors tend to happen at the same time. All ports are trunks however there are other trunks on the switch that are not having these issues.

GigabitEthernet0/4 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 53717

GigabitEthernet0/5 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 53732

GigabitEthernet0/6 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 50007

GigabitEthernet0/7 is up, line protocol is up (connected)

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 54484

Gi 0/4 is configured:

switchport trunk encapsulation dot1q

switchport mode trunk

srr-queue bandwidth share 10 10 60 20

queue-set 2

priority-queue out

mls qos trust dscp

auto qos voip trust

We have other switches running a similar config and connected to microwave equipment without these errors.

Looking into the issue we noticed that the actual drop counts in ASIC are not increasing., port 2 (gi0/4) show, the rest similar

RTR1#sh platform port-asic stats drop port 2

Port-asic Port Drop Statistics - Summary

========================================

  RxQueue Drop Statistics Slice0

   RxQueue 0 Drop Stats Slice0: 0

   RxQueue 1 Drop Stats Slice0: 0

   RxQueue 2 Drop Stats Slice0: 0

   RxQueue 3 Drop Stats Slice0: 0

  RxQueue Drop Statistics Slice1

   RxQueue 0 Drop Stats Slice1: 0

   RxQueue 1 Drop Stats Slice1: 0

   RxQueue 2 Drop Stats Slice1: 0

   RxQueue 3 Drop Stats Slice1: 0

  Port  0 TxQueue Drop Stats: 0

  Port  1 TxQueue Drop Stats: 0

  Port  2 TxQueue Drop Stats: 0

  Port  3 TxQueue Drop Stats: 0

  Port  4 TxQueue Drop Stats: 0

  Port  5 TxQueue Drop Stats: 0

  Port  6 TxQueue Drop Stats: 0

  Port  7 TxQueue Drop Stats: 0

  Port  8 TxQueue Drop Stats: 0

  Port  9 TxQueue Drop Stats: 0

  Port 10 TxQueue Drop Stats: 0

  Port 11 TxQueue Drop Stats: 0

  Port 12 TxQueue Drop Stats: 0

  Port 13 TxQueue Drop Stats: 0

  Port 14 TxQueue Drop Stats: 0

  Port 15 TxQueue Drop Stats: 0

  Port 16 TxQueue Drop Stats: 0

  Port 17 TxQueue Drop Stats: 0

  Port 18 TxQueue Drop Stats: 0

  Port 19 TxQueue Drop Stats: 0

  Port 20 TxQueue Drop Stats: 0

  Port 21 TxQueue Drop Stats: 0

  Port 22 TxQueue Drop Stats: 0

  Port 23 TxQueue Drop Stats: 0

  Port 24 TxQueue Drop Stats: 0

  Port 25 TxQueue Drop Stats: 0

  Port 26 TxQueue Drop Stats: 0

  Port 27 TxQueue Drop Stats: 0

  Supervisor TxQueue Drop Statistics

    Queue  0: 0

    Queue  1: 0

    Queue  2: 0

    Queue  3: 0

    Queue  4: 0

    Queue  5: 0

    Queue  6: 0

    Queue  7: 0

    Queue  8: 0

    Queue  9: 0

    Queue 10: 0

    Queue 11: 0

    Queue 12: 0

    Queue 13: 0

    Queue 14: 0

    Queue 15: 0

Port-asic Port Drop Statistics - Details

========================================

  RxQueue Drop Statistics Slice0

      Queue 0 DropThreshold 0 0 DropThreshold 1 0  DropThreshold 2 0

      Queue 1 DropThreshold 0 0 DropThreshold 1 0  DropThreshold 2 0

      Queue 2 DropThreshold 0 0 DropThreshold 1 0  DropThreshold 2 0

      Queue 3 DropThreshold 0 0 DropThreshold 1 0  DropThreshold 2 0

  RxQueue Drop Statistics Slice1

   Queue 0

      DropThreshold 0 0

      DropThreshold 1 0

      DropThreshold 2 0

   Queue 1

      DropThreshold 0 0

      DropThreshold 1 0

      DropThreshold 2 0

   Queue 2

      DropThreshold 0 0

      DropThreshold 1 0

      DropThreshold 2 0

   Queue 3

      DropThreshold 0 0

      DropThreshold 1 0

      DropThreshold 2 0

  Port 2 TxQueue Drop Statistics

    Queue 0

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 1

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 2

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 3

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 4

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 5

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 6

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 7

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

  Supervisor TxQueue Drop Statistics

    Queue  0: 0

    Queue  1: 0

    Queue  2: 0

    Queue  3: 0

    Queue  4: 0

    Queue  5: 0

    Queue  6: 0

    Queue  7: 0

    Queue  8: 0

    Queue  9: 0

    Queue 10: 0

    Queue 11: 0

    Queue 12: 0

    Queue 13: 0

    Queue 14: 0

    Queue 15: 0

Any help would be appreciated.

[nkarpysh]

Hello,

Ports can experience bursty traffic and drop due to no buffers. Please attach following logs:

show ver

show buffer

show platform pm if-number

sh controllers ethernet-controller port-asic statistics

for all affected ports:

show int gix/y switching

show queueing int gix/y

sho controller ethernet gix/Y

show int gix/y counter

Nik

[chris-gibbs]

Hey,

attached are the command outputs.

"show int gix/y switching" is not available on the IOS version.

Also i have only attached outputs for the gi 0/4 interface, the rest are pretty similar.

cheers

Chris

[chris-gibbs]

A few more uploads.

Cheers

Chris

[nkarpysh]

Thanks,

Can you please also capture this command:

sh platform port-asic stats drop asic 1 Port 2

Please also try this command once more "show int gix/y switching". Don't expect that to come out with question mark - that is hidden. Just copy paste as is.

Thanks

Nik

[chris-gibbs]

Thanks for having a look Nik.

Attached the commands requested.

Cheers

Chris

[nkarpysh · Accepted solution]

Hi,

It seems you oversubscribe queue 3. Port Asic command showing that:

  Port 2 TxQueue Drop Statistics

    Queue 0

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 1

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 13555

    Queue 2

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 3

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 49495569

As per your config queue 2 has much bigger share:

srr-queue bandwidth share 10 10 60 20

You can check the following commands to see which DSCP-QOS are matching that queue and possibly then adjust it:

sho mls qos int gi0/4 stat
show mls qos maps dscp-cos

sh mls qos maps cos-output-q

Nik

[chris-gibbs]

Thanks for your assistance Nik.

It was troubling me what was using the links and causing congestion, as they are not typically heavily utilised, especially at night. I setup a packet capture and noticed that there was a large amount of traffic going down the trunk that should not have been.

Seems we ended up having an issue with incorrect vtp and there were incorrect vlans on the remote site.

After removing the vlans, the problem has been resolved.

On the plus side I learnt a lot about QoS.

Cheers

Chris