Total output drops

[raulalejandroortegameza]

Hi people

I have a problem with drop packet in many interefaces of a WS-C6506-E (R7000) (s72033-ipservices_wan-mz.122-33.SXJ3.bin)

in the picture you can see that

most of the interfaces have connected servers

which can be the cause of this?, what troubleshooting I can do  to find the source of the problem?
thank

[Reza Sharifi]

I can't see the numbers, as your pictures are very small. 

What is the speed of the interfaces on the switch and the servers?

Are the interfaces configured as auto or hard set?

Are they copper or fiber interfaces?

What type of devices are the switch ports connected to? i.e server, storage, etc...

[raulalejandroortegameza]

Hi.

Same interface have auto  and other are hard set, but both have output drop.

the media type is 10/100/1000BaseT

[raulalejandroortegameza]

GigabitEthernet1/39 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0018.bad2.5f18 (bia 0018.bad2.5f18)
  Description: ** netprovider(10.1.9.103) **
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseT
  input flow-control is off, output flow-control is on
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:44, output hang never
  Last clearing of "show interface" counters 2d08h
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1378678
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 80000 bits/sec, 46 packets/sec
  5 minute output rate 835000 bits/sec, 129 packets/sec
     10434844 packets input, 2193769543 bytes, 0 no buffer
     Received 38563 broadcasts (5027 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     121317428 packets output, 60527336994 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

[Reza Sharifi]

Is QOS configured on this interface?

sh run int gi1/39

[raulalejandroortegameza]

interface GigabitEthernet1/39
 switchport
 switchport access vlan 35
 switchport mode access
 logging event link-status
 spanning-tree portfast edge
 spanning-tree bpduguard enable
end

[Emmanouil Patiniotakis]

The first action is to hard set speed of 100Mbps and full duplex. Do the same at the other side. Then try to pass from an interface with drops low rate data in order to see if the drops are increasing. 

Output drops are always related to L1/L2 issues. The cables you are using are 5e? 

[raulalejandroortegameza]

mm I don't know what kind of clable they are using, but should be cat 6, beacause the interface are Gi, therefore the speed is 1000Mbps.

the biggest problem  is many  interface with output drop packet and all same vlan, but other interface in diferent vlan they don't have output drop packet, this is confused. in the picture you can see the topology.
The interface vlan 30 has three ip secundary with GLBP and all port with vlan 30 have output drop packet

[Aninda Chatterjee]

Hello,

What line card is inserted into slot 1? When you're troubleshooting input/output drops, it is important to understand the architecture of the card as well to consider possible over-subscription points.

It is interesting that you state that all ports in this VLAN are seeing these drops increment - sounds like you might be experiencing a unicast flood in this VLAN (possibly, just a wild guess here going off of your symptoms).

Can you perhaps connect a laptop in the same VLAN and sniff the NIC using Wireshark/tcpdump? If you're receiving traffic that is not meant for that laptop (and it is a unicast packet), then that's a unicast flood in the VLAN.

Can you also post the following outputs from one of the interfaces experiencing output drops:

1. show counters interface <interface>

2. show queueing interface <interface>

Are these drops constantly incrementing or do they increment in bursts and then stay stable for a while and then increment in a burst again and so on?

Regards,

Aninda

[raulalejandroortegameza]

Thank for you response aninchat

Yes, these drops increment in bursts and then stay estable for a while, just as you say 

Below are output command show counter, show queueing, show module, show interface vlan 30

#show counters interface g1/21
64 bit counters:
 0.                      rxHCTotalPkts = 361202
 1.                      txHCTotalPkts = 1411690021
 2.                    rxHCUnicastPkts = 361200
 3.                    txHCUnicastPkts = 410142854
 4.                  rxHCMulticastPkts = 0
 5.                  txHCMulticastPkts = 900686865
 6.                  rxHCBroadcastPkts = 2
 7.                  txHCBroadcastPkts = 100860302
 8.                         rxHCOctets = 33528526
 9.                         txHCOctets = 374407450833
10.                 rxTxHCPkts64Octets = 678400385
11.            rxTxHCPkts65to127Octets = 352093183
12.           rxTxHCPkts128to255Octets = 157537314
13.           rxTxHCPkts256to511Octets = 35132219
14.          rxTxHCpkts512to1023Octets = 21865647
15.         rxTxHCpkts1024to1518Octets = 167022475
16.                    txHCTrunkFrames = 0
17.                    rxHCTrunkFrames = 0
18.                     rxHCDropEvents = 0

32 bit counters:
 0.                   rxCRCAlignErrors = 0
 1.                   rxUndersizedPkts = 0
 2.                    rxOversizedPkts = 0
 3.                     rxFragmentPkts = 0
 4.                          rxJabbers = 0
 5.                       txCollisions = 0
 6.                         ifInErrors = 0
 7.                        ifOutErrors = 0
 8.                       ifInDiscards = 0
 9.                  ifInUnknownProtos = 0
10.                      ifOutDiscards = 79993129
11.            txDelayExceededDiscards = 0
12.                              txCRC = 0
13.                         linkChange = 1
14.                   wrongEncapFrames = 0
All Port Counters
 1.                          InPackets = 361204
 2.                           InOctets = 33528702
 3.                        InUcastPkts = 361202
 4.                        InMcastPkts = 0
 5.                        InBcastPkts = 2
 6.                         OutPackets = 1411691426
 7.                          OutOctets = 374408006548
 8.                       OutUcastPkts = 410143207
 9.                       OutMcastPkts = 900687369
10.                       OutBcastPkts = 100860850
11.                           AlignErr = 0
12.                             FCSErr = 0
13.                            XmitErr = 0
14.                             RcvErr = 0
15.                          UnderSize = 0
16.                          SingleCol = 0
17.                           MultiCol = 0
18.                            LateCol = 0
19.                       ExcessiveCol = 0
20.                       CarrierSense = 0
21.                              Runts = 0
22.                             Giants = 0
23.                         InDiscards = 0
24.                        OutDiscards = 79993129
25.                           InErrors = 0
26.                          OutErrors = 0
27.                    InUnknownProtos = 0
28.                              txCRC = 0
29.                      TrunkFramesTx = 0
30.                      TrunkFramesRx = 0
31.                         WrongEncap = 0
32.     Broadcast_suppression_discards = 0
33.     Multicast_suppression_discards = 0
34.       Unicast_suppression_discards = 0
35.                 rxTxHCPkts64Octets = 678400875
36.            rxTxHCPkts65to127Octets = 352093605
37.           rxTxHCPkts128to255Octets = 157537435
38.           rxTxHCPkts256to511Octets = 35132277
39.          rxTxHCpkts512to1023Octets = 21865661
40.         rxTxHCpkts1024to1518Octets = 167022777
41.                         DropEvents = 0
42.                     CRCAlignErrors = 0
43.                     UndersizedPkts = 0
44.                      OversizedPkts = 0
45.                       FragmentPkts = 0
46.                            Jabbers = 0
47.                         Collisions = 0
48.              DelayExceededDiscards = 0
49.                        bpduOutlost = 0
50.                        qos0Outlost = 79993129
51.                        qos1Outlost = 0
52.                        qos2Outlost = 0
53.                        qos3Outlost = 0
54.                        qos4Outlost = 0
55.                        qos5Outlost = 0
56.                        qos6Outlost = 0
57.                        qos7Outlost = 0
58.                        qos8Outlost = 0
59.                        qos9Outlost = 0
60.                       qos10Outlost = 0
61.                       qos11Outlost = 0
62.                       qos12Outlost = 0
63.                       qos13Outlost = 0
64.                       qos14Outlost = 0
65.                       qos15Outlost = 0
66.                       qos16Outlost = 0
67.                       qos17Outlost = 0
68.                       qos18Outlost = 0
69.                       qos19Outlost = 0
70.                       qos20Outlost = 0
71.                       qos21Outlost = 0
72.                       qos22Outlost = 0
73.                       qos23Outlost = 0
74.                       qos24Outlost = 0
75.                       qos25Outlost = 0
76.                       qos26Outlost = 0
77.                       qos27Outlost = 0
78.                       qos28Outlost = 0
79.                       qos29Outlost = 0
80.                       qos30Outlost = 0
81.                       qos31Outlost = 0
82.                    bpduCbicOutlost = 0
83.                    qos0CbicOutlost = 0
84.                    qos1CbicOutlost = 0
85.                    qos2CbicOutlost = 0
86.                    qos3CbicOutlost = 0
87.                         bpduInlost = 0
88.                         qos0Inlost = 0
89.                         qos1Inlost = 0
90.                         qos2Inlost = 0
91.                         qos3Inlost = 0
92.                         qos4Inlost = 0
93.                         qos5Inlost = 0
94.                         qos6Inlost = 0
95.                         qos7Inlost = 0
96.                         qos8Inlost = 0
97.                         qos9Inlost = 0
98.                        qos10Inlost = 0
99.                        qos11Inlost = 0
100.                        qos12Inlost = 0
101.                        qos13Inlost = 0
102.                        qos14Inlost = 0
103.                        qos15Inlost = 0
104.                        qos16Inlost = 0
105.                        qos17Inlost = 0
106.                        qos18Inlost = 0
107.                        qos19Inlost = 0
108.                        qos20Inlost = 0
109.                        qos21Inlost = 0
110.                        qos22Inlost = 0
111.                        qos23Inlost = 0
112.                        qos24Inlost = 0
113.                        qos25Inlost = 0
114.                        qos26Inlost = 0
115.                        qos27Inlost = 0
116.                        qos28Inlost = 0
117.                        qos29Inlost = 0
118.                        qos30Inlost = 0
119.                        qos31Inlost = 0
120.                         pqueInlost = 0
121.                           Overruns = 0
122.                           maxIndex = 0

show queueing interface g1/21
Interface GigabitEthernet1/21 queueing strategy:  Weighted Round-Robin
  Port QoS is enabled
Trust boundary disabled

  Port is untrusted
  Extend trust state: not trusted [COS = 0]
  Default COS is 0
    Queueing Mode In Tx direction: mode-cos
    Transmit queues [type = 1p3q8t]:
    Queue Id    Scheduling  Num of thresholds
    -----------------------------------------
       01         WRR                 08
       02         WRR                 08
       03         WRR                 08
       04         Priority            01

    WRR bandwidth ratios:  100[queue 1] 150[queue 2] 200[queue 3]
    queue-limit ratios:     50[queue 1]  20[queue 2]  15[queue 3]  15[Pri Queue]

    queue tail-drop-thresholds
    --------------------------
    1     70[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]
    2     70[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]
    3     100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]

    queue random-detect-min-thresholds
    ----------------------------------
      1    40[1] 70[2] 70[3] 70[4] 70[5] 70[6] 70[7] 70[8]
      2    40[1] 70[2] 70[3] 70[4] 70[5] 70[6] 70[7] 70[8]
      3    70[1] 70[2] 70[3] 70[4] 70[5] 70[6] 70[7] 70[8]

    queue random-detect-max-thresholds
    ----------------------------------
      1    70[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]
      2    70[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]
      3    100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]

    WRED disabled queues:

    queue thresh cos-map
    ---------------------------------------
    1     1      0
    1     2      1
    1     3
    1     4
    1     5
    1     6
    1     7
    1     8
    2     1      2
    2     2      3 4
    2     3
    2     4
    2     5
    2     6
    2     7
    2     8
    3     1      6 7
    3     2
    3     3
    3     4
    3     5
    3     6
    3     7
    3     8
    4     1      5

    Queueing Mode In Rx direction: mode-cos
    Receive queues [type = 1q2t]:
    Queue Id    Scheduling  Num of thresholds
    -----------------------------------------
       1         Standard            2

    queue tail-drop-thresholds
    --------------------------
    1     100[1] 100[2]

    queue thresh cos-map
    ---------------------------------------
    1     1      0 1 2 3 4 5 6 7
    1     2

  Packets dropped on Transmit:
    BPDU packets:  0

    queue thresh             dropped  [cos-map]
    ---------------------------------------------------

    1     1                 5001508  [0 ]
    1     2                       0  [1 ]
    2     1                       0  [2 ]
    2     2                       0  [3 4 ]
    3     1                       0  [6 7 ]
    4     1                       0  [5 ]

  Packets dropped on Receive:
    BPDU packets:  0

    queue thresh              dropped  [cos-map]
    ---------------------------------------------------
    1     1                       0  [0 1 2 3 4 5 6 7 ]

show module
Mod Ports Card Type                              Model
--- ----- -------------------------------------- ---------------
  1   48  48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX
  2   48  48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX
  3   48  48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX
  5    5  Supervisor Engine 720 10GE (Active)    VS-S720-10G

interface Vlan30 (CORE 1)
 ip address x.x.x.x a.b.c.d secondary
 ip address x.x.x.x a.b.c.d secondary
 ip address x.x.x.x x.x.x.x secondary
 ip address x.x.x.x x.x.x.x
 no ip redirects
 no ip proxy-arp
 glbp 30 ip x.x.x.x
 glbp 30 ip x.x.x.x secondary
 glbp 30 ip x.x.x.x secondary
 glbp 30 ip x.x.x.x secondary
 glbp 30 priority 105

interface Vlan30 (core 2)
 ip address x.x.x.x x.x.x.x secondary
 ip address x.x.x.x x.x.x.x secondary
 ip address x.x.x.x x.x.x.x secondary
 ip address x.x.x.x x.x.x.x
 no ip redirects
 no ip proxy-arp
 glbp 30 ip x.x.x.x
 glbp 30 ip x.x.x.x secondary
 glbp 30 ip x.x.x.x secondary
 glbp 30 ip x.x.x.x secondary
end