×

Warning message

  • Cisco Support Forums is in Read Only mode while the site is being migrated.
  • Cisco Support Forums is in Read Only mode while the site is being migrated.

WS-C2960S-24TS-L - Output Drops when Low Utilization

Unanswered Question
Aug 1st, 2013
User Badges:

Hi,


I'm trying to troubleshoot output drops on an uplink interface. Here is the situation:


  • Model: WS-C2960S-24TS-L
  • 1 GE uplink, trunking ( gi1/0/23)
  • QoS is not enabled ( no mls qos )
  • Link utilization for the uplink is never more than 20%
  • The output of "show buffers" does not show incrementing "missed" counters
  • The "show platform port-asic stats misc gi1/0/23" does counter TxBufferFullDropCount is nonzero


Some output below:


as2-1-1#show platform port-asic stats miscellaneous gi1/0/23



Port-asic Misc Statistics

=====================================

    TxBufferFullDropCount               676470

    TxBufferFrameDescBadCrc16DropCountA 0

    TxBufferFrameDescBadCrc16DropCountB 0

    TxBufferBandwidthDropCount          0

    TxQueueBandwidthDropCount           0

    TxQueueMissedDropStatistics         0

    SneakQueueDropCount                 0

    RxBufferDropDestIndexCountA         0

    RxBufferDropDestIndexCountB         0

    LearningQueueOverflowFrames         1

    RxNetworkProcessorPacketCount0      0

    RxNetworkProcessorPacketCount1      0

as2-1-1#


as2-1-1#show platform port-asic stats drop gi1/0/23


  Interface Gi1/0/23 TxQueue Drop Statistics

!!! Output omitted

      Weight 0 Frames 648845

      Weight 1 Frames 0

      Weight 2 Frames 4

    Queue 2

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 3

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 26293

!!! Output omitted

as2-1-1#



Has anyone overcome this issue without QoS or replacing hardware?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Leo Laohoo Thu, 08/01/2013 - 18:00
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Post the following output:


1.  sh version;

2.  sh interface Gi 1/0/23; and

3.  sh controll e Gi 1/0/23

coredialeng Thu, 08/01/2013 - 18:52
User Badges:

Hi Leo,


Thanks for your reply. I've included a zip file containing the outputs you requested and several graphs. There are discontinuities in the graphs because of a memory issue we have with our monitoring server ( unrelated ). I've also include the output of "show platform port-asic stat drop gi1/0/23" in the file "stat_drop.txt"


One thing I notice is that there are drops in multiple queues ( stat_drop.txt) when QoS is disabled. My understanding is that when QoS is disabled globally, there is a single queue ( i.e.e Queue 1 ).


I've done usual checks while output drops were incrementing:


  • show proc cpu sorted ==> No individual process with high utilization
  • show mem failures alloc ==> No output
  • show buffer failures ==> None listed
  • show proc cpu history ==> No spikes, nomial utilziation is ~30 - 40%
  • show buffers ==> No incrementing "missed counters"
  • Link speed and duplex match on either end


As you'll note from the graphs, the utilization on the link is really low. I've searched through the 111 published bugs on the Cisco bugtracker tool as well and found none relevant to our environment.


This seems to be a widespread issue on the 2960S platform at and before the code level I'm runing ( 12.2(55)SE2 ). Multiple forums suggested upgrading to at least 12.2(58) but others have shown this not to have solved the issue. Other forum post show implementing QoS ( or disabling) it to be a fix, however these things are not currently implemented in our environment and so do not apply.


T.I.A,

Joe

Attachment: 
Leo Laohoo Thu, 08/01/2013 - 19:14
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

H0ly crap!  I can't read your file.


Cut and paste the outputs I've requested in the thread.


I would upgrade your IOS to 12.2(55)SE8.

coredialeng Fri, 08/02/2013 - 12:43
User Badges:

Hi Leo,


Sorry for the late reply. Was off today. See below:



#### Show Ver ####


Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(55)SE2, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2011 by Cisco Systems, Inc.

Compiled Tue 11-Jan-11 02:23 by prod_rel_team

Image text-base: 0x00003000, data-base: 0x01B00000



ROM: Bootstrap program is Alpha board boot loader

BOOTLDR: C2960S Boot Loader (C2960S-HBOOT-M) Version 12.2(53r)SE, RELEASE SOFTWARE (fc3)



as2-1-1 uptime is 1 week, 3 days, 8 hours, 11 minutes

System returned to ROM by power-on

System restarted at 13:19:33 EDT Mon Jul 22 2013

System image file is "flash:/c2960s-universalk9-mz.122-55.SE2/c2960s-universalk9-mz.122-55.SE2.bin"





This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.



A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html



If you require further assistance please contact us by sending email to

[email protected].



cisco WS-C2960S-24TS-L (PowerPC) processor (revision C0) with 131072K bytes of memory.

Processor board ID FOC1508X3U0

Last reset from power-on

1 Virtual Ethernet interface

1 FastEthernet interface

28 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

If you require further assistance please contact us by sending email to

[email protected].



cisco WS-C2960S-24TS-L (PowerPC) processor (revision C0) with 131072K bytes of memory.

Processor board ID FOC1508X3U0

Last reset from power-on

1 Virtual Ethernet interface

1 FastEthernet interface

28 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.



512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address       : B8:BE:BF:30:1C:80

Motherboard assembly number     : 73-11910-06

Power supply part number        : 341-0328-02

Motherboard serial number       : FOC15077RQ0

Power supply serial number      : DCA1432M6W8

Model revision number           : C0

Motherboard revision number     : A0

Model number                    : WS-C2960S-24TS-L

Daughterboard assembly number   : 73-11933-04

Daughterboard serial number     : FOC150643E7

System serial number            : FOC1508X3U0

Top Assembly Part Number        : 800-30954-02

Top Assembly Revision Number    : B0

Version ID                      : V02

CLEI Code Number                : COMGG00ARB

Daughterboard revision number   : A0

Hardware Board Revision Number  : 0x01





Switch Ports Model              SW Version            SW Image                

------ ----- -----              ----------            ----------              

*    1 28    WS-C2960S-24TS-L   12.2(55)SE2           C2960S-UNIVERSALK9-M    





Configuration register is 0xF



### Show int ###


as2-1-1#show int gi1/0/23

GigabitEthernet1/0/23 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is b8be.bf30.1c97 (bia b8be.bf30.1c97)

  Description: PHY|L2|rhost:cs2-1-1|rport:gi1/1

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:01, output 00:00:03, output hang never

  Last clearing of "show interface" counters 06:52:22

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2647

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 1659000 bits/sec, 1287 packets/sec

  5 minute output rate 2558000 bits/sec, 1198 packets/sec

     76224138 packets input, 11348240054 bytes, 0 no buffer

     Received 468086 broadcasts (114552 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 114552 multicast, 0 pause input

     0 input packets with dribble condition detected

     79685680 packets output, 36998897195 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

as2-1-1#


### Show controllers ###


as2-1-1#show controllers ethernet-controller gi1/0/23       



     Transmit GigabitEthernet1/0/23           Receive

   1360350908 Bytes                       4218320450 Bytes                   

   1594259627 Unicast frames              2733236679 Unicast frames          

       371647 Multicast frames               4347119 Multicast frames        

       564581 Broadcast frames              12786960 Broadcast frames        

            0 Too old frames              2854689159 Unicast bytes           

            0 Deferred frames              545250746 Multicast bytes         

            0 MTU exceeded frames          818380545 Broadcast bytes         

            0 1 collision frames                   0 Alignment errors        

            0 2 collision frames                   0 FCS errors              

            0 3 collision frames                   0 Oversize frames         

            0 4 collision frames                   0 Undersize frames        

            0 5 collision frames                   0 Collision fragments     

            0 6 collision frames      

            0 7 collision frames           201288077 Minimum size frames     

            0 8 collision frames           867680707 65 to 127 byte frames   

            0 9 collision frames           107819549 128 to 255 byte frames  

            0 10 collision frames           60471382 256 to 511 byte frames  

            0 11 collision frames           79485747 512 to 1023 byte frames 

            0 12 collision frames         1433625296 1024 to 1518 byte frames

            0 13 collision frames                  0 Overrun frames          

            0 14 collision frames                  0 Pause frames            

            0 15 collision frames     

            0 Excessive collisions                 0 Symbol error frames     

            0 Late collisions                      0 Invalid frames, too large

            0 VLAN discard frames                  0 Valid frames, too large 

            0 Excess defer frames                  0 Invalid frames, too small

     96727731 64 byte frames                       0 Valid frames, too small 

    414347781 127 byte frames         

    292401594 255 byte frames                      0 Too old frames          

    247938903 511 byte frames                      0 Valid oversize frames   

    201842506 1023 byte frames                     0 System FCS error frames 

    341937340 1518 byte frames                     0 RxPortFifoFull drop frame

            0 Too large frames        

            0 Good (1 coll) frames    

            0 Good (>1 coll) frames   



as2-1-1#


### Show platform ... ###


as2-1-1#show platform port-asic stats drop gi1/0/23



  Interface Gi1/0/23 TxQueue Drop Statistics

    Queue 0

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 1

      Weight 0 Frames 651608

      Weight 1 Frames 0

      Weight 2 Frames 4

    Queue 2

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 3

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 27867

    Queue 4

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 5

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 6

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 7

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

as2-1-1#


As mentioned, there is no QoS enabled globally. This is a simple layer 2 switch configuration.


Also attached the graphs.


Thanks again.

Joe

Leo Laohoo Fri, 08/02/2013 - 17:35
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

As mentioned, there is no QoS enabled globally

Ok, thanks for the output.  I see no line errors.  I see, however, "output drops".


Disable interface-level QoS and see if output drops still increments.


Can you tell me what's connected to Gi 1/0/23?

coredialeng Fri, 08/02/2013 - 17:42
User Badges:

Leo,


Interface QoS is also disabled:



as2-1-1#show mls qos inter gi1/0/23

GigabitEthernet1/0/23

QoS is disabled. When QoS is enabled, following settings will be applied <<<<<<<<<<< QoS disabled

trust state: not trusted

trust mode: not trusted

trust enabled flag: ena

COS override: dis

default COS: 0

DSCP Mutation Map: Default DSCP Mutation Map

Trust device: none

qos mode: port-based


as2-1-1#



A Cisco 4948 switch is connected on the gi1/0/23 interface (in trunk mode ).

Leo Laohoo Fri, 08/02/2013 - 17:50
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Did you disable QoS on the 4948-side too?

coredialeng Fri, 08/02/2013 - 17:56
User Badges:

Good catch,


Inteface QoS is enabled on the other side of the trunk, something I hadn't known about/looked at before:


cs-2-1-1#show qos inter gi1/1

QoS is disabled globally

Port QoS is enabled

Administrative Port Trust State: 'untrusted'

Operational Port Trust State: 'dscp'

Trust device: none

Default DSCP: 0 Default CoS: 0

Appliance trust: none

Tx-Queue   Bandwidth   ShapeRate   Priority   QueueSize

             (bps)       (bps)                (packets)

  1        250000000   disabled    N/A        2336    

  2        250000000   disabled    N/A        2336    

  3        250000000   disabled    normal     2336    

  4        250000000   disabled    N/A        2336    



cs-2-1-1#


However, how would enabling QoS on the far end affect Output drops on the near end?

Leo Laohoo Fri, 08/02/2013 - 18:00
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

However, how would enabling QoS on the far end affect Output drops on the near end?

It's the opposite.  If you enable QoS in a LAN, there's a significant chance of getting output drops.

coredialeng Fri, 08/02/2013 - 18:09
User Badges:

Leo Laohoo wrote:


However, how would enabling QoS on the far end affect Output drops on the near end?

It's the opposite.  If you enable QoS in a LAN, there's a significant chance of getting output drops.


That make sense if QoS is enabled on every switch in the LAN, in this case, only the upstream switch has port QoS enabled. On the downlink switch, there is no QoS and all traffic is treated equally.


I guess my question is, how would having QoS enabled on the upstream switch cause output drops on the downstream switch (e.g. for traffic egressing to the upstream switch )?

Actions

This Discussion

Related Content