Solved: Re: 2960-S Supervisor TxQueue Drop Statistics

lcaruso · ‎07-26-2012

Hi,

I have users connected to a 2960-S running 12.2(53r)SE complaining of slow network, specifically Internet performance while the upstream 2960 on the same vlan has no such issues, so I don't suspect link congestion on the face of things.

I'm planning on upgrading the code, but wanted to see what could be checked first. I'm seeing some drops but could use some tips on how to proceed from here as to what might be the issue. CPU is under 10% whenever I check it.

Thanks.

Supervisor TxQueue Drop Statistics

Queue 0: 0

Queue 1: 0

Queue 2: 0

Queue 3: 0

Queue 4: 0

Queue 5: 0

Queue 6: 0

Queue 7: 0

Queue 8: 0

Queue 9: 0

Queue 10: 0

Queue 11: 0

Queue 12: 0

Queue 13: 0

Queue 14: 1315685992

Queue 15: 0

Leo Laohoo · ‎07-26-2012

I have users connected to a 2960-S running 12.2(53r)SE

Doesn't help. The IOS you've quoted is the bootstrap and NOT the IOS version.

Can you please post the output to the following command:

1. sh interface ;

2. sh controll e ;

3. sh version; and

4. sh log

View solution in original post

Leo Laohoo · ‎07-27-2012

Ok, now the output is getting very disturbing.

1. Link speed of Gi 0/11 is 10 Mbps. Why is that? What is connected to the far end?

2. Under the output of the "sh interface Gi 0/11", look at "Total Output Drops". This is generated when the switch tries to send data downstream (or down the wire) and the remote end is dropping packets. This counter should never drop to zero. It this event happens, the switch will record this and the counter increments.

3. Under the output of the "sh controll e Gi 0/11" look under "Invalid frames, too large". That's not good.

So what is connected to Gi 0/11???

Can you ask the person making the complaints to use another computer and see if he/she is exhibiting the same symptoms?

May I ask if you can run the following commands:

1. Command: test cable tdr int Gi 0/11;

2. Wait for approximately 7 seconds;

3. Command: sh cable tdr int Gi 0/11; and

4. Please post the output.

View solution in original post

Leo Laohoo · ‎07-26-2012

I have users connected to a 2960-S running 12.2(53r)SE

Doesn't help. The IOS you've quoted is the bootstrap and NOT the IOS version.

Can you please post the output to the following command:

1. sh interface ;

2. sh controll e ;

3. sh version; and

4. sh log

lcaruso · ‎07-26-2012

Thanks much for your reply! Hadn't seen one of those commands before!

The IOS you've quoted is the bootstrap and NOT the IOS version.

lacking sleep...oh wait...so is everybody else in this business...

needs a reboot

needs an upgrade

I see a lot of multicasts

s-hw-2-1#sh int gi0/25

GigabitEthernet0/25 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is e804.62db.1899 (bia e804.62db.1899)

Description: *** Uplink to hw-mc-3550-24-1 ***

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive not set

Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:28, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 1000 bits/sec, 2 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

59005727 packets input, 43109590387 bytes, 0 no buffer

Received 10434049 broadcasts (8956069 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 8956069 multicast, 0 pause input

0 input packets with dribble condition detected

33685706 packets output, 15044276646 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

s-hw-2-1#sh controll e gi0/25

Transmit GigabitEthernet0/25 Receive

2159382383 Bytes 159929089 Bytes

32886697 Unicast frames 48571756 Unicast frames

508431 Multicast frames 8956115 Multicast frames

290623 Broadcast frames 1477990 Broadcast frames

0 Too old frames 3399750986 Unicast bytes

0 Deferred frames 672471348 Multicast bytes

0 MTU exceeded frames 148450015 Broadcast bytes

0 1 collision frames 0 Alignment errors

0 2 collision frames 0 FCS errors

0 3 collision frames 0 Oversize frames

0 4 collision frames 0 Undersize frames

0 5 collision frames 0 Collision fragments

0 6 collision frames

0 7 collision frames 299940 Minimum size frames

0 8 collision frames 26385265 65 to 127 byte frames

0 9 collision frames 3233677 128 to 255 byte frames

0 10 collision frames 2032307 256 to 511 byte frames

0 11 collision frames 1202392 512 to 1023 byte frames

0 12 collision frames 3481180 1024 to 1518 byte frames

0 13 collision frames 0 Overrun frames

0 14 collision frames 0 Pause frames

0 15 collision frames

0 Excessive collisions 0 Symbol error frames

0 Late collisions 0 Invalid frames, too large

0 VLAN discard frames 22371100 Valid frames, too large

0 Excess defer frames 0 Invalid frames, too small

299768 64 byte frames 0 Valid frames, too small

19569412 127 byte frames

3147244 255 byte frames 0 Too old frames

1727353 511 byte frames 0 Valid oversize frames

1226616 1023 byte frames 0 System FCS error frames

883772 1518 byte frames 0 RxPortFifoFull drop frame

6831586 Too large frames

0 Good (1 coll) frames

0 Good (>1 coll) frames

s-hw-2-1#sh ver

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 21-Apr-10 06:08 by prod_rel_team

Image text-base: 0x00003000, data-base: 0x01B00000

ROM: Bootstrap program is Alpha board boot loader

BOOTLDR: C2960S Boot Loader (C2960S-HBOOT-M) Version 12.2(53r)SE, RELEASE SOFTWARE (fc3)

s-hw-2-1 uptime is 1 year, 42 weeks, 6 days, 1 hour, 50 minutes

System returned to ROM by power-on

System restarted at 15:59:11 CDT Thu Sep 30 2010

System image file is "flash:/c2960s-universalk9-mz.122-53.SE2/c2960s-universalk9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C2960S-24TS-S (PowerPC) processor (revision B0) with 131072K bytes of memory.

Processor board ID FOC1433X2WD

Last reset from power-on

2 Virtual Ethernet interfaces

1 FastEthernet interface

26 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : E8:04:62:DB:18:80

Motherboard assembly number : 73-12423-06

Power supply part number : 341-0328-01

Motherboard serial number : FOC14340BXQ

Power supply serial number : DCA1409M22C

Model revision number : B0

Motherboard revision number : A0

Model number : WS-C2960S-24TS-S

Daughterboard assembly number : 73-11933-04

Daughterboard serial number : FOC14341T1M

System serial number : FOC1433X2WD

Top Assembly Part Number : 800-32448-02

Top Assembly Revision Number : A0

Version ID : V02

CLEI Code Number : COMGJ00ARB

Daughterboard revision number : A0

Hardware Board Revision Number : 0x01

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 26 WS-C2960S-24TS-S 12.2(53)SE2 C2960S-UNIVERSALK9-M

Configuration register is 0xF

s-hw-2-1#sh log

Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 23382 messages logged, xml disabled,

filtering disabled

Monitor logging: level debugging, 0 messages logged, xml disabled,

filtering disabled

Buffer logging: level debugging, 23382 messages logged, xml disabled,

filtering disabled

Exception Logging: size (4096 bytes)

Count and timestamp logging messages: disabled

File logging: disabled

Persistent logging: disabled

No active filter modules.

Trap logging: level informational, 23382 message lines logged

Log Buffer (4096 bytes):

e GigabitEthernet0/11, changed state to up

023345: Jul 26 14:47:36 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to down

023346: Jul 26 14:47:38 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to up

023347: Jul 26 14:47:39 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to up

023348: Jul 26 15:10:47 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

023349: Jul 26 15:10:48 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

023350: Jul 26 15:10:51 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up

023351: Jul 26 15:10:52 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up

023352: Jul 26 15:16:47 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to down

023353: Jul 26 15:16:49 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to up

023354: Jul 26 15:19:22 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to down

023355: Jul 26 15:19:23 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to down

023356: Jul 26 15:19:25 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to up

023357: Jul 26 15:19:26 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to up

023358: Jul 26 15:38:15 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

023359: Jul 26 15:38:16 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down

023360: Jul 26 15:39:39 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to down

023361: Jul 26 15:39:40 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to down

023362: Jul 26 15:39:47 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to up

023363: Jul 26 15:39:50 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to down

023364: Jul 26 15:39:52 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to up

023365: Jul 26 15:39:53 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to up

023366: Jul 26 15:57:48 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/3, changed state to down

023367: Jul 26 15:57:50 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/3, changed state to up

023368: Jul 26 16:19:03 CDT: %SYS-5-CONFIG_I: Configured from console by vty0 (10.10.223.1)

023369: Jul 26 16:35:54 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to down

023370: Jul 26 16:35:56 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to up

023371: Jul 26 16:37:38 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to down

023372: Jul 26 16:37:40 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to up

023373: Jul 26 16:38:13 CDT: %SYS-5-CONFIG_I: Configured from console by vty0 (10.10.223.1)

023374: Jul 26 16:40:08 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/4, changed state to down

023375: Jul 26 16:40:10 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/4, changed state to up

023376: Jul 26 17:02:18 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to down

023377: Jul 26 17:02:19 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to down

023378: Jul 26 17:02:27 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, changed state to up

023379: Jul 26 17:02:28 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to up

023380: Jul 26 17:03:18 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to down

023381: Jul 26 17:03:20 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/11, changed state to up

s-hw-2-1#

Leo Laohoo · ‎07-26-2012

Hmmmm ....

I don't see any line errors on your Gi1/0/25, which I hope, is your uplink. Now that's good news.

Now here are things that are getting quiet "interesting".

Firstly, your IOS. 12.2(53)SE (and anything in the "53"), is an IOS I would NOT use. Period.

Next, I don't want to call you a "liar" but I nearly did. Look at your "sh logs". I have two questions:

1. Did you get this "sh log" output from a 2960S? Look at the interface, it is showing that it could be a non-2960S or a non-3750G/E/X.

2. Is the client having issues connected to Gi0/11?

Can you please further post the following output:

1. sh interface ; and

2. sh controll e

lcaruso · ‎07-27-2012

Sorry about the lag time. This site went offline last night just as I attempted my post. Then when it came back online, I could not login.

Did you get this "sh log" output from a 2960S? Look at the interface, it is showing that it could be a non-2960S or a non-3750G/E/X.

I can assure you I did.

I wasn't yet provided the , but let's assume it's gi0/11 as it appears likely. Thanks.

s-hw-2-1#sh int gi0/11

GigabitEthernet0/11 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is e804.62db.188b (bia e804.62db.188b)

Description: User Access

MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 10Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 351

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

4731239 packets input, 621556977 bytes, 0 no buffer

Received 79724 broadcasts (16622 multicasts)

0 runts, 19 giants, 0 throttles

19 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 16622 multicast, 0 pause input

0 input packets with dribble condition detected

14753576 packets output, 10011869811 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

s-hw-2-1#sh controll e gi0/11

Transmit GigabitEthernet0/11 Receive

1421937036 Bytes 621556977 Bytes

8719754 Unicast frames 4651515 Unicast frames

5136702 Multicast frames 16622 Multicast frames

897139 Broadcast frames 63102 Broadcast frames

0 Too old frames 611275655 Unicast bytes

0 Deferred frames 4792405 Multicast bytes

0 MTU exceeded frames 5348636 Broadcast bytes

0 1 collision frames 0 Alignment errors

0 2 collision frames 0 FCS errors

0 3 collision frames 19 Oversize frames

0 4 collision frames 0 Undersize frames

0 5 collision frames 0 Collision fragments

0 6 collision frames

0 7 collision frames 3862892 Minimum size frames

0 8 collision frames 310945 65 to 127 byte frames

0 9 collision frames 132197 128 to 255 byte frames

0 10 collision frames 121005 256 to 511 byte frames

0 11 collision frames 205777 512 to 1023 byte frames

0 12 collision frames 98423 1024 to 1518 byte frames

0 13 collision frames 0 Overrun frames

0 14 collision frames 0 Pause frames

0 15 collision frames

0 Excessive collisions 0 Symbol error frames

0 Late collisions 19 Invalid frames, too large

0 VLAN discard frames 0 Valid frames, too large

0 Excess defer frames 0 Invalid frames, too small

6355271 64 byte frames 0 Valid frames, too small

999963 127 byte frames

413354 255 byte frames 0 Too old frames

455073 511 byte frames 0 Valid oversize frames

548471 1023 byte frames 0 System FCS error frames

5981463 1518 byte frames 0 RxPortFifoFull drop frame

0 Too large frames

0 Good (1 coll) frames

0 Good (>1 coll) frames

s-hw-2-1#

Leo Laohoo · ‎07-27-2012

Ok, now the output is getting very disturbing.

1. Link speed of Gi 0/11 is 10 Mbps. Why is that? What is connected to the far end?

2. Under the output of the "sh interface Gi 0/11", look at "Total Output Drops". This is generated when the switch tries to send data downstream (or down the wire) and the remote end is dropping packets. This counter should never drop to zero. It this event happens, the switch will record this and the counter increments.

3. Under the output of the "sh controll e Gi 0/11" look under "Invalid frames, too large". That's not good.

So what is connected to Gi 0/11???

Can you ask the person making the complaints to use another computer and see if he/she is exhibiting the same symptoms?

May I ask if you can run the following commands:

1. Command: test cable tdr int Gi 0/11;

2. Wait for approximately 7 seconds;

3. Command: sh cable tdr int Gi 0/11; and

4. Please post the output.

lcaruso · ‎07-27-2012

The system on that port Gi0/11 is probably a really old pc. I'll get more information next week and ask if more than one person is having the problem.

I did the upgrade so all stats will be based on

WS-C2960S-24TS-S 15.0(1)SE3 C2960S-UNIVERSALK9-M

Thanks much for the cable test commands. Results look normal.

s-hw-2-1#sh cable tdr int Gi 0/11

TDR test last run on: July 27 21:52:04

Interface Speed Local pair Pair length Remote pair Pair status

--------- ----- ---------- ------------------ ----------- --------------------

Gi0/11 10M Pair A 16 +/- 0 meters N/A Normal

Pair B 18 +/- 0 meters N/A Normal

Pair C 17 +/- 0 meters N/A Normal

Pair D 17 +/- 0 meters N/A Normal

After the upgrade and running for a full day on the new code after reboot, have no invalid frames too large but (again) have Total output drops of 21

Odd thing about this 10Mb/s system, whatever it is, look at the proportion of multicast frames xmit to everything else

s-hw-2-1#sh controll e Gi0/11

Transmit GigabitEthernet0/11 Receive

5504814 Bytes 0 Bytes

652 Unicast frames 0 Unicast frames

49952 Multicast frames 0 Multicast frames

13834 Broadcast frames 0 Broadcast frames

0 Too old frames 0 Unicast bytes

Leo Laohoo · ‎07-27-2012

Ok, I'll be checking the client at the remote end. 10 Mbps and you still get output drops does not sound right.