3750 switch Interface Packet Drops

[zhenningx]

We have wireless access points connect to c3750 switches version 12.2(53)SE2. The wireless performance is slow. I noticed packet drops on the switch interfaces connected to APs:

show mls qos interface f3/0/26 statistics

  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           2           0        1064
 queue 1:      322452      682655     4984891
 queue 2:           0           0     2273818
 queue 3:           0      139438  2757101931

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:           0           0           0
 queue 2:           0           0           0
 queue 3:           0       20103   171024282

Currently the switch port is configured as:

interface FastEthernet3/0/26
 description AP
 switchport access vlan XXXX
 switchport mode access
 switchport port-security aging time 1
 switchport port-security aging type inactivity
 srr-queue bandwidth share 10 10 60 20
 mls qos trust dscp
 snmp trap mac-notification change added
 snmp trap mac-notification change removed
 no mdix auto
 storm-control broadcast level pps 25 10
 storm-control multicast level pps 25 10
 storm-control action trap
 spanning-tree portfast
end

Question: I know the "srr-queue bandwidth share 10 10 60 20" command sets bandwidth share for the four queues. On the "show mls qos interface f3/0/26 statistics" command, I see lots of packet drops on queue 3 and the queue numbers from  that command starts from queue 0. So is the queue 0 actually the first queue in the "srr-queue bandwidth share 10 10 60 20" command? In another words, does queue 0 gets 10% and queue 3 gets 20% bandwidth from the above srr-queue command? If that is the case, I think I should increase the bandwidth share for queue 3?

Thanks.

[Leo Laohoo]

1. Post the complete output to the command "sh interface Fa 3/0/26 controll".
2. "no mdix auto" < -- Really???
3. 12.2(53)SE2 < --- Geez.

[zhenningx]

s1409_LIB(71)2-2#sh interface Fa 3/0/26 controll
FastEthernet3/0/26 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 001a.a180.cc9e (bia 001a.a180.cc9e)
  Description: AP
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 102/255, rxload 9/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:09, output 00:00:52, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 171907274
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 3783000 bits/sec, 2479 packets/sec
  5 minute output rate 40090000 bits/sec, 5889 packets/sec
     2897705676 packets input, 719234427386 bytes, 0 no buffer
     Received 377384 broadcasts (294833 multicasts)
     0 runts, 0 giants, 0 throttles
     5 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 294833 multicast, 0 pause input
     0 input packets with dribble condition detected
     7091168674 packets output, 5676998644242 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

     Transmit FastEthernet3/0/26              Receive
   3393529393 Bytes                       1978486710 Bytes
   2792820142 Unicast frames              2897347672 Unicast frames
      2959679 Multicast frames                294833 Multicast frames
       476804 Broadcast frames                 82551 Broadcast frames
            0 Too old frames              1867067097 Unicast bytes
            0 Deferred frames              105974603 Multicast bytes
            0 MTU exceeded frames            5441768 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames              569398 Minimum size frames
            0 8 collision frames           145100068 65 to 127 byte frames
            0 9 collision frames          2428678185 128 to 255 byte frames
            0 10 collision frames           67058487 256 to 511 byte frames
            0 11 collision frames           71811749 512 to 1023 byte frames
            0 12 collision frames          184507174 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 5 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
   2875289289 64 byte frames                       0 Valid frames, too small
     94968290 127 byte frames
    372819643 255 byte frames                      0 Too old frames
     82877497 511 byte frames                      0 Valid oversize frames
     93257585 1023 byte frames                     0 System FCS error frames
   3572011617 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

So does the "queue 0" in "sh mls qos interface statistics" command actually means "queue 1"from the queue-set command? And "queue 3" means "queue 4"? This part is confusing to me.

Thanks.

[Leo Laohoo]

The errors are from Total Output Drops. This means the switch, at FastEthernet speed, is trying to push data down to a downstream client but the downstream client could not process fast enough.
You'll need to have QoS or policy shaping enabled on this interface.

[Joseph W. Doherty]

So does the "queue 0" in "sh mls qos interface statistics" command actually means "queue 1"from the queue-set command? And "queue 3" means "queue 4"?

Yes, sometimes the 3750 refers to its egress queues as 0..3 and sometimes 1..4.

[Joseph W. Doherty]

I think I should increase the bandwidth share for queue 3?

You might try that, although recall increasing bandwidth share of one queue decreases if for another which unfortunately sometimes just moves the drops from one queue to another.

In the grand scheme, traffic queues when there's insufficient egress bandwidth. When that happens, if there's "too much" queuing, packets will be dropped. What bandwidth setting basically allows is determine what traffic, during such congestion is going to get forwarded and what traffic will get dropped. I.e., again, changing bandwidth percentage often just moves the drops about.

That said, there are two kinds of congestion, one being sustained, the other transient. For transient, insuring you have sufficient buffers for the transient congestion can avoid drops.

The 3750 series, do not have a lot of buffer RAM (2 MB per 24 copper port or per uplink ports), and what they have, might not be managed optimally for your environment (especially using Cisco's default settings). I've found using a buffer management approach, more like what was used by the earlier 35xx L3 switches (a shared common pool), often mitigates drops.

[zhenningx]

Thank you for the replies.

I changed the "srr-queue bandwidth share 10 10 60 20"to "srr-queue bandwidth share 10 10 20 60" to give queue 3 more bandwidth because queue 3 has lots of drops and no drops for other queues:

 output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:           0           0           0
 queue 2:           0           0           0
 queue 3:           0       20103   173799603

But still see packet drops at queue3 after this change.

I also changed the interface to use queue-set2 instead of queue-set1 as queue-set2 has more buffers for queue3:

#sh mls qos queue-set 2
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      16       6      17      61
threshold1:     149     118      41      42
threshold2:     149     118      68      72
reserved  :     100     100     100     100
maximum   :     149     235     272     242
#sh mls qos queue-set 1
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      10      10      26      54
threshold1:     138     138      36      20
threshold2:     138     138      77      50
reserved  :      92      92     100      67
maximum   :     138     400     318     400

Still has packet drops on queue3. Any other suggestions I should try?

The interface is used to connect a Cisco c3702 AP and that AP has lots of clients. Maybe there isn't much we can do with an old C3750 switch and we need to upgrade to c3850. Just trying to see if there are any parameters we can adjust to make it work.

[Joseph W. Doherty · Accepted solution]

You can try pushing your drop thresholds up, even to max for all. (Later IOS versions support 1600 3200 max, earlier versions 400 max.)

You can also try cutting way back (try 25, or less, for all queues) on your "reserved" settings. Your settings leave no pool buffers in queue set 2, and few in queue set 1, to "borrow" when needed.

[zhenningx]

I understand your points. Now I use queue-set 1 for the interface and it already sets to maximum 400 for queue 4. I am only concerning queue 4 now as it is the only queue dropping packets.

I changed the "reserved" setting to 25 for all queues:

#show mls qos queue-set 1
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      10      10      26      54
threshold1:     138     138      36      20
threshold2:     138     138      77      50
reserved  :      25      25      25      25
maximum   :     138     400     318     400

The issue is still not resolved.

  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           2           0        1105
 queue 1:      323933      685761     5078521
 queue 2:           0           0     2316935
 queue 3:           0      139593  2939270834

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:           0           0           0
 queue 2:           0           0           0
 queue 3:           0       20103   176129847

Thanks.

[Joseph W. Doherty · Accepted solution]

400 is the max allowed on your IOS version?

Did the drop frequency decrease?

If this is sustained congestion, only additional bandwidth, or advanced QoS techniques (not supported by the 3750), would help.

[zhenningx]

The drop frequency did not change much after the previous change.

The configurable threshold for max is 1-3200. I increased to 2000 and it made significant improvement. I still see packet drops but much less than before. My wireless speed test result from the AP is also greatly improved from 2Mbps to 10Mbps. Thanks a lot!