Packet drops increasing

[julia.joseph1]

Hi,

We have a Cisco 3750G stack and the WAN interface running at 1Gbps shows dropped packets; I am monitoring the BW, and it rarely seems to exceed 80%; is this oversubscribed, based on the output:

GigabitEthernet1/0/3 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 0022.5549.9803 (bia 0022.5549.9803)

  Description: 1G WAN

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 23/255, rxload 2/255

  Encapsulation ARPA, loopback not set

  Keepalive not set

  Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 14w5d, output 00:00:00, output hang never

  Last clearing of "show interface" counters 02:31:19

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1362

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 10287000 bits/sec, 5932 packets/sec

  5 minute output rate 91640000 bits/sec, 10410 packets/sec

     79843553 packets input, 18662140428 bytes, 0 no buffer

     Received 3103 broadcasts (3103 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 3103 multicast, 0 pause input

     0 input packets with dribble condition detected

     126785418 packets output, 135976477470 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

Thanks

[julia.joseph1]

Transmit GigabitEthernet1/0/3            Receive

   3999890550 Bytes                       3947062977 Bytes

    390923854 Unicast frames              2331172726 Unicast frames

      8526329 Multicast frames               3062910 Multicast frames

           65 Broadcast frames                     5 Broadcast frames

            0 Too old frames              3744910607 Unicast bytes

            0 Deferred frames              202152050 Multicast bytes

            0 MTU exceeded frames                320 Broadcast bytes

            0 1 collision frames                   0 Alignment errors

            0 2 collision frames                   0 FCS errors

            0 3 collision frames                   0 Oversize frames

            0 4 collision frames                   0 Undersize frames

            0 5 collision frames                   0 Collision fragments

            0 6 collision frames

            0 7 collision frames           694895175 Minimum size frames

            0 8 collision frames          3374814333 65 to 127 byte frames

            0 9 collision frames          1845528768 128 to 255 byte frames

            0 10 collision frames         2089710764 256 to 511 byte frames

            0 11 collision frames         1394601402 512 to 1023 byte frames

            0 12 collision frames         1524619791 1024 to 1518 byte frames

            0 13 collision frames                  0 Overrun frames

            0 14 collision frames                  0 Pause frames

            0 15 collision frames

            0 Excessive collisions                 0 Symbol error frames

            0 Late collisions                      0 Invalid frames, too large

            0 VLAN discard frames                  0 Valid frames, too large

            0 Excess defer frames                  0 Invalid frames, too small

    797007145 64 byte frames                       0 Valid frames, too small

   2316941009 127 byte frames

   2554922012 255 byte frames                      0 Too old frames

   4090372558 511 byte frames                      0 Valid oversize frames

   1815186702 1023 byte frames                     0 System FCS error frames

   1709922710 1518 byte frames                     0 RxPortFifoFull drop frame

            0 Too large frames

            0 Good (1 coll) frames

            0 Good (>1 coll) frames

[Leo Laohoo]

I'm going to "trust" the output of the command "sh controller e Gi 1/0/3" more than the "sh interface Gi 1/0/3".

Judging from that output, I would say "where's the problem"?  The output of the "sh controller e" is showing your lines are clean.  No suspicious Transmit or Receive line errors which leads me to believe to ignore the "output drops".

Last input 14w5d, output 00:00:00

I am, however, suspicious of this.  This is tellling me that the Gi 1/0/3 interface has not received any inbound packets for the last 14 weeks but is seeing outbound packets.  What is this downstream device?  Is it a client with dual/multiple NICs?

[julia.joseph1]

This interface is connected to the ISP MPLS switch. Conections to and from remote sites suppose to be through this link.

The discards are increasing as the traffic does too; in the last hour we got 53,000 discards; is this normal?

[Leo Laohoo]

ISP switch?  What is the the WAN speed?

Discards would mean that you are sending your data across without any traffic shaping and the next hop is pushing back.

[julia.joseph1]

The WAN speed is 1Gbps for this site (it is the data center). the traffic from  all the remote sites (at different speeds - 10Mbps and 100 Mbps) are coming through MPLS to the datacenter on this link.

There is QoS applied through MPLS on ISP's devices.

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Do you have QoS enabled?  If so, 2960/3560/3750 default buffer allocations often cause packets drops not seen when QoS is disabled.  If you don't need the QoS, disable if and see if your drop rate decreases.

If you believe you need QoS enabled, then tuning buffer settings may lower your drops.  (NB: one size doesn't fit all, so unable to suggest buffer settings without much, much more information.)

PS:

We have a Cisco 3750G stack and the WAN interface running at 1Gbps shows dropped packets; I am monitoring the BW, and it rarely seems to exceed 80%; is this oversubscribed, based on the output:

You cannot tell (whether you're oversubscribed) from just interface stats.  If aggregate ingress bandwidth is greater than max egress bandwidth, you're oversubscribed (which is quite common in networks).  Such oversubscription might show drops, but if you have multiple ingress interfaces, even when their combined bandwidth doesn't exceed max egress bandwidth, you may see drops too.  That's why I cannot tell from just egress interface stats.

Also, bandwidth utilization, alone, doesn't always correlate with drops.  You could have drops at 1% and no drops at 100%.  The fact you have any drops, though, tells us you have times of congestion where you run out of buffering.

[julia.joseph1]

We don't have QoS enabled on the switch; but the ISP has QoS enabled within the MPLS network.

It is interesting that a while ago I called TAC, and the  guy run the interface stats once and he gave the resolution: oversubscribed. He said that for 1GB link, there is way too much traffic and we need to either allocate a 10GB link or use etherchannels;  this is how I got the impresion that we might be oversubscribed.

Is there a way, or it is a good idea to increase buffering space?

Thank you,

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

If QoS is not enabled on the switch, buffer allocation is as good as it gets.

If you are oversubscribing your bandwidth, you could enable QoS to determine which traffic will be dropped first, but tuning 2960/3560/3750 buffers is "tricky".