We have several routers - 7201, 3825. On GE interfaces on all routers we see growing number of output drops even though inerfaces are highly underutilized. All speed/duplex settings are correct (mostly auto), on both side of link there are no ethernet errors, nothing. And still I see output drops.
And on the same interfaces there are input flushes. What is the meaning of flushes? I read that it's packets discarded from queue. But for what reason may packets be discarded from queue?
Here is show inter output. Note, it's early morning here, very low output rate, hence only 1000 drops in 40 minutes.Later it will be more.But output rate in this interface will not be above 1mb
sh int gi0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is 001d.701e.6c1b (bia 001d.701e.6c1b)
MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 18/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is RJ45
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:39:30
Input queue: 0/75/1/377 (size/max/drops/flushes); Total output drops: 1057
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 71086000 bits/sec, 12064 packets/sec
30 second output rate 36000 bits/sec, 21 packets/sec
26971664 packets input, 2017299015 bytes, 0 no buffer
Received 1856 broadcasts, 0 runts, 0 giants, 0 throttles
1 input errors, 0 CRC, 0 frame, 0 overrun, 1 ignored
0 watchdog, 11123 multicast, 0 pause input
0 input packets with dribble condition detected
82168 packets output, 39108559 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
no ip address
If utilization is low its possible you have a misconfiguration on the way you set up your jumbo frame support for your interfaces. We haven't done jumbo frames much but know it could cause symptoms like you are seeing. Might want to check out the jumbo frame config docs on the cisco website.
Hello Glen and Olga,
enabling the jumbo frames could have the effect to shorten the frame buffers on the port because they are reshaped to be able to host frames up to 9216 byte size.
If this happen the probability to fill these buffers even with low traffic increase and you can see input and output errors.
Hope to help
It is also worth a try to hard set the speed/duplex settings. Also you may check the mtu settings of the device connected to the interface.
The trouble with "load" is it's based on an average, even when set for 30 seconds, you can't easily see short term bursts (especially with jumbo frames).
Assuming a 1 ms latency, 40 packets is really too small for gig for 1500 byte packets. Try doubling it or try WRED.
You input flushes aren't real high, but since you show an actual drop, we know the queue has overflowed. For it, you might try a slight increase, try 100.
Glen and Giuseppe, I think that very well may be a problem. I'll try to change mtu as soon as possible and report results here.
Naveen, in some configs we have hard settings and too have drops :(
josephdoherty, I thought that input queue will be filled only when packets are going to be processed switched. We have cef enabled, I was hoping queue will not be filling that much. But nevertheless, I'll give it a try.
Thank you all for your input!
Well, we found out why we have output drops. We have several subinterfaces with rate-limits and this drops are from rate-limit drops :) Once we removed all rate-limits all drops are gone.
Still we have flushes and playing with input queue had no success. I have a thought - may it be packets, destined to IP's that have arp unresolved? on these subinterfaces there are incomplete entries (it's normal situation). I wonder where goes packets for which router can't find arp entry after several retries?
"josephdoherty, I thought that input queue will be filled only when packets are going to be processed switched. We have cef enabled, I was hoping queue will not be filling that much."
Even with CEF, I believe it's still possible to have some process switched packets for some situations. Showing the switching stats (with hidden the hidden command) should give some indication whether this is happening (and how much).
I also believe some other conditions can cause an input queue to fill. On some routers, when the BGP scanner fires off, packets tend to queue in the input queue.
I read you other note that you found the primary cause being rate limits, for the outbound drops, and that you did try to increase the input queue. With regard to the latter, you didn't see any reduction in the percentage of input queue flushes?
No, I think not, it stayed the same. when i do "show int gi0/0" I see flushes growing, but I've never seen more then 1 packet in queue (first number, size of queue). And "sh buffers input-interface gi0/0" shows nothing no matter how fast i hit "up and enter" :))
It's not totally unexpected that flushes continue to increment, but unless you record detail stats, it might be hard to determine if the rate of flushes growth has decreased.
What did you try, 100?
I have the same problem of increasing input flushes on Cisco 7201.
Did you solve the problem on your router?