GRE Tunnel Output Queue Drops

Unanswered Question
Mar 12th, 2008

Running a GRE tunnel between two 3845s with full DS3s. DS3/tunnel only running at around 8Mbps and picking up significant "output drops" on the Tunnel interface but not on the DS3 which is carrying the tunnel. Trying to see what these drops represent. Not seeing any problems with Apps but may want to run voice and need to understand these drops. Standard GRE tunnel with FIFO. No IPSec.




I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
royalblues Thu, 03/13/2008 - 00:57

What is the IOS version you are running. This may be a cosmetic bug

Also what is the bandwidth configured under the tunnel interface?

By default the tunnel assumes a bandwidth of just 9K and hence can show drops. Make sure you have the correct bandwidth configured under the tunnel interface


jtorjman Thu, 03/13/2008 - 06:31

3845 Enterprise Version 12.4(12)

Bandwidth is 44210 to reflect the DS3.

Tunnel1 is up, line protocol is up

Hardware is Tunnel

Internet address is

MTU 1514 bytes, BW 44210 Kbit, DLY 500000 usec,

reliability 255/255, txload 21/255, rxload 18/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 1.x.x.x (Serial1/0.1), destination 2.x.x.x

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255

Fast tunneling enabled

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

Last input 00:00:04, output 00:00:04, output hang never

Last clearing of "show interface" counters 09:35:07

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 122496

Queueing strategy: fifo

Output queue: 0/0 (size/max)

30 second input rate 3125000 bits/sec, 1624 packets/sec

30 second output rate 3734000 bits/sec, 1523 packets/sec

7059033 packets input, 1565205618 bytes, 0 no buffer

Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

5990442 packets output, 2562730255 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out



royalblues Thu, 03/13/2008 - 06:43


From your output

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

This should show the DS3 bandwidth

Can you post the interface configuration

As said earlier, it could also be a cosmetic bug.



Joseph W. Doherty Thu, 03/13/2008 - 06:50

What's your CPU look like when you peak out at 8 Mbps?


Do you use the ip tcp mss-adjust?

jtorjman Thu, 03/13/2008 - 08:20

CPU averages 10% regardless of load and CPU history shows no spikes.

I am trying to look for commands for any fragmentation/reassembly etc....

Show IP traffic is for the whole router. Checking the code for any cosmetic bugs.

One thing I omitted is that the Serial carrying the tunnel has a policy applied but shows no drops.

interface Serial1/0

no ip address

encapsulation frame-relay

load-interval 30

dsu bandwidth 44210


serial restart-delay 0

service-policy output WAN_EDGE_OUT

class-map match-any SERVICE_CONTROL

match ip dscp af31

match ip dscp cs3

class-map match-any NETWORK_CONTROL

match ip dscp cs6

class-map match-all REAL_TIME

match ip dscp ef

class-map match-any BULK_DATA

match ip dscp af11



policy-map WAN_EDGE_OUT


priority percent 34


bandwidth percent 5


bandwidth percent 3


bandwidth percent 33

class class-default


queue-limit 350

interface Tunnel1


bandwidth 44210

ip address 10.

ip mtu 1400

ip tcp adjust-mss 1400

load-interval 30

tunnel source Serial1/0.1

tunnel destination

Thanks Again


Joseph W. Doherty Thu, 03/13/2008 - 09:00

Reason I asked about CPU, believe GRE is often (always?) not in the fast switch path.

I too have seen drops on GRE tunnels, when they shouldn't be stressed. Never discovered exactly why. Have suspected during bursts CPU drops some. I've also tried increasing outbound queue on tunnel, but seems to make little difference.


I wonder about:

interface Tunnel1


bandwidth 44210

ip address 10.

ip mtu 1400

ip tcp adjust-mss 1400

load-interval 30

tunnel source Serial1/0.1

tunnel destination

Shouldn't mss be at least 40 bytes less than mtu?

Why mtu so small? I don't recall GRE using 100 bytes.


Also recall there's a dynamic MTU probe you can configure for tunnel; don't recall the exact command.

rseiler Fri, 09/12/2008 - 01:00

Several problems here.

First, you don't want to adjust the ip mtu at all, particularly on a GRE tunnel. You should only be changing the ip tcp adjust-mss to 1460 or 1380. By changing both the ip mtu and adjust-mss to the same value, you broke tcp across this link because you rendered useless the benefit of tcp adjust-mss on this interface. Also, depending on IOS version and hardware platform, you may need the tcp adjust-mss statement on the ingress interface of the router, not the tunnel.

Second, never set the queue-limit to anything greater than 100 packets, particularly as a part of a service-policy. Remember, every class has a 64 packet queue-limit, so the total hold-queue for the interface will be the sum of all the class queue-limits. By making this number crazy high, you are essentially allowing duplicate packets to get into the hold-queue. In other words, by the time the tcp sender timed out and resent the packet, the first packet may not have been sent!

This is bad, the tcp mss-adjust misconfiguration only made it worse.

Joseph W. Doherty Sat, 09/13/2008 - 04:52

Richard, you make a great point about not setting the tcp adjust-mss value to the same value as ip mtu, but my thinking is a bit different on some of your other points, so thought I present them for your feedback.

First, about not wanting to set ip mtu at all, as I mentioned in my prior post, I would lean toward using the dynamic tunnel mtu probe(, using the command tunnel path-mtu-discovery [if supported]). Both, however, are recommended by Cisco in the following whitepaper, in the "More Recommendations" section (near the bottom of the document). Could you expand upon why not to use ip mtu?

On your second point about never setting queue-limit greater than 100 packets, and that every class has a (default) 64 packet limit, and the interface total would be the sum of all, you make another great point about TCP needlessly resending packets that have timed out, but are actually still sitting in a queue.

(Item to note, most platforms CBWFQ classes do default to 64 packets, but not all do. I know the 7500s don't. I suspect the 6500/7600 FlexWAN or SIP-200 may not either, since they appear to support 7500 QoS.)

TCP's RTO (retranmission timeout) should be dynamically adjusted while it's measuring RTT (round-trip time). So, TCP might not send lots of needless packet retransmissions just because the packet is sitting in a deep queue, although I too think it is more likely, especially if the active queue depth is very variable.

Sizing overall queue buffer depth, in packets, should depend on link's bandwidth ("faster" can drain a larger packet queue in the same time), whether BDP (bandwidth delay product) is to be supported, and expected packets sizes (e.g. VoIP packets generally are smaller than typical FTP packets). So, I believe proper sizing queue parameters can be a bit complex, but unsure providing or exceeding some total value, such as never greater than 100, is always bad or "crazy". Is there a particular reason for limiting overall queue depth to 100 or less in all cases, or do you really mean to emphasis that queues can be sized too large? (If the latter, I agree; there's also sometimes the consideration of buffer resources available on the platform.)

rseiler Sun, 09/14/2008 - 05:40

Are you running GRE inside an IPSEC tunnel? Your configurations posted so far do not reflect this. That would be an entirely different scenerio.

If not, and you are simply running a GRE tunnel between these two endpoints, and the end-to-end ip mtu OUTSIDE the tunnel is 1500 bytes (the tunnel's ip path) then you do not (and must not) change the ip mtu under the tunnel interface. The document you reference only mentions changing the default ip mtu on the tunnel in two scenerios: 1) the path mtu of the tunnel is less than 1500 bytes (ipsec, pppoe, etc.) or 2) the entire network supports large frames and can accept GRE packets of 1500 bytes (and therefore GRE/IP frames of 1524 bytes. In either case, this should be deterministic and incorporated into your design so should be hard set and not rely on path-mtu discovery, which rarely works outside the gre tunnel.

You do, however, need to use tcp adjust-mss on the ingress interface of this router, using this command on the tunnel does not always work depending on IOS version. I use a value of 1380 but values up to 1460 should work. This is why certain tcp transactions will fail across the tunnel, udp packets are rarely larger than 1200 bytes and work by default.

Regarding the inbound hold-queue size, you will not find *any* document on Cisco's web site that lists an inbound hold-queue of anything greater than 100, particularly if using CBWFQ. Ignore any documents that are more than 4 years old or refer to modems, isdn, etc.

Usually, you will set the hold-queue or CBWFQ queue-limit *SMALLER* to more aggressively tail-drop traffic in the default class so that tcp makes the right decision about bandwidth.

Remember, the input hold-queue does not drain at all unless the output interface tx-limit and tx queue are nearly empty. IOS puts back-pressure on the input interface when the output interface is saturated and you want the input interface to tail-drop, not buffer a bunch of packets unrealistically.

Joseph W. Doherty Sun, 09/14/2008 - 09:26

Richard, again you're quite correct where IP MTU shouldn't need to be configured on a pure GRE tunnel supporting and end-to-end MTU of 1500 bytes (assuming standard Ethernet?), although you now note a couple of exceptions. It was your usage of "at all", in your earlier post's, "First, you don't want to adjust the ip mtu at all, particularly on a GRE tunnel.", I thought the statement might be somewhat misleading, i.e. that there weren't any cases where the command should be used. I agree with "hard set" in what you now write "In either case, this should be deterministic and incorporated into your design so should be hard set . . .". (Another instance I could see using it [IP MTU], would be on the LAN side interface of a WAN router which only supports traffic to/from a GRE tunnel.)

In cases of running GRE tunnels over something like the Internet, I'm unsure I agree with not using tunnel path-mtu-discovery since often the end-to-end path MTU is never guaranteed. (I do agree PMTU, itself, can be an issue.)

I'm a bit lost about your references to "inbound" hold-queues, unless you meant outbound.

I disagree with making outbound queues "*SMALLER*" to aggressively tail-drop so that TCP makes the right decision about bandwidth. Just as you're correct, I believe, to be concerned about oversizing queues; making them too small can have a very adverse impact with TCP performance. Too small queues will often not allow TCP flows to correctly determine and utilize bandwith. Lots of tail drops, besides also causing retransmissions, can often force a TCP flow back into slow start instead of just congestion avoidance.


This Discussion