mGRE tunnel packet drops

mike.rootvik · ‎04-16-2008

I have an mGRE tunnel configured between 6504 switches ( running IOS12.2(18)SXF7). The two switchs are at remote sites with an IPSec tunnel connecting the 2 sites.

My issue is that I am seeing Output drops on the Tunnel interface, which is usually associated with failed network connection or data transfers. While the load level on the interface looks to be pretty minimal. This looks to be leading to lost connections during large data transfers and application connections that tend to send large data packets.

I have tried setting the MTU size to 1400 on the Tunnel interface and the next hop equipment just in case it is an issue with GRE tunnel packet fragmentation but it has not seems to make a difference. I have also increased teh Hold-Queue with out any affect.

This only seems to be a problem with connections that are sending large data packets.

The interfaces\ports used for the link between the switch and the IPSec tunnel device are set to be routeed ports (as opposed to switch ports).

The Tunnel interface config is below, this is the hub mGRE interface:

interface Tunnel1

description Test

ip address 172.23.254.1 255.255.255.0

no ip redirects

ip mtu 1400

ip flow ingress

ip nhrp authentication test

ip nhrp map multicast dynamic

ip nhrp network-id 123

ip nhrp holdtime 600

ip route-cache flow

no ip split-horizon eigrp 123

load-interval 30

tunnel source GigabitEthernet5/1

tunnel mode gre multipoint

tunnel key 123

hold-queue 500 out

end

Show Interface for the Tunnel interface:

Tunnel1 is up, line protocol is up

Hardware is Tunnel

Description: Test

Internet address is 172.25.254.1/24

MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec,

reliability 255/255, txload 113/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 172.25.242.5 (GigabitEthernet5/1), fastswitch TTL 255

Tunnel protocol/transport multi-GRE/IP, key 0x313, sequencing disabled

Checksumming of packets disabled, fast tunneling enabled

Last input 00:00:01, output 00:00:00, output hang never

Last clearing of "show interface" counters 00:20:27

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 115

Queueing strategy: fifo

Output queue: 0/500 (size/max)

30 second input rate 0 bits/sec, 0 packets/sec

30 second output rate 4000 bits/sec, 3 packets/sec

L2 Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes

L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast

L3 out Switched: ucast: 0 pkt, 0 bytes mcast: 0 pkt, 0 bytes

555 packets input, 72176 bytes, 0 no buffer

Received 0 broadcasts (332 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

3481 packets output, 935513 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

hadbou · ‎04-22-2008

check for the cable issues and also the speed and duplex configuration on interfaces where the ends of cable are connected as the issue happens during large data transfer.The speed and duplex configs should be the same.High cpu usage can also cause this issue.Altering MTU can also be a fix.

lamav · ‎04-22-2008

Mike:

Try applying the ip tcp adjust-mss 1436 command under the GRE interface at both ends. What this does is allow each side to advertise (not negotiate) the maximum size of the data portion of the TCP segment that each will accept.

And then change the ip mtu setting under the GRE interface at both ends to 1500.

These numbers aren't arbitrary. If you add the TCP header of 20 bytes and the IP header of 20 bytes, plus the GRE header overhead of 24 bytes to the TCP segment size of 1436, the resulting IP datagram will be 1500 bytes in length.

Apply these numbers and come back and tell us if the problem has gotten any better.

HTH

Victor

mike.rootvik · ‎04-28-2008

Thanks Victor,

Unfortunately that command is not supported on an mGRE tunnel using DMVPN.

I have found the source of the issue though. There is an ACL that is applied at a different location that is limiting access to the source system to only TCP established traffic. Whe the source tries to send a packet that is larger than the IP MTU setting, the switch at the source of the GRE tunnel issues an ICMP request to the source system requesting a smaller MTU. since the ICMP paket is getting blocked, then initial connections ends up failing.

Once the ACL is adjusted to allow the ICMP request to pass, the connection begins to work using a smaller MTU.

Thanks Mike