Two tunnels, two different mtu values, no OSPF adjacency

warren.sullivan · ‎04-09-2015

Hi all,

Well this is an interesting one, i have a tunnel configured between two 2921 routers that used to work just fine, we had an management access issue on one of the routers and it had to be rebooted, all looked fine until one of OSPF adjacencies didn't come up, the one across the tunnel in question, ran a dubug and saw;

Apr 10 2015 09:58:30.319 AEST: OSPF-100 ADJ Tu100: Nbr 10.0.6.12 has larger interface MTU

OK, so i check both ends of the tunnel, MTU different;

bne-rt01#sh int tunn 100
Tunnel100 is up, line protocol is up
Hardware is Tunnel
Internet address is 10.0.3.62/30
MTU 17874 bytes, BW 204800 Kbit/sec, DLY 50000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel source 10.255.255.1 (GigabitEthernet0/1/0), destination 10.255.255.5
Tunnel Subblocks:
src-track:
Tunnel100 source tracking subblock associated with GigabitEthernet0/1/0
Set of tunnels with source GigabitEthernet0/1/0, 2 members (includes iterators), on interface <OK>
Tunnel protocol/transport GRE/IP
Key disabled, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Tunnel transport MTU 1434 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Tunnel protection via IPSec (profile "PTQ-IPSEC-PROFILE-2")
Last input 00:00:02, output never, output hang never
Last clearing of "show interface" counters 27w0d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 103048
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
2526404 packets input, 944901813 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2701610 packets output, 706777352 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

brn-rt01#sh int tunn 100
Tunnel100 is up, line protocol is up
Hardware is Tunnel
Description: PIPE_Backup
Internet address is 10.0.3.61/30
MTU 17866 bytes, BW 204800 Kbit/sec, DLY 50000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel source 10.255.255.5 (GigabitEthernet0/1/0), destination 10.255.255.1
Tunnel Subblocks:
src-track:
Tunnel100 source tracking subblock associated with GigabitEthernet0/1/0
Set of tunnels with source GigabitEthernet0/1/0, 2 members (includes iterators), on interface <OK>
Tunnel protocol/transport GRE/IP
Key disabled, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Tunnel transport MTU 1426 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Tunnel protection via IPSec (profile "PTQ-IPSEC-PROFILE-2")
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 01:04:19
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 3000 bits/sec, 1 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1723 packets input, 1069064 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
962 packets output, 91128 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out

checked the IP MTU, different;

bne-rt01#sh ip int tunn 100
Tunnel100 is up, line protocol is up
Internet address is 10.0.3.62/30
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1434 bytes

brn-rt01#sh ip int tunn 100
Tunnel100 is up, line protocol is up
Internet address is 10.0.3.61/30
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1426 bytes

Checked the source interfaces MTU, same;

bne-rt01#sh int gi0/1/0
GigabitEthernet0/1/0 is up, line protocol is up
Hardware is EHWIC-1GE-SFP-CU, address is 442b.03e5.8870 (bia 442b.03e5.8870)
Description: OPTUS IPVPN WAN
Internet address is 10.255.255.1/30
MTU 1500 bytes, BW 204800 Kbit/sec, DLY 10 usec,

brn-rt01#sh int gi0/1/0
GigabitEthernet0/1/0 is up, line protocol is up
Hardware is EHWIC-1GE-SFP-CU, address is 5057.a819.3620 (bia 5057.a819.3620)
Description: OPTUS IPVPN WAN
Internet address is 10.255.255.5/30
MTU 1500 bytes, BW 204800 Kbit/sec, DLY 10 usec,

Checked IP MTU on source, same;

bne-rt01#sh ip int gi0/1/0
GigabitEthernet0/1/0 is up, line protocol is up
Internet address is 10.255.255.1/30
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes

brn-rt01#sh ip int gi0/1/0
GigabitEthernet0/1/0 is up, line protocol is up
Internet address is 10.255.255.5/30
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes

below is the configuration of the physical source;

bne-rt01#sh run int gi0/1/0
Building configuration...

Current configuration : 209 bytes
!
interface GigabitEthernet0/1/0
description OPTUS IPVPN WAN
bandwidth 204800
ip address 10.255.255.1 255.255.255.252
ip traffic-export apply WAN-Traffic-capture size 1000000
duplex auto
speed auto
end

brn-rt01#sh run int gi0/1/0
Building configuration...

Current configuration : 209 bytes
!
interface GigabitEthernet0/1/0
description OPTUS IPVPN WAN
bandwidth 204800
ip address 10.255.255.5 255.255.255.252
ip traffic-export apply WAN-Traffic-capture size 1000000
duplex full
speed 1000
end

below is the configuration of the tunnel interfaces;

bne-rt01#sh run int tunn 100
Building configuration...

Current configuration : 245 bytes
!
interface Tunnel100
bandwidth 204800
ip address 10.0.3.62 255.255.255.252
ip flow ingress
ip flow egress
tunnel source GigabitEthernet0/1/0
tunnel destination 10.255.255.5
tunnel protection ipsec profile PTQ-IPSEC-PROFILE-2 shared
end

brn-rt01#sh run int tunn 100
Building configuration...

Current configuration : 270 bytes
!
interface Tunnel100
description PIPE_Backup
bandwidth 204800
ip address 10.0.3.61 255.255.255.252
ip flow ingress
ip flow egress
tunnel source GigabitEthernet0/1/0
tunnel destination 10.255.255.1
tunnel protection ipsec profile PTQ-IPSEC-PROFILE-2 shared
end

To be honest i'm a bit lost as to the reason, any ideas guys?

regards

warren

paul driver · ‎04-11-2015

Hello

has anything been physically changed regards connectivity between these routers?

you can negate the ospf mtu check by appling

int x/x

ip ospf mtu-ignore

res

paul

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Peter Paluch · ‎04-11-2015

Hi Warren,

According to this thread:

https://supportforums.cisco.com/discussion/11305311/understanding-mtu-given-gre-tunnel

the first displayed MTU of over 17000 bytes (let's call it simply tunnel MTU as opposed to the tunnel transport MTU) is computed from platform buffer sizes. I suspect that the memory-size iomem command may have something to do with this. Without this command, the router decides how much memory to set aside for packet buffers depending on the installed interfaces and network modules - and of course, based on the IOS version as well, as this automatic IOMEM sizing is performed by the IOS when booting, and different IOSes may do things differently. With this command configured, a fixed percentage of RAM can be reserved. Could something of this have changed between reboots - a different IOS version, perhaps, or adding/removal of this command, or change in installed network modules and interfaces?

Nonetheless, the difference between the tunnel MTUs (8 bytes) seems to correspond to the difference between the tunnel transport MTUs and IP MTUs. Somehow, the change in the tunnel MTU could have affected the resulting tunnel transport MTU.

Yet another question is whether the IPsec policy (transform set, key sizes, etc.) is identical to the IPsec policy before the reload. If my memory serves me, in IPsec, if two peers have identical but differently ordered ISAKMP and IPsec policies, the resulting IPsec operation depends on who first started the ISAKMP negotiation (the idea is: the initiator of the IPsec tunnel proposes all its ISAKMP, and afterwards, all its IPsec policies, and the target of the IPsec tunnel chooses the first one that matches one of its policies). If by any chance the configuration of ISAKMP policies, IPsec transform sets, etc. is not letter-identical on this router and the other endpoint, it is possible that the router now operates the IPsec differently than before the reload, which could affect the overall overhead, and thereby the tunnel transport MTU as well.

In any case, with tunnels, it is my strong recommendation to configure a conservatively low MTU manually. With IPsec-protected GRE tunnels over IPv4, the recommended manual MTU is 1400 and so the TCP MSS should be clamped to 1360. The recommendation for MTU of 1400 is taken from this document:

http://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html#t16

I would personally suggest configuring the MTU manually on all your tunnels to 1400. This will both provide for a reasonable reserve in case your own ISP uses some additional overhead to carry your packets, and at the same time, it will prevent your routers from automagically (and obscurely) changing your tunnel transport MTUs between restarts.

Best regards,
Peter