OSPF stuck in INIT over GRE tunnel with IPsec

[jonmo2578]

Hi there,

Im having problems re-establishing ospf adj over a GRE tunnel as soon as I apply any ipsec configuration onto it..

Ive seen a few threads covering similar problems but not found a solution yet..

When the tunnel is set up without any ipsec configuration, ospf adj successfully comes up

Spoke end ( Cisco 2800 )

interface Tunnel5111 

ip address 10.180.51.134 255.255.255.252

no ip redirects

ip mtu 1438

ip pim query-interval 1

ip pim sparse-mode

ip virtual-reassembly

ip ospf network point-to-point

ip ospf hello-interval 1

ip ospf dead-interval 3

tunnel source 84.x.x.x

tunnel destination 81.x.x.x

Head end. ( Cisco 3900 )

interface Tunnel5111

ip address 10.180.51.133 255.255.255.252

no ip redirects

ip mtu 1438

ip pim query-interval 1

ip pim sparse-mode

ip virtual-reassembly in

ip ospf network point-to-point

ip ospf dead-interval 3

ip ospf hello-interval 1

tunnel source 81.x.x.x

tunnel destination 84.x.x.x

As soon as I apply the following ipsec configuration, OSPF adj falls over..

tunnel mode ipsec ipv4

tunnel protection ipsec profile ipsec-test

The spoke end then gets stuck in an OSPF state of INIT

RTR#sh ip ospf ne

Neighbor ID     Pri   State           Dead Time   Address         Interface

10.10.12.3        0   INIT/  -        00:00:02    10.180.51.133   Tunnel5111

but the Head end router does not see any OSPF state at all to the spoke.

Reading  various threads, this would indicate that the router is receiving  Hellos but not getting the router ID.. to progress to EXSTART..

Both routers can ping either end on the source and destination addresses… 84.x.x.x. and 81.x.x.x

Also both routers can ping the other end of their tunnel, 10.180.51.133 and 134

Have tried creating two different tunnels now to the 3900 router and both encounter the same problem.

To  confuse matters, if I set up the tunnel to our older 3700 VPN head  router using exactly the same config with ipsec enabled on the tunnel,  OSPF adj does come up !!..

although I did have to lower the mtu  value to 1438 for it to establish.

So to me, this says that this should work using ipsec over the tunnel ?

Has anyone encounter this problems before and managed to solved it.

Any help would be much appreciated..

Thanks.

Jon.

[sathvik k v]

Hi,

configure ip ospf mtu-ignore on the interface and try.

Refer: http://networker.technolabs.org/?p=7

Regards,

Sathvik

[jonmo2578]

Hi Sathvik,   thanks for the suggestion.. i have tried the " mtu-ignore" command already and it didnt work..

[sathvik k v]

Do you see any any retransmissions when u debug ip ospd adj ??

Try this :

interface Tunnel5111

ip tcp adjust-mss

The purpose of tcp mss-adjust is to avoid any packet fragmentation from the initial TCP handshake.  If PMTUD is active and working correctly, fragmentation will be detected, but you still take a slight performance hit as packets are rejected then retransmitted; then repeat.  (NB: also the older version of PMTUD doesn't inform the sender of the actual MTU and slows the process even more doing trial and error.)

Normally you set the tcp mss-adjust to reduce the normal MSS by whatever your MTU reduction is.  So, for example if sending "normal" size Ethernet across a typical GRE tunnel, MTU is 1500 and MSS is 1460, so you subtract 24 (GRE overhead) from MSS and use a setting of 1436.  Even smaller might be set to allow for expected, or unexpected, header options.

As to where to set it, best is anywhere along the path where only the necessary traffic will be modified.  If you have a router with only such traffic passing through it, you could set on the LAN side or WAN side.  However, if the router has multiple interfaces, and might be routing LAN to LAN traffic, then you, ideally, only want to apply command on the necessary WAN interface.

The command only affects TCP traffic, and only TCP during it's initial handshake.  If traffic has different paths, with different MTUs, especially if the path selection can vary dynamically, it's possible either traffic won't be adjusted that should be or conversely traffic is adjusted that shouldn't be.

PMTUD should still be used in conjunction with tcp mss-adjust.

Also refer : http://packetlife.net/blog/2008/nov/5/mtu-manipulation/

http://www.bitsonwire.com/?p=30

Regards,

Sathvik

[jonmo2578]

Hi Sathvik,

Thanks again for the reply.. Im afraid I looked at the same link earlier and tried the same thing.. Our orginal config did have " ip tcp adjust mss 1436 " in there.. 

we have tried several mtu values and have just amended the "ip tcp adjust mss" value in turn.. No luck..

Im wondering if its mtu related at all now..

Not seeing any retranmissions appearing in the debug ip ospf adj..

The confusion for me here is that the spoke router is in state INIT and the head router shows nothing ?..

but this only happens from our Cisco 2800 to new Cisco3900..

Everything is good when try 2800 to older 3700 router.. ?!

[sathvik k v]

Hi,

INIT state implies that router showing neighbhor in INIT state  has seen OSPF hellos from the neighbor but the neighbor routers has not seen hello packets from the local router. In your scenario spoke is seeing hello's.

you can give a  try  by reducing the tcp adjust-mss to a lower value 1350 or configuring MTU and TCP adjust-mss on the physcial interface.

Regards,

Sathvik

[jonmo2578]

Thanks again for the advice Sathvik..

Still getting no luck when adjusting the mtu and adjust-mss values.. have also tried applying to the physical interface too and not getting any change there.. Not seeing anything new in the ospf debugs either.

Will keep plugging away ..

Jon.

[Rolf Fischer]

Hi Jon and Sathvik,

MSS Clamping (ip tcp adjust-mss) affects TCP data-plane traffic.

There is no relation to OSPF, GRE or IPSec, which all use IP for transport.

MTU mismatches can lead to problems when the LSDB synchronization starts, but here the problem is that we're not receiving a hello packet which confirms bidirectional connectivity.

Since you're stucking in INIT, have you tried a debug ip ospf hello?

Your're using quite aggressive timers, maybe until the cause of the problem is found you should increase them a little bit?

One more idea for isolating the problem: If possible, could you configure the tunnel-interfaces as network type NBMA with static neighbor-peerings? This setup uses unicast instead of the AllSPFRouters multicast address. You said, you have connectivity between the tunnel-addresses, so I'd be courious what happens when you change the OSPF communication to unicast.

Hope that helps

Rolf

[jonmo2578]

Hi Rolf,

thanks for the reply.. i had a play with static ospf statements the other day but couldnt get them to work.. anyway, just had another go with the static statements after your suggestion and it seems we run into the same problem, where OSPF comes up across the tunnel without encryption but as soon as I reapplied the ipsec config on the tunnel it breaks..

So this works...

interface Tunnel5111 ( at spoke end )

ip address 10.180.51.134 255.255.255.252

no ip redirects

ip mtu 1438

ip pim query-interval 1

ip pim sparse-mode

ip virtual-reassembly

ip tcp adjust-mss 1374

ip ospf network non-broadcast

ip ospf priority 0

tunnel source 8.x.x.x

tunnel destination 81.x.x.x

But adding this breaks it..

tunnel mode ipsec ipv4

tunnel protection ipsec profile ipsec-test

You'd think that this would point to a problem with the crypto config.. but it seems to work from Cisco 2800 -> 3700..  weird..

On the original setup, I also tried setting the hello and dead timer back to default ( rather than 1 / 3 ).. and that made no difference.... as you say, it does seem to be hello pkt related.. as we are seeing hellos coming from the hub ( 3900 ) to the spoke ( 2800)  but the reply does not seem to get back to the 3900

Hub. ( sends out hellos but gets nothing back )

OSPF-100 HELLO Tu5111: Send hello to 224.0.0.5 area 51 from 10.180.51.133

OSPF-100 HELLO Tu5111: Send hello to 224.0.0.5 area 51 from 10.180.51.133

OSPF-100 HELLO Tu5111: Send hello to 224.0.0.5 area 51 from 10.180.51.133

repeats

Spoke ( recieves hellos and sends reply but does not reach Hub )

OSPF: Send hello to 224.0.0.5 area 51 on Tunnel5111 from 10.180.51.134

OSPF: Tunnel5111: Interface state change to UP, new ospf state P2P

OSPF: Tunnel5111: Route adjust notification: UP/UP

OSPF: Rcv hello from 10.10.12.3 area 51 from Tunnel5111 10.180.51.133

OSPF: Send immediate hello to nbr 10.10.12.3, src address 10.180.51.133, on Tunnel5111

OSPF: Send hello to 224.0.0.5 area 51 on Tunnel5111 from 10.180.51.134

OSPF: End of hello processing

OSPF: Send hello to 224.0.0.5 area 51 on Tunnel5int tunnel 5111         

OSPF: Send hello to 224.0.0.5 area 51 on Tunnel5111 from 10.180.51.134

OSPF: Rcv hello from 10.10.12.3 area 51 from Tunnel5111 10.180.51.133

OSPF: Send immediate hello to nbr 10.10.12.3, src address 10.180.51.133, on Tunnel5111

OSPF: Send hello to 224.0.0.5 area 51 on Tunnel5111 from 10.180.51.134

repeats

[sathvik k v]

Hi Rolf,

Thanks for correction.

MTU mismatches would cause problems during LSDB syncronisation.

Regards,

Sathvik

[Rolf Fischer]

Hmm, when adding the IPSec-config you still can ping the other end of the tunnel but unicast OSPF hellos are not transmitted?

I have to say I'm running out of ideas.

Have you considered to try another IOS?

Best regards

Rolf

[jawad-mukhtar]

strange but it was resolved check this link

http://www.sadikhov.com/forum/index.php?/topic/184650-ospf-stuck-in-init/

Jawad

[Rolf Fischer]

Hi Jawad,

thanks for sharing the link.

To me it appears a little bit confusing, first he says a format flash solved the problem, later he found an error in the tunnel config.

The problem here is a little bit different; we have OSPF perfectly working when using an unencrypted GRE tunnel and as soon as the tunnel is changed to VTI IPSec mode we miss OSPF hellos in only one direction while having basic connecitvity to the tunnel end. And when the Head End router is replaced by an older one with the same setup, everything works.

Regards

Rolf

[paul driver]

Hello,

debug  ip ospf adj

Is there any ACls probhiting the hello messages? test on sides-
access-list 100 permit ip host xxxxx host 224.0.0.5

debug ip packet detail 100

I can also see you have change the ospf type to non broadast, which mean you need a neigbour statment for a unicast hello packet.

Do you have any authentication set?

Lasty what kind of interface are you sourcing the tunnel from?

res

Paul

Please don't forget to rate any posts that have been helpful.

Thanks.

[Rolf Fischer]

Hi Paul,

thanks for joining!

The NBMA type was just a try to make sure that the problem remains when OSPF communication is done in unicast, I think they changed it back to point-to-point meantime (debug shows the AllSPFRouters multicast destination).

I'd like to add two more questions for Jon:

With the VTI IPSec setup,

1) have you tried pinging the other tunnel end with larger packets and the df option? Since IPSec adds some overhead, the encrypted OSPF hellos have a larger size. I don't think we have a MTU problem but it's worth checking.

2) have you tried other control plane communication between Head- and Spoke-end, e.g. telnet or ssh? I wonder if the problem only affects OSPF.

Maybe a show ip traffic shows some conspicuous counter rising (IP and OSPF statistics)?

Regards

Rolf