Solved: DMVPN issue with dual IPS links at branch end

Erik Jacobsen · ‎12-03-2009

I have a setup (see drawing) where I have

dual ISP links at branch end, with with wireless and another with 3G,

Wireless should always be the primary path, when it is working (it is a ship so when it is in harbor)

If I use OSPF then it works fine the failover, but as soon as I enable IPSEC on the tunnel, then it will only failover once, and it will not failover to the primary again, without rebooting the router, and then it works for one failover again.

I'm using tracking also, since there is no interfaces there is going down

Are there anyone there have a working config, where ec. in the headend (normal setup) there is dual ISP links to the same router or ofcause the same as I have.

I'm willing to use any kind of protocols to get it to work, so RIPv2 (preferred), EIGRP, OSPF, tracking, IP SLA

Laurent Aubert · ‎01-12-2010

Who is 80.198.195.138 ?

The Hub peer address is 80.1.1.1 so can you ping this address when the primary link is down ?

Also it seems you can have IPSec tunnel 0 UP but not tunnel 0 and tunnel 1 at the same time. Verify you have the shared keyword on the hub router as you are using the same source IP address for both IPSec tunnel.

This message means the IKE database between the two routers are out of sync but should recover on its own.

HTH

Laurent.

View solution in original post

Laurent Aubert · ‎01-13-2010

Hi Erik,

Thanks for the clarification !! I think you don't need to hide this address anymore ;-)

If the remote is using the wrong source address for tunnel 1, it's a bug.

Can you try the following on the remote:

- Configure another profile IPSec (with the same parameters as the first one) with a different name, and applied it to tunnel 1.

- Try the lattest version

When you have the issue, try clearing the SA instead of rebooting (clear crypto isakmp sa and clear crypto sa).

You can debug IPSec with the following commands: debug crypto isakmp and debug crypto ipsec.

Run them before shutting down the primary link and see how the remote try to build IPSec tunnel1 and then clear all the SA's.

Laurent.

View solution in original post

kav · ‎02-16-2010

here is working, "multiple spokes/hubs with multiple ISPs to multiple spokes/hubs with multiple ISPs" example ( MSHWMI-2-MSHWMI :] )

techs used: dmvpn, ipsec, vrf, bgp (mostly for inter-vrf route redistribution (route-leaking)) (you can also use some IGP routing and redistribute its routes with BGP-inter-vrf-only for zero-touch configuration)

because of BGP is only routing protocol i used here, it is not 'zero-touch'-configurable, sorry. BGP - the best!

View solution in original post

Laurent Aubert · ‎12-03-2009

Hi Erik,

Can you post your config and some outputs of your fialover scenario ?

Thanks

Laurent.

Erik Jacobsen · ‎12-03-2009

Hi,

I have found this, there maybe can put me on the right track, so I will test this tomorrow:

http://www.cisco.com/en/US/technologies/tk583/tk372/technologies_white_paper0900aecd801d5004.html

Or else I will post my configuration tomorrow.

Erik

Erik Jacobsen · ‎12-07-2009

I have now been testing in regards to the link I have posted further down, and if I use Static IP's on interface Fastethernet 0/0, then it works fine. or let me put it this way it did friday via my GNS3, but today I can't get to work, but I think it is my GNS3 there is doing funny stuff.

But what I need is to have either DHCP on fastethernet0/0 or 2 statics as I have in the branch config attach.

But then I also need to change the default to the interface instead of the next hop, and I need to change the tracking to interface instead of static ip, and I need to change the route-map to 2 next hops IPs.

But this I can not get to work at all, maybe it is my GNS3 there is not working correctly or it is just not possible.

Anyone have any ideers?

Laurent Aubert · ‎12-07-2009

Hi Erik,

On your spoke site, you need to use your public address as the tunnel source address otherwise you will not be able to route your IPSec packets over the Internet:

interface Tunnel0

tunnel source fast0/0

!

Then because you are learning your address via DHCP on F0/0, your route-map to send the ping should be changed to this one:

route-map lokal_policy permit 10

match ip address 101

 set ip next-hop dynamic dhcp
!


With this configuration, the local ping will be sent only to the default GW learned from DHCP

Also, the tcp mss-adjust needs only to be applied on the tunnel interface of the hub router.

HTH

Laurent.

Erik Jacobsen · ‎12-08-2009

Hi Laurent,

Thanks for looking into this. I totally agree with you in regards with the route-map, but I have actually tried it, and it did not work.

But what I found out is this command is not working correct

IP route 0.0.0.0 0.0.0.0 fastethernet0/0 track 2

Because if I use static IP then the failover works fine, but as soon as I use the "interface" then it will not failover, the only guess I have here is that it is because the interface never goes down, because there is a bridge behind.

In regards to your command:

Interface tunnel0

tunnel source fast0/0

This will not work, since I have 2 interfaces. One primary interface and one secondary, so I need to find a solution where I can have failover. And I can not have 2 source interfaces, so the only solution would be a loopback. But I did notice yesterday that I could not get the tunnel up other than I made some static routes to the loopback ip address thrue my network, and this is ofcause not possible on a real setup, or else I could just have just a normal GRE tunnel, If I know the public IP in both ends.

Laurent Aubert · ‎12-08-2009

Erik,

After thinking about your design, here is my analysis:

- You could replace the interface by DHCP in the static route, but the track option is not available anymore so I would forget the tracking feature in your case.

- What you should do is to create two tunnels each using a different physical source interfaces and terminated on the same HUB. The Hub will see two different remote sites and by playing with the IGP inside the tunnel, you will be able to select one tunnel as primary and the other one as backup.

If something gets wrong on one physical path, the tunnel running over it will be impacted and you will converge on the other one

In this case, you dont need the following static route:

ip route 0.0.0.0 0.0.0.0 fastethernet0/0 track 2

I agree in a normal situation (both links are UP) the trafic from both tunnels will use the same path from the remote to the hub but the returning traffic will be received on both interfaces due to the different sources addresses which is enough to converge on the primary phyiscal path failure.

HTH

Laurent.

Erik Jacobsen · ‎12-21-2009

Hi Laurent,

I have been testing several options since you posted last time. And if I cancel the tracking and just have 2 default static IP route, then it only works if the interface goes down.

But in my case none of my interface will ever go down, since there is some equipment behind there is keeping the interface up.

I have also tried to remove my default statics, but then the router don't know how to get to the hubs public IP address.

So the only way I have found a working solution is with Statics and Tracking. BUT then again the crypto don't like it, it will failover once but not back. And it does it every time, it is like the IPSEC is hanging and will not switch over to the other tunnel.

Where if I use the DMVPN tunnel without IPSEC, then it works (only with statics) So I can not use DHCP.

And if I use 2 networks on my interface, then I can not use TRACKING, because I can only have one ip address to reach.

So again is there anyone there actually have a working configuration, where I can have failover interface?

Laurent Aubert · ‎12-21-2009

Erik,

As I tried to explain you in my previous post, your environment makes the failover based on the physical interface difficult so you need an indirect way to detect something is wrong with one physical path. What I'm suggesting is to create two tunnels instead of one and bind each tunnel to a WAN interface (via the tunnel source). Then you run an IGP inside both tunnels and you configure it to select one tunnel as primary and the second one as backup.

interface Tunnel0

...

tunnel source fastethernet0/0

tunnel destination 80.1.1.1

...

!

interface Tunnel10

...

tunnel source fastethernet0/1

tunnel destination 80.1.1.1

...

!

interface FastEthernet0/0

ip address 192.168.1.2 255.255.255.0

duplex auto

speed auto

!

interface FastEthernet0/1

ip address 88.1.1.2 255.255.255.252

no ip proxy-arp

duplex auto

speed auto

!

router rip

!

address-family ipv4 vrf dmvpn

redistribute static

network 10.0.0.0

no auto-summary

version 2

exit-address-family

!

ip route 0.0.0.0 0.0.0.0 fastethernet0/0

ip route 0.0.0.0 0.0.0.0 88.1.1.1 200

You will have to create a second tunnel interface on the hub site as well to terminate this new tunnel as two interfaces on a router can't belong to the same subnet.

If you loose your primary path, you should stop receiving updates via your primary tunnel, so you will converge to the routes learned from your backup tunnel.

HTH

Laurent.

Erik Jacobsen · ‎12-22-2009

Sorry for not posting my configuration from yesterday.

This time I have posted all the configurations plus explanitions to what I'm testing and with different setups with the statics.

So if you go true my configs and test, then you will see that:

Statics routes don't do exactly what we in theory believe it should

IPSEC is just a pain in the neck, and is giving a lot of issues.

I have seen the same issues in different codes, I'm using the newest code I could find:

Cisco IOS Software, C2600 Software (C2600-ADVSECURITYK9-M), Version 12.4(15)T11,

I have also tried RIP, but I got other issues with this, but I will gladly go back and try it. Just to prove my point that DMVPN is not working that well with Redundant links.

The HUB file is only the configuration of this.

Remote_lokation is with explanitions, tests and configs. (WITHOUT IPSEC)

Remote_lokation_IPSEC is with explanitions, test and configs with IPSEC.

Since I'm in Europe I don't always see my work e-mail in the evening, but try my private if you want something clarified:

erikholtemann@hotmail.com and I will gladly give you a call, just send me a e-mail with a phone number or I could give my number Laurent.

Again Thanks for helping out with this.

Best regards,

Erik Jacobsen (a desperate man)

Laurent Aubert · ‎12-23-2009

Don't give up ;-)

Few things about your testing:

1- You should remove your default static routes in the vrf on the remote site otherwise you will never use OSPF to converge from one tunnel to another:

no ip route vrf dmvpn 0.0.0.0 0.0.0.0 10.1.1.1

no ip route vrf dmvpn 0.0.0.0 0.0.0.0 10.1.2.1 50

2- On the hub, replace the redistribute connected by a redistribute static to announce the default route:

router ospf 1 vrf dmvpn

log-adjacency-changes

no redistribute connected

redistribute static subnet metric 10

!

Try again with the following two default routes on the remote site:

ip route 0.0.0.0 0.0.0.0 fastEthernet 0/0

ip route 0.0.0.0 0.0.0.0 88.1.1.1 200

For each test, you need to check both your global routing table and your VRF routing table to track your traffic: Which tunnel is chosen using which route (from the vrf RIB) and then which interface is used to route the GRE packets (global routing table). It's only this way you will be able to see if it's working as expected. Don't add the IPSec part if it doesn't work as IPSec will not help. It's a pure routing policy issue here.

Let me know your results

Thanks

Laurent.

Erik Jacobsen · ‎12-28-2009

Hi Laurent,

Sorry for first coming back now, I have been on christmas break. I have now tried your suggestions and if I remove the tracking from the defaults then it will not change over to the backup at all.

If I use tracking then the failover works everytime for the outside links.

But the OSPF for the tunnel is not working correctly, as you can see in the test attach. I have also attach the configuration again, so you can see what I have changed and so on.

The primary link and tunnel is working everytime I do failover and back.

But I can never get the backup line to work via the tunnel.

I have removed the IPSEC for now.

Best regards,

Erik Jacobsen

Erik Jacobsen · ‎12-28-2009

Hi Laurent,

PS. the defaults were actually there for a reason, because I need to have all other traffic (internet surfing and so on) over the tunnel, because we have one primary firewall. Where all traffic to the outside needs to go true.

So thats why I was using the commands:

ip route vrf dmvpn 0.0.0.0 0.0.0.0 10.1.1.1

ip route vrf dmvpn 0.0.0.0 0.0.0.0 10.1.2.1 50

Best regards,

Erik

Laurent Aubert · ‎01-01-2010

Hi Erik,

Happy new year !!!

You were right about the tracking option, it's required here.

The hub is not propagating the default route via OSPF but this is my mistake (sorry for that), please replace the redistribute static by

default-information originate under the router ospf 1 vrf dmvpn

Do not remove the default static route in the DMVPN vrf on the hub.

If after loosing your primary link, you also lost the OSPF adj in tunnel 1, it means there is something wrong in your router simulating the Internet.

Please verify you can still ping  88.1.1.2 from 80.1.1.1 when your primary link is down. If not fix this issue first and then everything should work fine.

HTH

Laurent.

Erik Jacobsen · ‎01-05-2010

Hi Laurent,

We are getting close now. But the DHCP is not working well with Tracking.

So I had to configure Static IP's on the uplinks on the remote lokation. But when I does this, then it is working pretty much without IPSEC.

The routing issue I had, the only way I could fix it was with changing the ARP TIMEOUT, on the primary interface on the remote router, and this fixed the issue.

I need to get the solution to work with IPSEC, and when I enable this, then I actually got it to work more or less. Other than I have to have a primary IP address and secondary network on the "primary interface" on the remote router. Because it is a ship there need to be able to connect to 2 different networks, one in each harbour.

But it look like the secondary network is not working at all, it just take down the primary interface and the secondary interface takes over, see attach with the tests I have been doing.

I changed the default-originate on the hub, and that works fine.

But we are getting very close.

Best regards,

Erik Jacobsen