DMVPN Deployment with existing IPSec/GRE Tunnels - Big trouble

adam · ‎05-31-2007

In a valiant effort to deploy Dynamic Multipoint VPN (DMVPN) technology across 14 corporate office locations, the process has been met with nothing but extreme trouble. These locations have always had an "almost mesh" of legacy static crypto-map based IPSec/GRE tunnels. The DMVPN deployment was to cleanup the configuration, simplify everything and provide the full mesh necessary and desired (hub to spoke and spoke to spoke). The entire architecture was setup in our lab environment, EIGRP tested and everything looked great.

A wonderful strategy of overlaying the DMVPN tunnel interfaces onto the production routers and slowly migrating traffic onto the DMVPN structure was drafted and agreed to. Change control windows were approved and the deployment phase of the project kicked off.

The first phase of the deployment simply called for adding a DMVPN Tunnel0 interface to each SPOKE location's router and letting it register with a single HUB. No actual EIGRP neighboring or production traffic to pass back and forth, just a simple mGRE interface with NHRP registration. A sample of this SPOKE configuration is shown below.

When the DMVPN interfaces were configured and NHRP registration occurred, all looked good. Within 5 to 30 minutes, randomly existing traditional static IPSec/GRE tunnels (crypto map style) began to simply fail. Either the SA would completely disappear for the tunnel, or the tunnel would simply blackhole the traffic. Of course, the first indication of failure is the EIGRP neighbor down.

Shutting down the Tunnel0 interfaces on the spokes and hub would eventually allow the legacy static tunnels to operate again.

I have had a case open with Cisco TAC, but really have not been able to get any good response. They would like to issue some simple "show crypto" commands while it is happening. Each time I reproduce this problem, it affects production traffic of a $6B company with 19000+ employees. So getting a change window arranged knowing you are going to break things is not always easy.

My initial take on the situation was, there is a severe conflict in the IOS code when DMVPN and traditional crypto maps are used together and involve the same public peer addresses. I discovered the ever under documented and scarcely mentioned "shared" keyword for the tunnel protection statement, and thought that was going to be my hero. But, unfortunately it had absolutely no affect on things, the problem still remained.

Next, I decided to come up with a deployment plan that avoided ever establishing a DMVPN tunnel with the same public peer address, but again as soon as I started up the Tunnel0 interfaces, random static tunnels would start to fail.

I have tried extensively to reproduce the problem in our lab setting (mixture of 2800 and 7200 routers), but everything runs perfectly.

The hub location is actually a new router, with now legacy static tunnels configured.

Does anyone in the community have any thoughts?

Has anyone experiencing anything similar?

Are there some hints and tricks to overlaying DMVPN on top of legacy tunnels?

The IOS utilized on all the routers is 12.4 series, most of which are 12.4(13b).

carenas123 · ‎06-06-2007

I think while deploying DMVPN over existing setup, terminating the tunnel temporarily or manually clearing the SA's could help. Following links may help you

http://www.cisco.com/en/US/tech/tk583/tk372/technologies_configuration_example09186a008014bcd7.shtml

http://www.cisco.com/en/US/tech/tk583/tk372/technologies_white_paper09186a008018983e.shtml

JOSH GANT · ‎06-06-2007

I've done a few DMVPN implementations but none where there is a DMVPN connection between routers and a GRE/IPSec between the same routers.

Is there any way you can bring in a temporary hub router at the man site? You could cut the spokes over slowly to DMVPN and then migrate them back to the real hub router after the static GRE/IPSec configs are gone? (Use your lab 7200?) You probably thought about this already....

kbingeman · ‎06-07-2007

I have done what you are trying to do and hav experienced the same problem. The issue I found was that the ISAKMP key for the static crypto tunnel and the ISAKMP key for the DMVPN tunnel would conflict.

When the SA expires, it would find the other key and the tunnel dies. Removing the static crypto maps and isakmp keys fixed the issue to allow the DMVPN tunnels to work.

My recommendation, without knowing too much about the production environment and resource available to do this would be to setup dedicated DMVPN hub router. This can be on the Internal network or at the edge and "cut-over" each site one by one.

The process would be:

a) have your dmvpn hub configured completely with dmvpn, eigrp, etc....

b) find the cut-over window for the remote site and remove the static crypto ipsec setup and replace it with the dmvpn configuration.

c) on the hub site, ensure eigrp picks up the new routes for the hub.

If you need spoke-to-spoke tunnels available then you'll have to upgrade the spoke sites more agressively.

I think this was a pretty clean process...just took some time but in the end - along the way everything worked very well and is sstill functioning great.

adam · ‎06-07-2007

You are almost exactly correct.

After much time with the TAC and many hours in my router lab, it was finally concluded that the ISAKMP SAs do conflict. It actually has to do with the outbound traffic being grabbed by the DMVPN tunnel protection profile (which is really a crypto map entry underneath the hood). Once I had a good understand of the problem, I was able to reproduce it by either clearing the existing SAs or setting the lifetime extremely short.

The drafted solution to this mess, is to establish secondary public peer addressing on each location's tunnel router, either with secondary interface addressing or a loopback interface. This makes DMVPN run on one set of addressing, and static tunnels on the other set of addressing. The only trick to this is, the traditional static tunnels must utilize the primary address of an interface. This is because the crypto map local-address command only accepts interface names, not IP addresses (which I think should be enhanced).

I have tested both secondary addressing and loopback addressing in my lab environment. I was able to get traditional static tunnels operational with DMVPN. Again, the key to the problem is separating the source addressing (or destination, depending on which angle you are looking at). This seems to cure the confusion entirely.

I am getting ready to establish secondary/loopback addressing on all my production routers, then proceed with a very similar deployment plan to my original, overlapping the technologies and migrating site by site off the static onto dynamic.

This has to be one of the most painful cases I have had with Cisco in my 10+ years of networking. It really just boils down to not enough intelligence in the IOS code to handle the situation.

If more information comes about, I will post it to this thread for all to benefit from in the future.

Adam Hudson