In a valiant effort to deploy Dynamic Multipoint VPN (DMVPN) technology across 14 corporate office locations, the process has been met with nothing but extreme trouble. These locations have always had an "almost mesh" of legacy static crypto-map based IPSec/GRE tunnels. The DMVPN deployment was to cleanup the configuration, simplify everything and provide the full mesh necessary and desired (hub to spoke and spoke to spoke). The entire architecture was setup in our lab environment, EIGRP tested and everything looked great.
A wonderful strategy of overlaying the DMVPN tunnel interfaces onto the production routers and slowly migrating traffic onto the DMVPN structure was drafted and agreed to. Change control windows were approved and the deployment phase of the project kicked off.
The first phase of the deployment simply called for adding a DMVPN Tunnel0 interface to each SPOKE location's router and letting it register with a single HUB. No actual EIGRP neighboring or production traffic to pass back and forth, just a simple mGRE interface with NHRP registration. A sample of this SPOKE configuration is shown below.
When the DMVPN interfaces were configured and NHRP registration occurred, all looked good. Within 5 to 30 minutes, randomly existing traditional static IPSec/GRE tunnels (crypto map style) began to simply fail. Either the SA would completely disappear for the tunnel, or the tunnel would simply blackhole the traffic. Of course, the first indication of failure is the EIGRP neighbor down.
Shutting down the Tunnel0 interfaces on the spokes and hub would eventually allow the legacy static tunnels to operate again.
I have had a case open with Cisco TAC, but really have not been able to get any good response. They would like to issue some simple "show crypto" commands while it is happening. Each time I reproduce this problem, it affects production traffic of a $6B company with 19000+ employees. So getting a change window arranged knowing you are going to break things is not always easy.
My initial take on the situation was, there is a severe conflict in the IOS code when DMVPN and traditional crypto maps are used together and involve the same public peer addresses. I discovered the ever under documented and scarcely mentioned "shared" keyword for the tunnel protection statement, and thought that was going to be my hero. But, unfortunately it had absolutely no affect on things, the problem still remained.
Next, I decided to come up with a deployment plan that avoided ever establishing a DMVPN tunnel with the same public peer address, but again as soon as I started up the Tunnel0 interfaces, random static tunnels would start to fail.
I have tried extensively to reproduce the problem in our lab setting (mixture of 2800 and 7200 routers), but everything runs perfectly.
The hub location is actually a new router, with now legacy static tunnels configured.
Does anyone in the community have any thoughts?
Has anyone experiencing anything similar?
Are there some hints and tricks to overlaying DMVPN on top of legacy tunnels?
The IOS utilized on all the routers is 12.4 series, most of which are 12.4(13b).