Mike Sullenberger is a Distinguished Cisco Support Engineer and industy expert on DMVPN. This document contains the answers provided for the questions asked during the live "Ask the Expert" Webcast session on the Topic - Dynamic Multipoint VPN (DMVPN): Design and Positioning.
The Complete Recording of this live Webcast is present below:
Dynamic Multipoint VPN (DMVPN)
A. A virtual private network (VPN) is a network that uses a public telecommunication infrastructure and its technology, such as the Internet, to provide remote offices or individual users with secure access to their organization's network. It aims to avoid an expensive system of owned or leased lines that can be used by only one organization. The goal of a VPN is to provide the organization with the same secure capabilities but at a much lower cost.
3. Q. We've been using DMVPN v2 for a long time. At Networkers, you talked about the benefits of v3 (summarization, etc). Assuming my routers support the v3 code, is there any reason NOT to move to v3?
A. There really isn't. Basically, v3 is better than v2 in all respects.There's nothing in v3 that's worse than v2. I'm not forcing people to go to v3 because I know in a large network it can be work because some configuration changes are needed. In that case you can stick with v2 for a while. I'm not planning to get rid of v2, but you will eventually want to go to v3. The best way to migrate to V3 is using the same routers that I have out there I build a v3 network in parallel with my v2 network and then I can slowly move my spokes one by one to v3 without having to take them down so it's sort of a migration. Once I migrate all my spokes over to v3 then I can start tearing out my V2 network. In other words you can have your V3 and V2 network running in parallel. To do that you're going to play with the routing protocal metrics, but that's the usual technique I have for a large network. I don't like flash cutovers. I prefer more of a migration system.
4. Q. Is ASR currently supporting the DMVPN phase 3 features. I have not seen any CCO document that shows phase 3 support with ASR. Even the feature navigator does not show it.
A. We're going to have to fix that . DMVPN phase 3 support on the ASR started in release 5. I believe that's 2.5.x. It has been out for 6 months or better now. We're actually up to release 7. If you need to use DMVPN with front-door VRF then you have to pick up release 7 which is out now because that's where that's support came in. All the base code for version 3 came out in release 5 on the ASR.
A. You can. There's one restriction, which is that each DMVPN spoke must get a unique outside NAT address. We can't do PAT. Even though IPsec is NAT aware and it can handle PAT situations NHRP knows nothing about ports or anything else. All it knows is about IP addresses so the only way it can differentiate spokes behind the same NAT router is if those spokes get a unique outside NAT address.
6. Q. Do I need direct connection between hubs in a dual DMVPN dual hub scenario or should those hubs be in diferent locations?
A. When we're running with multiple hubs on the DMVPN network there is no requirement that the hubs be next to each other. They could be in the same room, or different buildings, or different cities. You will need a tunnel connection between them except for in a DMVPN phase 1 network where we can use a backend land multi hop connection between them, but in any phase 2 or phase 3 network we must have a tunnel connection between them.
A. 500 is supported by many different platforms. The most important thing on a hub is the number of routing protocol neighbors. Depending on the routing protocol, in fact you're within the number that, let's say a 7200, that easily supports 500. OSPF, EIGRP, BGP, whatever routing protocol you choose, will support 500. If you want to go with a smaller box a 3945e is a great box. It's a brand new box, but it's showing to be quite nice for DMVPN. It's going to support around the same numbers, maybe 10% less than a 7200. If you need a lot more encryption throughput than those boxes can support, like in the 1 gig or up range then I would probably recommend an ASR 1,000 because we then have access to the very powerful encryption modules, the 2 gig and up modules. With that size you have some choices.
In general, we don't run GETVPN over DMVPN. You end up with double encryption. There's a couple different ways of combining GETVPN and DMVPN. In fact, we're looking at methods inside Cisco to make some changes in the code to make it a little more transparent when you want to use GETVPN or DMVPN. If you run DMVPN without encryption you certainly can run GETVPN. In other words you encrpyt the data traffic with GETVPN and then you use DMVPN as part of your transport. The point is that your host addresses are going to be visible in the middle network. Now somebody would have to get a pack and dig past the GREIP header down into the GRE packet and find the IPheader in there to see those addresses. So in general, no, we usually recommend either building a GETVPN network or a DMVPN network. But you can do this. That is probably something you'd want to come to Cisco AS with to decide if that's an appropriate network for you and what other options you might have.
Q. In a 300-spoke DMVPN network over MPLS how do I monitor the creation and deletion of spoke-to-spoke tunnels?
A. We're improving that and we've done some improvements in the last year or so. It used to be that the only way to monitor that was to look at the IPsec MIB and have it tell you when we're building IPsec encryption and decryption tunnels because when we build a new DMVPN tunnel the first thing we do is call IPsec and say 'IPsec I need you to set up encryption for this.'We have added some DMPVN MIBS. We just recently put out some traps so we can actually tell you when NHRP is creating or tearing down tunnels. That is probably your best way, using either the IPsec MIBS or the NHRP / DMVPN MIBS.
10. Q. Can we have unequal cost load sharing riding on an EIGRP network with an unequal dual ISP bandwidth subscription at the spoke router?
A. I think I could design that. We certainly can have setups or designs where I have two ISPs and I basically want to load balance my traffic across both ISPs. In a DMVPN network the best way to do that is actually to build two DMVPN networks, one that runs over ISP 1, and one in parallel that runs over ISP 2. Then I load balance my data traffic into the two different tunnels. That gives me the best load balancing across the two. The problem in this is there's an assumption that the ISPs are basically equal and it doesn't matter which way I go. I can influence the traffic going into the two tunnels by using the EIGRP metrics. I can say Tunnel 1 is better than Tunnel 2 and therefore the traffic is going to use Tunnel 1. And I can use some unequal load balancing under EIGRP to even that out. It might be trickly to try to match that against the real ISP throughput capabilities or bandwith capabilities because that is sort of invisible when you're looking at the EIGRP running at the tunnel layer. We do support OER, so you can have OER running at the physical layer and basically run all your traffic over your primary ISP and OER can say 'This looks like it's getting a little loaded. Let me shift the tunnel over to the second ISP.' This is a tunnel-by-tunnel shift. We don't always know how much traffic is going through a particular tunnel. We can also run OER at the tunnel layer so if I have my data traffic and I have a choice of Tunnel 0 or Tunnel 1 I can send everything over Tunnel 0 and let OER shift some of my data flows over to Tunnel 1. We have a number of different options and I think we'd have to play with these to get what you want.
11. Q. Can I implement 802.1x port authentication with DMVPN? If so, should I do it using the same phases 0 to 3 cisco recommends on its Cisco.com site?
A. We don't use 802.1x authentication with DMVPN. I think you're trying to run 802.1x on the tunnel interface to authenticate your spokes as they come into the hub. We use ISAKMP authentication as our authentication mechanism. When a spoke comes in the first thing it goes over between the spoke and the hub is an ISAKMP session and one of the first things that happens as part of that session is the spoke has to authenticate with the hub. If it doesn't authenticate then we can't bring a tunnel up. In that way we really don't support 802.1x.
12. Q. We are looking to do a Fresh deployment of DMVPN as a migration from our Dialbackup solution. This will only be a Hub and Spoke solution. Hub is ASR1004 and Spokes are 2951 or 3845. The primary circuits are EVPL or Frame Relay. Do you see this a good fit for dial backup?
A. Yes, I have a number of customers who are doing this. One thing to keep in mind with DMVPN is that we bring up the spoke hub tunnels immediately and keep them up all the time. The assumption is that there are no per minute charges for your physical path from the spoke to the hub. If there is I've done some techniques...let's say I want a DMVPN as a backup or I have a primary MPLS network that I use and I want DMVPN as a backup in case the MPLS link goes down. There are techniques now that we can implement so that the DMVPN is all set up but it's held in a down state so no traffic goes over the dial backup link until we monitor using object tracking. We monitor the MPLS path. When that path goes down or is not available anymore then we trigger the DMVPN tunnel to come up and it takes about 10 seconds. Sometimes it comes up in 5 seconds. If we have to trigger up the tunnel and it has to trigger up the physical link then it may take 20 or 30 seconds. And then we switch back the DMVPN will go back into a quiescent or held down state.
A. Yes, you can run DMVPN on a C881 router as well. It is Cisco IOS software-based solution. You can certainly utilize DMVPN even if your router C881 is behind the PAT device.
A. You can't compare the two, since they are different technologies. DMVPN can run over the public network infrastructure. GETVPN can only run over a private WAN where you have end-to-end routing capability due to its header preservation requirements. Also, GETVPN is for always-on full mesh network topologies, while DMVPN is better targeted at either hub and spoke or dynamic spoke to spoke scenarios. There are other differences, but the bottom line is you'd really need to look at your deployment requirements to know which one is better suited for your particular environment.
A. By VTI, I assume you mean static VTI, since dynamic VTI is supported with EzVPN only. The main advantages of DMVPN are dynamic spoke to spoke tunnels and ease of management and configuration since you'd no longer need to define any point to point tunnels.
A. You can use a 2547oDMVPN to pass the MPLS traffic. It can run on your PE routers. DMVPN has multiple phases that allow you to have Hub-spoke and spoke-to-spoke design. When using 2547oDMVPN you can only use Hub-spoke topology.
17. Q. What's the best way to configure Phase 3 with 3 global hubs and allow connectivity directly between spokes without transitting back through my corporate network?
A. The DMVPN phase 3 allows you to have spoke to spoke functionality. You can utilize a hierarchal concept of phase 3 and have your hub in each separate region. If one region spoke wants talk to other region spoke, the NHRP resolution request will forward using routing path to central hub and then eventually get to destination spoke that will answer the NHRP resolution request.
18. Q. In reference to the last question, can I have two DMVPN clouds on one hub? For example, a hub in West Canada would have one DMVPN cloud for the primary connection of sites in West Canada (let's say 40 sites) and one DMVPN cloud for the secondary connection for sites in East Canada (+/- 40 sites).
A. Yes you can have dual DMVPN clouds to use in that setup.
19. Q. We use MPLS as a primary connection - but we often have problems with sites going offline due to a failed link. We could buy a redundant MPLS link, but that would double our costs. Can DMVPN be used to supplement an MPLS network; i.e. can it be setup to only be active when the primary MPLS link is down?
A. Yes, you can do this. With DMVPN, you have an overlay routing infrastructure in the VPN layer. So you can use routing metrics to keep the DMVPN network as a backup. However, it will be active in the sense that the control plane activities, ie., NHRP, crypto, and routing protocols, etc., will always be active unless you want to deploy things like EEM to manage the backup connection.
A. In a remote office/teleworker scenario, either EZVPN or DMVPN will work since you typically do not require dynamic spoke to spoke connectivity. The difference there is whether you want to have an overlay routing in the VPN layer, or keep it simple like EZVPN. This really depends on your specific deployment requirements.
A. The GETVPN allows you to preserve the IP header and let you use your native routing compare to DMVPN it adds another IP header and then you do overlay routing. GETVPN is group based encryption concept, it provides you method to encrypt the traffic over MPLS cloud , some customer has requirement to encrypt traffic even it is private cloud.
22. Q. I saw in scaling that DMVPN is around 10,000, 3000. Is this 10,000 for a hub-spoke network while 3,000 is for a mesh network? What is actually the scaling?
A. You can use an SLB design in DMVPN for large number spokes like 10k . This design is very common in POS (point of sale ) setup. The scaling number depends on the platform. You can use a 7600/6500 using a VPNSPA module or even go with ASR 1000 as well.
A. No, DMVPN is not supported on the ASA. It will only work between routers running IOS.
A. Unfortunately DMVPN is only a Cisco IOS based solution, it is not supported on ASA. However, you can certainly utilize site-to-site VPN in ASA without using DMVPN.
25. Q. We have a network where remote sites only talk to hub sites. So traffic is point-to-point. Remote just needs to get to the hub. Remote should not talk to remote. In this case, is it a good idea to use DMVPN with ACL since it does reduce the configuration on the hub site where one interface can serve all remotes? Or should we still use the p2p GRE/IPSEC tunnels?
A. You can use DMVPN phase 1. That will only be a Hub and spoke setup. However, one spoke can still talk to the other spoke through Hub. In that event if you don't want one spoke to talk to the other spoke, you can use ACL in the Hub to block it.
26. Q. We use DMVPN over the Internet. At times, we see Internet routing issues where the any-to-any model fails - like one spoke can't communicate with another spoke, but communication is ok between both spokes and the hub. During these times, we find traffic between spokes fails because the spoke-to-spoke tunnel fails to build. The problem is that a viable path still exists through the hub, but the traffic won't traverse the hub. Is there any way to fix this?
A. Typically, spoke-to-spoke traffic will traverse the hub while it waits for the direct s2s tunnel to establish. However, there are different scenarios where this may not work correctly. Without knowing the specifics of the failure condition, it's hard to say how to fix this. I would suggest you open a TAC case to have this looked at before looking to find a solution to fix this.
A. Yes. You can use your QoS and forward it through the tunnel. If you are doing classification based on IP ToS values then you don't have to do anything as that part of the IP header copies over to the outside header as well. We also have a per-tunnel QoS feature that lets you configure a QoS policy in the tunnel interface, that is egress from the hub-spoke direction. This feature helps you limit the amount of bandwidth to each spoke based on your requirements and prevents one greedy spoke from starving other spokes.
28. Q. What is the best way to solve the packet fragmentation issue? Many articles exist with different options like 'clear df-bit'. What is recommended for DMVPN?
A. "Clear df-bit" doesn't really come into play with DMVPN, since by default GRE encapsulation does not copy the DF bit in the GRE delivery header. The best practice to deal with fragmentation issues is to set "ip mtu" on the tunnel interface to 1400 bytes, and tcp adjust-mss to 1360. This should account for most of the GRE/ipsec encapsulation overhead such that the resulting ESP packets does not exceed the normal 1500 bytes of mtu.
A. No. The per-tunnel QoS feature only controls the policy management on the hub, and has nothing to do with the actual packet classification. You'd still need to configure pre-classify on the tunnel interface if your QoS classification requires L3/L4 info. Also, it's better to do inbound DSCP/ToS marking than to use pre-classify to avoid feature complexity.
A. We have a feature called per-tunnel QoS. But that will be used in the Hub to spoke direction not from the spoke to hub direction. For spoke to hub you can use regular QoS and apply the service policy on the physical interface of the spoke. If you are doing classification in Layer 4 above information then you need to use QoS pre-classify.
31. Q. DMVPN reduces IP MTU; Do you have a list of common Max MTU lengths with or whitout encryption? (like no encryption, with ESP-AES128 etc..)
A. I don't believe we have such a list publicly available on CCO. But here are some common ones. GRE overhead is 24 bytes (28 with tunnel key). Crypto in tunnel mode is somewhere around 60 bytes (it varies since there is padding involved, and AES has a 16-byte IV compared to DES/3DES's 8), NAT-T is another 8 bytes, PPPoE (for DSL) is 8 bytes. There are others, but they are not as common.
32. Q. If I have one hub in a data center in Eastern Canada and another hub in Western Canada, how many DMVPNs do I need on each hub to have all the sites in Eastern Canada to have their primary connection to the data center in the East and the secondary connection to the datacentre in the West and vice-versa for the sites in Western Canada?
A. You can use a Single DMVPN dual Hub design or a dual hub dual DMVPN. It all depends how many sites we are talking about and the traffic requirement. When using a Single DMVPN dual Hub, which means you have one IP network subnet in your tunnel interface and you are using two hubs for redundancy. You can have the Eastern Canada and Western Canada spokes connect to one data center as a primary and if that goes down routing protocol will utilize the second hub without interruption. As for spoke prospective they have dual IPSEC/NHRP tunnel with both Hub but for forwarding plane prospective only one is active at one given point.
33. Q. Can you implement the DMVPN through the ASA firewall, (i.e.) have the router behind an ASA firewall?
A. Yes, you can. In this case, you'd use NAT-T for the IPSec tunnel. You want to use transport mode ipsec for DMVPN over NAT. There are other restrictions as far as whether dynamic spoke to spoke tunnels will work. There is a document on CCO that talks about this topic, look for "dmvpn and nat".
A. No, currently DMVPN is only supported on IOS routers, and not on the ASA.
A. You can tune the NHRP holdtime to extend the time that the spoke-to-spoke tunnel will get torn down. But by definition, DMVPN is not best suited for a full mesh always-on topology. If you have an always-on requirement, and if you have a private IP infrastructure, then maybe GETVPN is a better fit?
36. Q. Do your hubs share the same IP address range or do you have two separate tunnels on your edge routers?
A. There are 2 different redundancy deployment scenarios. In a dual-hub dual-DMVPN setup, each spoke will have 2 tunnel interfaces towards the hub. In this case, your hubs would have ip addresses in different networks. There is also the dual-hub single DMVPN scenario where each spoke would only have 1 tunnel interface that points to both hubs as its NHS. In this case, the hubs would have to have their addresses in the same network.
A. If you put a Hub behind a Firewall, make sure the Firewall is doing static translation, not PAT. You need to open UDP 500 , ESP (Ip protocol 50 ) and AH (IP protocol 51) if using AH . If the spokes are behind a NAT device then you need to allow UDP 4500 as well because the ESP packet will be encapsulated in UDP 4500.
38. Q. What would be the equivalent the DMVPN on the ASA that allows for routing protocol or dynamic routing of the private address space?
A. As you know DMVPN is only Cisco IOS based solution so that is not supported on ASA. You can terminate site to site tunnel on ASA and have router behind ASA can have GRE tunnel terminating to pass the routing information within GRE packet.
A. Typically we only recommend having the multicast source behind the hub, and not behind a spoke. The multicast stream will only be sent to the spokes with receivers behind them that have joined the multicast group.
A. In most situations, clear IP NHRP will clear the dynamic NHRP entries, and subsequently clear the IPSEC sessions on the hub. Note: You can't clear the static NHRP sessions, so you'd have to manually clear the crypto session by using clear crypto session or clear crypto sa/isa.
A. Yes. It all depends on your type of connection and latency in your network. If your round trip delay is less then 150 msec then you won't see any issues even if you have DMVPN spoke to spoke communication.
A. Currently there is no plan to support DMVPN on the ASA.
A. Yes it is supported in UC500 as they are using Cisco IOS software.
44. Q. Is the only difference between Phase 2 and Phase 3 the hierarchical "additions" to Phase 3? Is there a document outlining the differences?
A. The main difference between phase2 and phase3 is that DMVPN phase3 provides the NHRP shortcut switching capability, thus eliminating the need for the spoke to have the specific network prefixes for all other spokes. For details, see http://www.cisco.com/en/US/docs/ios/ipaddr/configuration/guide/iad_nhrp_dmvpn.html.
45. Q. I have a site that migrated to DMVPN and RADIUS authentication between a Cisco 4400 WLC and a Microsoft IAS server that is not longer working. Since the IP-Sec/GRE MTU size is now 1380 (mss 1340), is there any known issue with 12.4(24)T2 that could cause issues at an application layer for UDP port 1812? Packet fragmentation maybe? Hardware on both ends is Cisco 7201. Routing protocol is EIGRP.
A. Typically RADIUS does not use large packets for access request/accept messages, so this shouldn't come into play. Also, most UDP applications do not have the DF bit set, so if indeed the packets are larger than the ip mtu on the tunnel, these packets would just be fragmented before GRE encapsulation, so it should still work. For this particular problem, I would suggest you open a TAC case.
A. We have a session in Networkers that covers the troubleshooting DMVPN (BRKSEC-3012). You can see the slide deck for that session.
A. DMVPN uses GRE (multipoint GRE in this case) over IPSec, whereas ASA uses native IPSec encapsulation. Currently, DMVPN is not supported on the ASA.
48. Q. I have a 2800 router behind a firewall which is solely acting as an MGCP voice gateway. If I configure it as a DMVPN hub also, is there anything I should be concerned about?
A. Make sure the NBMA address that will be using that router that will be acting as a hub has static translation in Firewall, not PAT. We suggest no translation for that address, as that is the address all your spokes will use to come in to build the tunnel.
49. Q. If you have two hubs in your network, do you prefer to have two tunnels on the spoke for redundancy, or just one tunnel with both hubs using IP NHRP mappings?
A. It depends. You can have a single DMVPN dual hub design. You only have a single tunnel interface in the spoke and two NHS servers configured in the tunnel interface of the spoke, so in the event one goes down or becomes unreachable the routing protocol will start to use the second one. Here is the link for some configuration: http://www.cisco.com/en/US/tech/tk583/tk372/technologies_white_paper09186a008018983e.shtml.
A. The main difference between phase 3 and phase 2 is the NHRP shortcut switching capability offered in Phase 3, which eliminates the need for a spoke to have all the routing prefixes for other spokes in order to build spoke to spoke tunnels. For details, take a look at: http://www.cisco.com/en/US/docs/ios/ipaddr/configuration/guide/iad_nhrp_dmvpn.html.
51. Q. What are the main differences between a straight site-to-site VPN and a DMVPN (I am not clear if the "site-to-site" term implies a specific technology or is a generic phrase).
A. Typically, a "straight" site-to-site VPN uses just IPSec. DMVPN uses GRE (in this case multipoint GRE) over IPSec and it's capable of building dynamic spoke to spoke tunnels where as site-to-site VPNs are almost always static point to point tunnels.
A. Yes, you can use DMVPN , as long as you have controlled latency below 100 msec. Spoke to spoke reduces the latency so instead of traffic going through hub to another spoke, it directly contacts the spoke.
53. Q. Can you provide a Cisco document which gives a complete picture on how to configure a working DMVPN? If so, please provide a link.
A. You can see the details at http://www.cisco.com/en/US/tech/tk583/tk372/technologies_white_paper09186a008018983e.shtml
A. Do you mean recommended version?
A. 1400 is the recommended IP MTU on tunnel interface.
57. Q. Is there any expection that DMVPN will be supported not only in routers, but ASA Firewalls as well?
A. Currently there is no plan to support DMVPN on the ASA platforms.
A. They are working on it as far as i know to include ISR G2 (39xx,19xx,29xx) as well.
A. DMVPN is supported only on Cisco IOS routers. It is not supported in ASA.
A. DMVPN is Cisco IOS based solution so it is supported.
A. Dual DMVPN is a little easier to manage when you have to fine tune the routing metrics, but in general, single DMVPN, dual/multi-hub is still recommended. That said, this really needs to be looked at based on the specific deployment requirements.
62. Q. If I am using DMVPN without encryption crypto then which ports are should be opened if the hub is behind the firewall?
A. Dual DMVPN is a little easier to manage when you have to fine tune the routing metrics, but in general, single DMVPN, dual/multi-hub is still recommended. That said, this really needs to be looked at based on the specific deployment requirements.
63. Q. Can you explain what is needed for a zero touch deployment and exactly how it connects to the hub the first time to get the configuration?
A. Zero touch configuration in a hub means nothing is needed to configure the hub even if you add more spokes. In spokes you configure Static NHRP mapping for the hub, so once the spoke bootup or no shut the tunnel interface, it will start contacting the hub NBMA address which is where you are mapping in a static NHRP configuration . Check this link for more configuration. http://www.cisco.com/en/US/tech/tk583/tk372/technologies_white_paper09186a008018983e.shtml.
A. This really depends on your specific deployment scenario. For small scale setups, you should be able to use point-to-point GRE, GRE over IPSec, VTI, etc., without having to use DMVPN. With large scale setups, you also have choices between EZVPN, GETVPN, and DMVPN. All are targeted at different deployment requirements. It really needs to be reviewed on a case by case basis: http://www.cisco.com/en/US/tech/tk583/tk372/technologies_white_paper09186a008018983e.shtml.
65. Q. In a dual hub scenario where the hubs are in different networks, is having dual DMVPN networks connect the spokes to each hub and the two hubs connected as a spoke to each other’s networks supported?
A. It is supported, although you are not going to be able to create dynamic spoke to spoke tunnels between 2 spokes that belong to the 2 different DMVPN domains.
66. Q. Can you provide more details about NHRP shortcut and redirect? is that applied on the HUB, SPOKE or BOTH?
A. Usually IP NHRP redirect applied on Hub and NHRP shortcut apply on spoke. However it all depends network design if you have a design in which hub is also utilize as a spoke and need to build spoke to spoke tunnel then you need to configure NHRP shortcut in hub as well.
A. For troubleshooting DMVPN, I would suggest check session BRKSEC-3012.
A. The VPN SPA does not support DMVPN phase 3.
A. Slides can be downloaded by clicking on the “Download PDF” button on the console (upper right corner).
Q. Can I download this live show?
A. The webcast cannot be downloaded, but you may get a copy of the slides by clicking on the “Download Slides” button on the console. An archive of the webcast will be available for viewing/replay by September 17, 2010.