Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

Design Single DMVPN with Dual Hubs as a redundant path over internet

Technical background

One of the major concerns and challenges that pertain to site to site VPN implementations using hub and spoke topology with a large number of sites is scalability. In spit of the fact that implementing many GRE tunnels over IPSec with a dynamic routing protocol can scale well. However the number of Access lists and point to point tunnels will be hard to manage when there is a large number of remote sites using fully of partially meshed topology. In addition to the scalability issues, the implementation of a large number of site to site VPNs using hub and spoke topology with a big number of spoke to spoke communications, this will lead to a high overhead on the CPU and memory of the hub router because all of the spoke to spoke traffic must transit the hub.

One of the most scaleable solutions to the above VPN issues using hub and spoke topology with a large number of spoke to spoke communications required is Cisco preparatory Dynamic Multipoint VPNs (DMVPNs). One of the main requirement of DMVPN implementations is the use of hub and spoke topology, but that dos not mean spoke to spoke traffic must traverse the hub. The hub required for spokes’ address registration using NHRP. NBMA Next Hop Resolution Protocol (NHRP) Defined in RFC 2332 used for spokes address registration in DMVPN implementations. With DMVPN any traffic flowing between routers is sent via a GRE tunnel, but the interesting feature that distinguishes DMVP among other VPN implementations is that this GRE tunnel is a multipoint GRE tunnel. In other words the hub and the spokes will require one tunnel each to achieve a fully meshed DMVPN connectivity.

From the given information above it is obvious that DMVPN can provide the following benefits:

·   Simplify the configuration portion of the hub router by eliminating the need of configuring crypto maps, tunnel interfaces, ACLs for each spoke

·   The spoke routers can obtain their IP addresses dynamically, for example an internet edge router connected to an ADSL link can obtain its IP automatically from the ISP and then the tunnel will register with Hub using NHRP

Design Example:

In this document we will apply some of the DMVP concepts and features to achieve a redundant path to an existing WAN link, as it shown in the bellow diagram the company has a primary WAN link to a service provider using MPLS VPN, HUB1 and HUB2 routers connected to the head office and all other sites use HUB1 as the primary path to reach the head office. Also each site has its own internet link such as ADSL with dynamic IP address assigned to each remote site by the ISP, except the head office routers Hub1 and HUB2 they have to have static IP addresses from the ISP in order to let the remote Sites register with the hub routers using NHRP to form direct spoke to spoke DMVPN communications.

dmvpn.jpg

As illustrated in the above diagram, the routing protocol between the remote sites and the MPLS SP is RIPv2 while the routing between the MPLS SP and the Hub routers is EBGP. With DMVPN, the dynamic routing protocol that will be used to form a redundant dynamic path over mGRE tunnels is OSPF. Bellow is the configuration portion of Hub1 DMVPN :

Note:

The network 172.16.1.0/24 is used in this example to represent the internet IP addressing ( in a real networks implementations this will be a dynamic or static public IPs in the spokes and a public static IP in the Hub side).

Hub1 internet IP address: 172.16.1.1

Hub2 internet IP address: 172.16.1.10

Hub1:

crypto isakmp policy 100

encr aes

hash md5

authentication pre-share

crypto isakmp key key address 0.0.0.0 0.0.0.0 no-xauth

!

!

crypto ipsec transform-set set1 esp-aes esp-md5-hmac

mode transport

!

crypto ipsec profile profile1  -- this will be used to protect mGRE tunnel with IPSec

set transform-set set1

!

!

interface Tunnel0

bandwidth 1000

ip address 150.1.1.1 255.255.255.0      --- DMVPN shared subnet

no ip redirects

ip mtu 1400

ip nhrp authentication key1

ip nhrp map multicast dynamic   

ip nhrp map 150.1.1.10 172.16.1.10     --- Static NHRP to Hub2

ip nhrp map multicast 172.16.1.10

ip nhrp network-id 150

ip nhrp holdtime 360  -- this tells the spoke how long to keep an NHRP reply

ip tcp adjust-mss 1360

ip ospf network broadcast  -- required for direct spoke to spoke communications with ospf

ip ospf priority 200    -- to make this router as the OSPF DR

tunnel source FastEthernet1/1

tunnel mode gre multipoint

tunnel key 1500

tunnel protection ipsec profile profile1   --- IPSec encryption

!

Now let’s see the configuration portion of one of the spokes (site 2):

!

interface Tunnel0

bandwidth 1000

ip address 150.1.1.2 255.255.255.0

no ip redirects

ip mtu 1400

ip nhrp authentication key1

ip nhrp map 150.1.1.1 172.16.1.1     ---Static NHRP mapping to Hub1 and Hub2

ip nhrp map multicast 172.16.1.1

ip nhrp map 150.1.1.10 172.16.1.10

ip nhrp map multicast 172.16.1.10

ip nhrp network-id 150

ip nhrp nhs 150.1.1.1   

ip nhrp nhs 150.1.1.10

ip tcp adjust-mss 1360

ip ospf network broadcast

ip ospf priority 0   --- 0 to make sure this spoke will never become an OSPF DR or BDR

tunnel source FastEthernet1/1

tunnel mode gre multipoint

tunnel key 1500

tunnel protection ipsec profile profile1

!

Site2 verifications:

SITE2#show ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface

192.168.1.1     200   FULL/DR         00:00:38    150.1.1.1       Tunnel0

192.168.1.10     20   FULL/BDR        00:00:38    150.1.1.10      Tunnel0

SITE2#show ip route

R       10.10.1.0 [120/1] via 10.2.1.10, 00:00:17, FastEthernet1/0

R       10.3.1.0 [120/1] via 10.2.1.10, 00:00:17, FastEthernet1/0

O IA 192.168.1.0/24 [110/101] via 150.1.1.10, 00:00:07, Tunnel0

                    [110/101] via 150.1.1.1, 00:00:07, Tunnel0

     150.1.0.0/24 is subnetted, 1 subnets

     30.0.0.0/24 is subnetted, 1 subnets

O       30.1.1.0 [110/101] via 150.1.1.3, 00:00:07, Tunnel0

SITE2#

From Site 2 routing table we can see that the MPLS link is up, however Site2 router prefers the path through DMVPN tunnel over the MPLS SP link. This is normal behavior as OSPF administrative distance AD is less than RIP, the simplest way to solve this issue using the above topology is to make RIP AD lees than OSPF AD ( this solution may not be the best for every topology ) .

router rip

distance 109

SITE2#show ip route

R       10.10.1.0 [109/1] via 10.2.1.10, 00:00:11, FastEthernet1/0

R       10.3.1.0 [109/1] via 10.2.1.10, 00:00:11, FastEthernet1/0

C       10.2.1.0 is directly connected, FastEthernet1/0

R    192.168.1.0/24 [109/1] via 10.2.1.10, 00:00:11, FastEthernet1/0

     150.1.0.0/24 is subnetted, 1 subnets

C       150.1.1.0 is directly connected, Tunnel0

     30.0.0.0/24 is subnetted, 1 subnets

R       30.1.1.0 [109/2] via 10.2.1.10, 00:00:11, FastEthernet1/0

Now all the routes is through the MPLS link using RIPv2.

After the routing issue has been resolved lets do some verification and see if Hub1 link to the MPLS SP goes down what will happen to the spoke routing table:

HUB1#show ip route

     20.0.0.0/24 is subnetted, 1 subnets

O       20.1.1.0 [110/101] via 150.1.1.2, 00:04:45, Tunnel0

     30.0.0.0/24 is subnetted, 1 subnets

O       30.1.1.0 [110/101] via 150.1.1.3, 00:04:45, Tunnel0

As we can see Hub1 now using DMVPN to reach the remote sites because the primary link to the MPLS SP is down.

SITE2#show ip route

    

R      10.10.1.0 [109/1] via 10.2.1.10, 00:00:17, FastEthernet1/0

R       10.3.1.0 [109/1] via 10.2.1.10, 00:00:17, FastEthernet1/0

R    192.168.1.0/24 [109/1] via 10.2.1.10, 00:00:17, FastEthernet1/0

     30.0.0.0/24 is subnetted, 1 subnets

R       30.1.1.0 [109/2] via 10.2.1.10, 00:00:17, FastEthernet1/0

Although Hub1 link to the MPLS SP is down the spokes will continue use the MPLS SP as the primary path, this because Hub2 is advertising the networks, which is the optimal path for this topology. This behavior happened as result to the adjustment we did to the RIP administrative distance above.

Now if Hub1 and Hub 2 links to the MPLS SP go down the only path to reach the HQ network ( 192.168.1.0) by the spokes will be over the internet using DMVPN:

SITE2#show ip route

R       10.10.1.0 [109/1] via 10.2.1.10, 00:00:06, FastEthernet1/0

R       10.3.1.0 [109/1] via 10.2.1.10, 00:00:06, FastEthernet1/0

C       10.2.1.0 is directly connected, FastEthernet1/0

O IA 192.168.1.0/24 [110/101] via 150.1.1.10, 00:00:33, Tunnel0

                    [110/101] via 150.1.1.1, 00:00:33, Tunnel0

     150.1.0.0/24 is subnetted, 1 subnets

C       150.1.1.0 is directly connected, Tunnel0

     30.0.0.0/24 is subnetted, 1 subnets

R       30.1.1.0 [109/2] via 10.2.1.10, 00:00:06, FastEthernet1/0   -- site 3 LAN is reachable via the MPLS SP link

SITE2#

From the above routing table of site 2 we can see that site 2 has both hub1 and hub 2 as next hops to reach the HQ network while the design requirements require hub1 to be the primary, this issue can be fixed by using several ways, the simplest way is by tuning the dynamic routing protocol used in the topology. This document will demonstrate the ability of achieving the same goal by using OSPF and EIGRP as the routing protocol over DMVPN tunnels. With OSPF in this example ospf cost will be used to control the path selection, by increasing the ospf cost on the LAN interface of Hub2 to make the path through Hub1 more preferred.

SITE2#show ip route

O IA 192.168.1.0/24 [111/101] via 150.1.1.1, 00:00:50, Tunnel0  -- Hub1 is the primary next hop to reach the HQ network over mGRE tunnel ( less ospf cost).

In the case of Site 2 link to the MPLS SP goes down the remote networks in the routing table will be shown as bellow:

SITE2#show ip route

O IA 192.168.1.0/24 [111/101] via 150.1.1.1, 00:00:50, Tunnel0

 

O       30.1.1.0 [111/101] via 150.1.1.3, 00:00:30, Tunnel0

SITE2#traceroute 30.1.1.1

Type escape sequence to abort.

Tracing the route to 30.1.1.1

  1 150.1.1.3 284 msec 272 msec *  -- direct spoke to spoke

EIGRP as the Routing Protocol over DMVPN Tunnels:

If EIGR used as the routing protocol over DMVPN mGRE Tunnels using the above topology, the first issue will be faced by Site 2 is that HQ network will be seen with two equal cost paths then Site2 will loadbalnce the trafficto that network. While the required behavior is the primary path to be through Hub1. Earlier in this document we have seen how this issue fixed with OSPF by changing the OSPF Cost of an interface, with EIGRP we will use administrative distance to prefer one eigrp neighbor over others:

Router eigrp 10     --- this configurations required in every spoke

Distance 80 150.1.1.1 0.0.0.0

EGRP internal route AD is 90, the above configuration changed the AD of all routes coming from hub1 150.1.1.1 to be 80 which will make it the preferred neighbor ( please note that if there is any external EIGRP route you may need to consider another approach such as delay, as with EIGRP you can not change administrative distance of external eigrp routes from a specific peer only you can change it globally from all peers).

SITE2#show ip route 192.168.1.0

Routing entry for 192.168.1.0/24

  Known via "eigrp 10", distance 80, metric 15362560, type internal

  Redistributing via eigrp 10

  Last update from 150.1.1.1 on Tunnel0, 00:01:36 ago

  Routing Descriptor Blocks:

  * 150.1.1.1, from 150.1.1.1, 00:01:36 ago, via Tunnel0

      Route metric is 15362560, traffic share count is 1

      Total delay is 500100 microseconds, minimum bandwidth is 1000 Kbit

      Reliability 255/255, minimum MTU 1400 bytes

      Loading 1/255, Hops 1

First issue fixed, let’s see what other facts need to be consider when we use EIGRP.

With EIGRP and RIP implementations over DMVPN tunnels we have to disable spilt horizon in the hub tunnel interface (same concept used with NBMA networks such as hub and spoke frame-relay topology)

SITE2#traceroute 30.1.1.1

Type escape sequence to abort.

Tracing the route to 30.1.1.1

  1 150.1.1.1 136 msec 304 msec 296 msec

  2 150.1.1.3 316 msec 268 msec *

also from the above traceroute from Site 2 to Site 3 LAN network, the traffic is traversing the hub, which means the communications between the spokes not direct !!

with ospf we used a network type of broadcast over the tunnels which resulted in direct communications between spokes after each spoke register with the hub using NHRP.

With eigrp we need to add an additional command under the hub tunnel interface to enable the direct spoke to spoke communications ( this behavior might be useful for some implementations that requires the spoke to spoke traffic to go through the hub router first for example to do traffic inspection )

Hub1:

interface Tunnel0

no ip next-hop-self eigrp 10

no ip split-horizon eigrp 10

SITE2#show ip route

D       30.1.1.0 [80/15490560] via 150.1.1.3, 00:00:16, Tunnel0

Now Site 2 see Site 3 LAN network with Site 3 tunnel interface IP

SITE2#traceroute 30.1.1.1

Type escape sequence to abort.

Tracing the route to 30.1.1.1

  1 150.1.1.1 140 msec

    150.1.1.3 276 msec 204 msec

SITE2#traceroute 30.1.1.1

Type escape sequence to abort.

Tracing the route to 30.1.1.1

  1 150.1.1.3 140 msec 192 msec *

SITE2#

The first initial traffic will go through the Hub then all the subsequent traffic will be direct spoke to spoke.

To Sum up, DMVPN is a very flexible, dynamic and secure feature that can suit various type of  implementations, also we can control our traffic and path selection by using simple dynamic routing protocol adjustments

.

Thank you

Marwan Alshawi

Version history
Revision #:
1 of 1
Last update:
‎01-05-2010 10:04 PM
Updated by:
 
Labels (1)
Comments
New Member

Hi Marwan,

This is an excellent doc and very useful when designing DMVPN as a backup solution and figuring out the routing metrics. I have a scenario that I wanted to discuss with you, if you would be kind enough to share your email address?

regards

Zahir Ahmed

Thanks I will review it carefully

Regards

Michael

New Member

Michael,

     We are looking into doing the similiar design, i would like to discuss with you on our design.  Wonder you would email me you contact?

Thx Neils

New Member

cool. learning new stuff everyday

New Member

Hi Marwan , 

  Perfect and excellent Document Sir 

I have question here regarding to the first Sinario  with 2 HUB  using OSPF .

It is any way to make the spokes prefer Hub 2 rather Hub 1 . But without using DR and BDR . It is possible that.

thank you , 

New Member

Awesome!

quick question:

 in your example there's 2 Hubs!,  Can we make more than 2 Hubs?

Thanks,

Ziyad

New Member

Hi Ziyad,

Yes you can have more than two hubs. To do this you will need to add additional statements on the spokes for the additional Hubs. This is common if you have for example two DC's and you wanted to have 2 Hub routers per DC.

So using Marwans example above, you would add the additional mapping entries on the spokes

ip nhrp map 150.1.1.1 172.16.1.1     ---Static NHRP mapping to Hub1 Hubs2, Hub3 and Hub4

ip nhrp map multicast 172.16.1.1

ip nhrp map 150.1.1.10 172.16.1.10

ip nhrp map multicast 172.16.1.10

Ip nhrp map 150 .x.x.x 172.16.1.x  Hub3

ip nhrp map multicast 172.16.1.10

Ip nhrp map 150 .x.x.x 172.16.1.x  Hub 4

ip nhrp map multicast 172.16.1.10

and then of course you would need to configure the additional 2 hub routers..

Hope this helps

Zahir

New Member

Thank you Zahir great explanation.

But it's matter if I want to use three Hubs EIGRP and one BGP?

New Member

Hello Marwan sir,

                             I Have One question. How DR and DC is communicate Virtually Which One Protocol I Use for load balancing.   I am Share Diagram.

New Member

Hello Marwan sir,

                             I Have One question. How DR and DC is communicate Virtually Which One Protocol I Use for load balancing.   I am Share Diagram.

New Member

HI, 

My case different, I have 04 Routers Concentrator  DMVPN using config below interface tunnel:

ROUTER A

interface Tunnel1566
ip vrf forwarding vpn-A
ip address 10.150.10.1 255.255.255.0
no ip redirects
no ip proxy-arp
ip mtu 1400
ip nhrp map multicast dynamic
ip nhrp network-id 106006
ip nhrp holdtime 300
ip nhrp registration no-unique
ip nhrp registration timeout 10
ip tcp adjust-mss 1360
load-interval 30
delay 1000
qos pre-classify
keepalive 5 3
tunnel source Loopback22
tunnel mode gre multipoint
tunnel key 106006
tunnel protection ipsec profile ipsec
end

ROUTER B

interface Tunnel1567
ip vrf forwarding vpn-B
ip address 10.150.10.2 255.255.255.0
no ip redirects
no ip proxy-arp
ip mtu 1400
ip nhrp map multicast dynamic
ip nhrp network-id 106006
ip nhrp holdtime 300
ip nhrp registration no-unique
ip nhrp registration timeout 10
ip tcp adjust-mss 1360
load-interval 30
delay 1000
qos pre-classify
keepalive 5 3
tunnel source Loopback22
tunnel mode gre multipoint
tunnel key 106006
tunnel protection ipsec profile ipsec
end

ROUTER C


interface Tunnel1568
ip vrf forwarding vpn-C
ip address 10.150.30.1 255.255.255.0
no ip redirects
no ip proxy-arp
ip mtu 1400
ip nhrp map multicast dynamic
ip nhrp network-id 106008
ip nhrp holdtime 300
ip nhrp registration no-unique
ip nhrp registration timeout 10
ip tcp adjust-mss 1360
load-interval 30
delay 1000
qos pre-classify
keepalive 5 3
tunnel source Loopback22
tunnel mode gre multipoint
tunnel key 106008
tunnel protection ipsec profile ipsec
end

ROUTER D


interface Tunnel1569
ip vrf forwarding vpn-D
ip address 10.150.30.2 255.255.255.0
no ip redirects
no ip proxy-arp
ip mtu 1400
ip nhrp map multicast dynamic
ip nhrp network-id 106008
ip nhrp holdtime 300
ip nhrp registration no-unique
ip nhrp registration timeout 10
ip tcp adjust-mss 1360
load-interval 30
delay 1000
qos pre-classify
keepalive 5 3
tunnel source Loopback22
tunnel mode gre multipoint
tunnel key 106008
tunnel protection ipsec profile ipsec
end

the CPE In my  Laboratory I using 01 branch 2921, two (02) tunnel DMVPN, another tunnel have two dress NHRP. show configuration below :

interface Tunnel10
ip address 10.150.10.10 255.255.255.0
no ip redirects
ip accounting output-packets
ip mtu 1400
ip flow ingress
ip flow egress
ip nhrp map 10.150.10.1 ( MY IP VALID ROUTER DMVPN)
ip nhrp map multicast ( MY IP VALID ROUTER DMVPN)
ip nhrp map 10.150.10.2 ( MY IP VALID ROUTER DMVPN)
ip nhrp map multicast ( MY IP VALID ROUTER DMVPN)
ip nhrp network-id 106006
ip nhrp holdtime 300
ip nhrp nhs 10.150.10.1
ip nhrp nhs 10.150.10.2
ip nhrp registration no-unique
ip nhrp registration timeout 10
ip tcp adjust-mss 1360
load-interval 30
delay 1000
qos pre-classify
keepalive 5 3
tunnel source Cellular0
tunnel mode gre multipoint
tunnel key 106006
tunnel path-mtu-discovery
tunnel protection ipsec profile ipsec
end

interface Tunnel30
ip address 10.150.30.10 255.255.255.0
no ip redirects
ip accounting output-packets
ip mtu 1400
ip flow ingress
ip flow egress
ip nhrp map 10.150.30.2 ( MY IP VALID ROUTER DMVPN)
ip nhrp map multicast ( MY IP VALID ROUTER DMVPN)
ip nhrp map 10.150.30.1 ( MY IP VALID ROUTER DMVPN)
ip nhrp map multicast ( MY IP VALID ROUTER DMVPN)
ip nhrp network-id 106008
ip nhrp holdtime 300
ip nhrp nhs 10.150.30.1
ip nhrp nhs 10.150.30.2
ip nhrp registration no-unique
ip nhrp registration timeout 10
ip tcp adjust-mss 1360
load-interval 30
delay 1000
qos pre-classify
keepalive 5 3
tunnel source GigabitEthernet0
tunnel mode gre multipoint
tunnel key 106008
tunnel path-mtu-discovery
tunnel protection ipsec profile ipsec
end

My question is , whats occurred the not close the DMVPN 

sh dmvpn detail    

  
1 MY IP VALID ROUTER DMVPN 10.150.10.1 IKE 00:50:10 S 10.150.10.1/32
1 MY IP VALID ROUTER DMVPN 10.150.10.2 UP 00:51:51 S 10.150.10.2/32

1 MY IP VALID ROUTER DMVPN 10.150.30.1 IKE 00:50:10 S 10.150.30.1/32
1 MY IP VALID ROUTER DMVPN 10.150.30.2 UP 00:51:51 S 10.150.30.2/32

Tanks,

Rodolfo Navero

PS: Sorry my English

New Member

Hello

we are planning to do same but with little different scenario, let me explain whats the requirement

Currently customer has HO and 4 branch offices. primary link is point to point wireless links and secondary is DMVPN with eigrp. it got tracked via ip sla

now customer is planning DR site, bcz of cost point to point wireless link cannot be installed, so it should be some other way around

in this scenario, shall we consider second HUB as mentioned above and with AD, we can control, or something else need to be taken care

New Member

what do i do if i want use 1 hub active 1 hub backup . for me 1 solution best practice. thank

New Member

Hi - Use your IGP metrics to weight the routes so that all traffic goes to the primary hub and then to the backup hub should the primary fail. I have done this with OSPF and it works like a dream..

New Member

i used parameter change cost but i don't success. do u have solution specific???