cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6970
Views
10
Helpful
14
Replies

DMVPN and INTERNET VIA HUB LOCATION ISSUES

lap
Level 2
Level 2

Hello everyboby,

I really wish that you can help me with the issue I have.

I explain. I have to test a Dual Hub - Dual DMVPN Layout for a customer before we configure it in real production.
The customer has some sites where routers are behind some ISP routers which are doing NAT.

Drawing-Lab-Setup.jpeg

How things are configured:

- All the traffic from spokes has to go via the Hub location so no local internet traffic on spokes.
- Hub 1 and Hub 2 sends a default route to spokes via EIGRP. But only Hub 1 is used.
- Hub 1 is the primary router for DMVPN. In case of hardware/Connection to Internet failure Hub 2 become active for DMVPN and Internet.
- Hub 1 and Hub 2 are both connected to one ISP and are Internet Gateway for spokes.
- Hub 1 and Hub 2 are configured with IOS Firewall.
- On spokes I have used VRF to seperate DMVPN routning table from Global routning table so I could receive a default route from Hub 1 and Hub 2 to route traffic from spokes to Internet via Hub location

What is working:

- All spokes can have access to local LAN at Hub location.
- All spokes can do spoke to spoke
- Failover working for DMVPN
- Spokes NOT behind NAT ISP router (that is to say having the public IP address directly attached at their outside interface) can go to Internet via hub location and all packets are inspected correctly by the IOS firewall and Nat correctly
 
What is not working:

- Spokes behind NAT ISP router cannot reach the Internet through Hub location. They can only reach local LAN at Hub location and do spoke to spoke.
  On hub router the IOS firewall sees the packets comming from theses spokes (behind NAT) with a source IP address which is the public IP address og the ISP router outside interface. Not the LAN private IP address behind spoke.
  Moreover packets are never natted. If I do some snifing on an Internet server the source private IP address is the LAN IP address of the LAN behind the spoke. That means that the Hub router never nat these packets.

 
How to solve this problem?

Well I don't know that is why I need your help/advices :-)

I don't know if I should configure a VRF on the hub location also as maybe things gets mess up.

The problem seems to be coming from NAT-T as the spokes which aren't behind NAT can find go on the Internet through Hub and both Cisco IOS inspection and NAT are working find.

As I was testing today with the customer at the begining the spoke behind nat could ping different server on the Internet but not open a HTTP session. DNS was working find. The IOS Firewall was actually

inspecting packets with the real private IP addresse. Then I thought that it was a MTU issue so I decided to ping out the Internet with bigger MTU size and suddenly the pings were not going through anymore.

I could see on the Hub1 router that the IOS firewall was inspecting the public IP address again of the ISP NAT router at spoke side and not anymore the real private IP address. Really strange!

Attached files:

I attach the following files: a drawing of the setup called Drawing-Lab-Setup.jpeg | All the configs files for HUB1, BRANCH1, BRANCH2 and ISP-ROUTER named respectively: HUB1.txt, BRANCH1.txt, BRANCH2.txt and ISP-ROUTER.txt

Logs from Hub1 When pinging host 200.200.200.200 on the Internet from Branch2 (behind NAT ISP router):

BRANCH2#ping vrf DMVPN-VRF 200.200.200.200 source vlan 100

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.200.200.200, timeout is 2 seconds:
Packet sent with a source address of 192.168.110.1
.....
Success rate is 0 percent (0/5)

*Jul 15 06:04:51.017 UTC: %FW-6-SESS_AUDIT_TRAIL_START: Start icmp session: initiator (110.10.10.2:8) -- responder (200.200.200.200:0)

So the IOS firewall doesn't inspect the real source private IP adresse which should be in this case: 192.168.110.2. It sess on the public IP address.

HUB1#sh ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 80.10.10.2:1      80.10.10.2:1       100.10.10.2:1      100.10.10.2:1
icmp 80.10.10.2:2      80.10.10.2:2       110.10.10.2:2      110.10.10.2:2
udp 80.10.10.2:4500    80.10.10.2:4500    110.10.10.2:4500   110.10.10.2:4500

There is no NAT entries present for thes packets

Snifing on Tunnel 1 interface on Hub1 (packets comming in):

7    7.355997    192.168.110.1    200.200.200.200    ICMP    Echo (ping) request
So although the IOS firewall inspect on the public IP 110.10.10.2:8 the sniffing capture says that the packet are comming from the real private IP address

Sniffing on the server (200.200.200.200) with wireshark:

114    14.123552    192.168.110.1    200.200.200.200    ICMP    Echo (ping) request

So the source private IP address from BRANCH2 LAN is never natted by HUB1

So the server sees the source private IP address not natted although Hub1 IOS firewall inspect the public IP address 110.10.10.2:8


Logs from Hub1 When pinging host 200.200.200.200 on the Internet from Branch1 (Not behind NAT ISP router):

BRANCH1#ping vrf DMVPN-VRF 200.200.200.200 source vlan 100

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.200.200.200, timeout is 2 seconds:
Packet sent with a source address of 192.168.100.1
!!!!!

*Jul 15 06:05:18.217 UTC: %FW-6-SESS_AUDIT_TRAIL_START: Start icmp session: initiator (192.168.100.1:8) -- responder (200.200.200.200:0)

So here the firewall sees the real private IP address which is 192.168.100.1

HUB1#sh ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 80.10.10.2:1      80.10.10.2:1       100.10.10.2:1      100.10.10.2:1
icmp 80.10.10.2:2      80.10.10.2:2       110.10.10.2:2      110.10.10.2:2
udp 80.10.10.2:4500    80.10.10.2:4500    110.10.10.2:4500   110.10.10.2:4500
icmp 80.10.10.2:22     192.168.100.1:22   200.200.200.200:22 200.200.200.200:22

The source real private IP address is also find natted to Hub 1 outside public IP address

Snifing on Tunnel 1 interface on Hub1 (packets comming in):

8    7.379997    192.168.100.1    200.200.200.200    ICMP    Echo (ping) request

Same Real private IP address as inspected by IOS firewall so everything is find there.

Sniffing on the server (200.200.200.200) with wireshark:

67                  10.441153     80.10.10.2     200.200.200.200                 ICMP            Echo (ping) request

So here everything is fine. The address is natted correctly.

__________________________________________________________________________________________

Best Regards,

Laurent

3 Accepted Solutions

Accepted Solutions

Hi,

Just saw your post, hopefully it is not too late.

I am not sure what your exact problem is, but I think we can work through it to figure it out.

One thing I did notice was that your NAT ACL is too general. You need to make it more

specific.  In particular you want to make sure that it doesn't match the VPN traffic coming

in to / going out of the router.

For example you shouldn't really have any of these entries in your NAT translation table.

HUB1#sh ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 80.10.10.2:1      80.10.10.2:1       100.10.10.2:1      100.10.10.2:1
icmp 80.10.10.2:2      80.10.10.2:2       110.10.10.2:2      110.10.10.2:2
udp 80.10.10.2:4500    80.10.10.2:4500    110.10.10.2:4500   110.10.10.2:4500

Instead of using:

ip access-list extended Nat
deny   ip any 192.168.0.0 0.0.255.255 log
permit ip any any
deny   ip any any log

Can you use:

ip access-list extended Nat
  deny   ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255 log
  permit ip 192.168.0.0 0.0.255.255 any
  deny   ip any any log

Also I would be very careful with using the 'log' keyword in a NAT ACL.

I have seen it cause problems.

What IOS versions are you using?

Try making the changes to NAT so that you no longer see NAT translation entries

for the NAT-T (UDP 4500) packets in the NAT translation table on the hub. It may be

that this is setting a flag on the packet structure, that IOS-Firewall and then NAT is

picking up on and then doing the wrong thing in this case.

If this doesn't work then let me know.

This may be something for which you will have to open a TAC case so that we can

debug this directly on your setup.

Mike.

View solution in original post

Okay your issue is that the spoke's IOS version doesn't support spoke-spoke tunnels

when one or both of the spokes are behind NAT.  You have to run IOS 12.4(6)T or later

on all nodes in your DMVPN network to support this.  I suggest that you run 12.4(15)T12

or later.

Also the spokes running 12.4M (mainline) can only do DMVPN phase 2, but from the

debugs it looks like your hubs are configured for DMVPN Phase 3 with the

'ip nhrp redirect' command, which isn't used in DMVPN Phase 2.

After you upgrade your spokes, you can continue to run Phase 2, just remove the

'ip nhrp redirect' command from the hub.

If you want to convert to Phase 3 then you need to:

  1. Leave the 'ip nhrp redirect' on the hub.

  2. Remove 'no ip next-hop-self eigrp 1' from the hub.

  3. Add 'ip nhrp shortcut' on the spoke.

Mike.

View solution in original post

The problem is that your tunnels are coming up in IPsec tunnel-mode.

With DMVPN when a node is behind NAT you must run in IPsec transport-mode.

Otherwise NHRP can't detect that there is NAT in the middle (IPsec still can).

This will cause NHRP to try to setup the wrong crypto socket. I checked the

configurations from your first message and they do say 'transport mode', but

perhaps that has changed since then.

Please check your configurations and make sure that they say your transform-set

is 'transport-mode'.  Note, IPsec does have the capability to automatically fallback

to 'tunnel-mode' even though your configured 'transport-mode'. You can force it to

only accept 'transport-mode' by configuring on the hub.

     crypto ipsec transform-set ....

       transport-mode required

Try this and see if it helps.

Mike.

View solution in original post

14 Replies 14

lap
Level 2
Level 2

Nobody to help me?

Regards.

Hi,

Just saw your post, hopefully it is not too late.

I am not sure what your exact problem is, but I think we can work through it to figure it out.

One thing I did notice was that your NAT ACL is too general. You need to make it more

specific.  In particular you want to make sure that it doesn't match the VPN traffic coming

in to / going out of the router.

For example you shouldn't really have any of these entries in your NAT translation table.

HUB1#sh ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 80.10.10.2:1      80.10.10.2:1       100.10.10.2:1      100.10.10.2:1
icmp 80.10.10.2:2      80.10.10.2:2       110.10.10.2:2      110.10.10.2:2
udp 80.10.10.2:4500    80.10.10.2:4500    110.10.10.2:4500   110.10.10.2:4500

Instead of using:

ip access-list extended Nat
deny   ip any 192.168.0.0 0.0.255.255 log
permit ip any any
deny   ip any any log

Can you use:

ip access-list extended Nat
  deny   ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255 log
  permit ip 192.168.0.0 0.0.255.255 any
  deny   ip any any log

Also I would be very careful with using the 'log' keyword in a NAT ACL.

I have seen it cause problems.

What IOS versions are you using?

Try making the changes to NAT so that you no longer see NAT translation entries

for the NAT-T (UDP 4500) packets in the NAT translation table on the hub. It may be

that this is setting a flag on the packet structure, that IOS-Firewall and then NAT is

picking up on and then doing the wrong thing in this case.

If this doesn't work then let me know.

This may be something for which you will have to open a TAC case so that we can

debug this directly on your setup.

Mike.

Hello Mike,

First of all thank you very much for taking time to look at my post.

I have tried your solution and it is working perfectly! I am really happy that you could help me. I would like to thanks you many times.

I have deleted the 'log' keyword in my NAT ACL and configured as you said and than I could find ping from BRANCH2 to 200.200.200.200.

Futhermore I no longer see NAT translation entries for the NAT-T (UDP 4500)  packets in the NAT translation table on the hub.

I have though a last small issue. As you saw in my configurations on BRANCH1 and BRANC2 I have used VRF (IVRF). I read on the Internet that it was a good idea to use FVRF ( Front-door VRF) instead   because if the spoke routers are managed over the tunnel (as it is the case in my situation), all management  functionality (SNMP, SSH, etc.) must be VRF-aware.

My issue is for the spoke to spoke traffic as the IPsec cannot be buildt when I use FVRF. I get the following mesage:

BRANCH1#ping 192.168.110.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.110.1, timeout is 2 seconds:
!.
*Aug 19 10:16:14.500 UTC: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Quick mode failed with peer at 110.10.10.2    ...
Success rate is 20 percent (1/5), round-trip min/avg/max = 160/160/160 ms

The problem is maybe because BRANCH2 is doing NAT-T ? Do you any ideas.

I attached the new configs for BRANCH2 and BRANCH1 configured this time wiht FVRF. The others configs remain the same.

Regards,

Laurent

I don't see anything obvioulsy wrong with your two branch configurations using VRFs.

WIll need to see the output from:

    show ip nhrp

    show crypto socket

    show crypto isakmp sa

    show crypto ipsec sa

    debug nhrp

    debug nhrp packet

    debug nhrp extention

    debug crypto engine

    debug crypto isakmp

    debug crypto ipsec

On both of the branch routers to see what NHRP and IPsec

are doing when we are trying to bring up the spoke-spoke

tunnel.  NAT-T should detect NAT between Branch1 and

Branch2 and do the right thing, even with fVRF. 

Note, that Branch2 must be NATted to the same outside

NAT IP address when going to the Hub and Branch1.

Mike.

Hi Mike,

Thank you very much for a fast reply.

BRANCH1 IOS and Router model:
-------------------------------
Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 12.4(25c)
Cisco 2811

BRANCH2 IOS and Router model:
------------------------------
Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 12.4(18)
Cisco 2811


**********************************************************************************************************************

Show commands Before to ping BRANCH2 (IP: 192.168.110.1) from BRANCH1 (Source IP: 192.168.100.1):
--------------------------------------------------------------------------------------------------

See attached file: BRANCH1-LOG-BeforePingFromBranch1ToBranch2.txt
See attached file: BRANCH2-LOG-BeforePingFromBranch1ToBranch2.txt

Debug commands after pinging BRANCH2 (IP: 192.168.110.1) from BRANCH1 (Source IP: 192.168.100.1):
-----------------------------------------------------------------------------------------------------------------------------------------------------------

See attached file: BRANCH1-LOG-DebugPingFromBranch1ToBranch2.txt
See attached file: BRANCH2-LOG-DebugPingFromBranch1ToBranch2.txt

Show commands after pinging BRANCH2 (IP: 192.168.110.1) from BRANCH1 (Source IP: 192.168.100.1):
-----------------------------------------------------------------------------------------------------------------------------------------------------------

See attached file: BRANCH1-BRANCH2-LOG-AfterPingFromBranch1ToBranch2.txt

Let me know if you need something else;-)


Best Regards,

Laurent

Okay your issue is that the spoke's IOS version doesn't support spoke-spoke tunnels

when one or both of the spokes are behind NAT.  You have to run IOS 12.4(6)T or later

on all nodes in your DMVPN network to support this.  I suggest that you run 12.4(15)T12

or later.

Also the spokes running 12.4M (mainline) can only do DMVPN phase 2, but from the

debugs it looks like your hubs are configured for DMVPN Phase 3 with the

'ip nhrp redirect' command, which isn't used in DMVPN Phase 2.

After you upgrade your spokes, you can continue to run Phase 2, just remove the

'ip nhrp redirect' command from the hub.

If you want to convert to Phase 3 then you need to:

  1. Leave the 'ip nhrp redirect' on the hub.

  2. Remove 'no ip next-hop-self eigrp 1' from the hub.

  3. Add 'ip nhrp shortcut' on the spoke.

Mike.

Hi Mike,

Sorry to tell you that but you are amazing! Your answers are really clear and have helped a lot.

I will follow your advice and upgrade to the right IOS version. I will let you know when is done.

Regarding DMVPN phase 3 and phase 2 what would be the advantage for our customer to migrate to phase 3?

I don't really understand what is the big advantage to migrate to phase 3.

Best Regards,

Laurent

Hi Mike,

So I have upgraded the spokes BRANCH1 and BRANCH2 to this IOS version: 12.4(24)T3. And now DMVPN phase 2 is working perfectly with spoke to spoke traffic and FVRF configured on both spokes. That is really great!

As I am very curious and like to try out new thinks out I have tried to configure DMVPN phase 3 and follow your advices regarding the configuration.


I wanted to try DMVPN phase 3 as I read that is possible to have only a default route on the spokes and still can do spoke-to-spoke. I think that would be great for our customer so we don't fill the routing tabe up at spokes with unnecessary routes and we can then save on memory.

I did the following to configure DMVPN phase 3:

On HUB1:

interface Tunnel1

     ip nhrp redirect

     ip next-hop-self eigrp 1

     ip summary-address eigrp 1 0.0.0.0 0.0.0.0  //To send a default route to spokes cause until now I was sending all.

On BRANCH1 and BRANCH2:

interface range tunnel 1 -2

     ip nhrp shortcut

Once I did that I try to ping from BRANCH1 to BRANCH2 and sure enough it is working:

BRANCH1#ping 192.168.110.2 source 192.168.100.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.110.2, timeout is 2 seconds:
Packet sent with a source address of 192.168.100.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 156/159/160 ms

But the problem is that the traffic is always going through HUB1 as you can see:

BRANCH1#traceroute 192.168.110.2 source 192.168.100.1

Type escape sequence to abort.
Tracing the route to 192.168.110.2

  1 10.1.0.1 156 msec 156 msec 156 msec
  2 10.1.0.3 156 msec 160 msec 160 msec
  3 192.168.110.2 156 msec 160 msec 156 msec

I attach therefore the debug/show commands you asked for last time applied to this situation this time.

**************************************************************************************************************************************

**************************************************************************************************************************************

Show commands Before to ping BRANCH2 (IP: 192.168.110.2) from BRANCH1  (Source IP: 192.168.100.1):

See attached file: BRANCH1-BRANCH2-LOG-DMPVN3-BeforePingFromBranch1ToBranch2.txt

Debug commands after pinging BRANCH2 (IP: 192.168.110.2) from BRANCH1  (Source IP: 192.168.100.1):

See  attached file: BRANCH1-LOG-DMVPN3-DebugPingFromBranch1ToBranch2.txt
See  attached file: BRANCH2-LOG-DMVPN3-DebugPingFromBranch1ToBranch2.txt

Show commands after pinging BRANCH2 (IP: 192.168.110.2) from BRANCH1  (Source IP: 192.168.100.1):

See attached file: BRANCH1-BRANCH2-LOG-DMVPN3-AfterPingFromBranch1ToBranch2.txt

*************************************************************************************************************************************

*************************************************************************************************************************************

I also attached the tunnel configuration of HUB1, BRANCH1 and BRANCH2: BRANCH1-BRANCH2-HUB1-TunnelsConfig-DMVPN3.txt

The rest of the configs hasn't changed since I have configured FVRF.

Again thank you so much for your help. I will award you with the max points possible;-)

Best regards,

Laurent

There is a problem with IPsec Phase 2 coming up.  It looks like we are not gettting the

NAT information in the NHRP resolution request packets, but I can't tell because I had

an error in the debugs that I gave you

   debug nhrp extention ---- should be ---->>>  debug nhrp extension

Please re-run your test with the above change in the debugs.

Also on the hub can you do:

   show ip nhrp

   show ip route

Note, send advertising default 0/0 through the tunnels is usually dangerous,

but since you are using VRFs on the spokes, as you noted, you are okay.

Also you asked why under 'show crypto ipsec sa' you were seeing it list your

VRF under both iVRF and fVRF.  Looking at the different points where an

encrypted packet is processed we can see what is happening.

   (WAN interface) --- (A) --- (IPsec) --- (B) --- (GRE tunnel interface) --- (C) ---

When an encrypted packet arrives it will get the VRF of the inbound interface.

So at point (A)  this is "DMVPN-VRF".  IPsec will decrypt the packet and place

the decrypted packet back into the same VRF (or global) unless otherwise

instructed. So in your case the decypted GRE packet at point (B) is again in

VRF "DMVPN-VRF".  GRE will take the decrypted GRE packet and decapsulate

it to produce the host IP packet. The host IP packet will be placed in the VRF

(or global) as configured on the inbound interface in this case the tunnel interface.

From IPsec's point of view it looks at points (A) and (B) which are both in VRF

"DVMPN-VRF" so IPsec will say that fVRF = iVRF = "DMVPN-VRF".

From GRE's point of view it looks at point (B) and (C) which are different VRFs

(or global) so GRE will say that fVRF = "DMVPN-VRF" and iVRF = (global).

In order to keep things simple, when doing VRFs with DMVPN from IPsec's

point of view the fVRF (A) and iVRF (B) must be the same, a VRF or global.

From GRE's point of view, the fVRF (B) (tunnel vrf ...) and the iVRF (C)

(ip vrf-forwarding ...) can be the same or different.  So that way we really

control changing (or not) between VRFs on the tunnel interface, rather than

in IPsec.

Thanks,

Mike.

Hi Mike,

I will like first to thank you very much for you really expert explanation regarding my question. I couldn´t imagine that someone could explain me that.

Secondly I have run the test a second time with debug nhrp extension and I have also included HUB1 in the debug + the 2 commands you asked me for regarding HUB1.

I attach therefore the debug/show commands to this post.

Show commands Before to ping  BRANCH2 (IP: 192.168.110.2) from BRANCH1  (Source IP: 192.168.100.1):

See attached file:HUB1-BRANCH1-BRANCH2-LOG-DMVPN3-BeforePingFromBranch1ToBranch2.txt

Debug commands after pinging  BRANCH2 (IP: 192.168.110.2) from BRANCH1  (Source IP: 192.168.100.1):

See attached files:HUB1-LOG-DMVPN3-DebugPingFromBranch1ToBranch2.txt

See attached files:BRANCH1-LOG-DMVPN3-DebugPingFromBranch1ToBranch2.txt

See attached files:BRANCH2-LOG-DMVPN3-DebugPingFromBranch1ToBranch2.txt

Show commands after pinging  BRANCH2 (IP: 192.168.110.2) from BRANCH1  (Source IP: 192.168.100.1):

See attached files:BRANCH1-BRANCH2-LOG-DMVPN3-AfterPingFromBranch1ToBranch2.txt

I am looking forward to hearing from you ;-)

Best regards,

Laurent

The problem is that your tunnels are coming up in IPsec tunnel-mode.

With DMVPN when a node is behind NAT you must run in IPsec transport-mode.

Otherwise NHRP can't detect that there is NAT in the middle (IPsec still can).

This will cause NHRP to try to setup the wrong crypto socket. I checked the

configurations from your first message and they do say 'transport mode', but

perhaps that has changed since then.

Please check your configurations and make sure that they say your transform-set

is 'transport-mode'.  Note, IPsec does have the capability to automatically fallback

to 'tunnel-mode' even though your configured 'transport-mode'. You can force it to

only accept 'transport-mode' by configuring on the hub.

     crypto ipsec transform-set ....

       transport-mode required

Try this and see if it helps.

Mike.

Hi Mike,

So again that was the right answer. I have configured  "transport-mode required" on all the nodes that is to say: HUB1,BRANCH1,BRANCH2 as it was not enough when configured only on HUB1.

I would like to thank you very much for your time and your BIG help with the issues I had.

Now The DMVPN is working perfectly with phase 3 and I will have a very happy customer.

Best regards,

Laurent

Hi Mike,

I am looking to deploy DMVPN for a customer with 1400 locations with future plans of having 4000 remote locations.  Is there an updated DMVPN Design Guide available?  I have only been able to find version 1.1, which was written in 2008.  In specific, I am looking for information on ISR G2 router recommendations and scalability.  Also looking for information on Single Cloud versus Dual Cloud.

Thanks Mike,

Tim

hello:

does this working example force all inet bought traffic from spoke to the hub? ie, i do not want spoke to access the inet directly, only through the hub

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: