converting from MPLS to point to point via site-to-site tunnels

Unanswered Question
Dec 16th, 2013
User Badges:

We are in the process of converting our remote offices from MPLS to Site-to-Site via ASA Tunnels.

I have no control over the routing of the MPLS, and it takes 30 days to get it turned off.

This site in question is circled in Blue.


my current configuration is like this

NJ1.JPG



My plan for the site is to remove it from the MPLS and move it to a tunnel like this:

NJ2.JPG

I can build the tunnel successfully between both ASA Devices, but I can't reroute traffic because the MPLS Router in my site thinks it knows the way to get to the 50 site.  I've told the core switch to route the 50 traffic via the ASA, but it goes through the router first, and I guess the router ignores my route.

ip default-gateway 172.16.100.2 (This is the ASA Address)

ip classless

ip route 0.0.0.0 0.0.0.0 172.16.100.2

ip route 172.16.100.0 255.255.255.0 192.168.3.254

ip route 192.168.50.0 255.255.255.0 172.16.100.2


My question is this.

Can I add a true internal address to an interface on my ASA, Attach it to the Core switch, and route the 50 traffic through that?

Like this?

NJ3.JPG



Eventually the MPLS Network is going to go away anyway, so I'm thinking eventually this will need to happen anyway, or I'll need to add another router to make a new DMZ.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (1 ratings)
Loading.
Jon Marshall Mon, 12/16/2013 - 12:52
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


So you have no control over the AT&T router ie. you cannot configure it ?


If so then yes you could run a connection from the inside interface of the ASA to your switch and bypass the router. The rouetr is probably not giving you any security, just make sure the firewall is tied down because anything allowed through hits your core switch.


By the way, if the core switch is routing for vlans and has "ip routing" enabled then you should remove -


"ip default-gateway 172.16.100.2"


as this is only needed for a L2 switch that is not routing. Your default route is enough as long as the switch is actually routing.


Jon

Lee Dress Mon, 12/16/2013 - 12:59
User Badges:

yes, no control over the AT&T MPLS Network router.


the ip default gateway used to say 192.168.3.254, but I changed it hoping to let the ASA do all the routing instead of the MPLS router.

I'll remove it.


so to make sure I'm straight here, I can add an interface (like inside2 i.e 192.168.3.253) and point my core switch to route the .50 traffic through that, correct?

Jon Marshall Mon, 12/16/2013 - 13:21
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee

Actually my apologies but i forgot about the return traffic.


The traffic would be routed over the VPN tunnel to site 50 but when traffic is returned it will use the existing interface you have because that is where the routes are pointing and i don't think that will work. I say don't think because the ASA still sees the entire connection but it would be using 2 different inside interfaces.


I'm going to post on the Firewalling forum to see if this would be an issue and i should get a response fairly soon.


In the meantime, what is the model of the L3 switch ?


Jon

Lee Dress Mon, 12/16/2013 - 13:52
User Badges:

3750X. 

I'm planning on disconnecting the MPLS Router (50.254) at the remote site and using the same IP Address on the ASA that's doing the tunnel (50.254),. so I think return traffic should be fine.


my issue is getting to the site because the MPLS router keeps pointing traffic to the MPLS network, and not to my tunnel.


I can add another router in my office and point mpls traffic to the MPLS network, and point Site2Site traffic to the ASA.


But I figured my 3750's should be able to do that themselves.



Let me know if you hear more.

Jon Marshall Mon, 12/16/2013 - 14:58
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


I'm planning on disconnecting the MPLS Router (50.254) at the remote site and using the same IP Address on the ASA that's doing the tunnel (50.254),. so I think return traffic should be fine.

In the following when i refer to HQ i justy mean the site you are initiating the VPN from.

Do you want to test the tunnel before disconnecting the MPLS router at site 50. If so then here are the problems -


1) outbound traffic goes to the AT&T router for site 50 and is sent to the MPLS cloud. We can fix this with a separate connection as discussed.


2) traffic from site 50 to your HQ subnets would go via the MPLS connection at site 50. We can only fix this by doing NAT on your src IPs from HQ before going down the VPN tunnel. Your 3750 doesn't do NAT but your spare router probably would. So we could NAT your HQ addresses so when they are returned from site 50 they are sent down the VPN tunnel. And when they get to HQ ASA that ASA won't have a route to these addresses because you have used NAT so we can add a route to ASA via the new inside interface.


I had a response on my post and they suggested using NAT at the remote site. If we did this you would not need a separate connection. You simply use a subnet that the AT&T does not receive via MPLS and it should forward it on to the ASA at HQ.  This would be instead of 1) above but we still need to do 2).


If you did NAT the remote site subnets it would have to be policy NAT as you only want to NAT the remote IPs when they come down the VPN tunnel not when they go to the internet.


Having said all that, once you disconnect the MPLS router at site 50 it should work without any extra config/kit anywhere because site 50 will no longer advertise it's subnets to your HQ site so when you  try to connect from your core switch the AT&T router at HQ should simply send it on to the ASA because it doesn't know about site 50s subnets anymore. This is assuming you are exchanging routes across MPLS with a routing protocol and not using statics.  If you are using statics on the AT&T router then we are back to the above solutions.


If removing the MPLS router at site 50 stops the HQ AT&T router learning the routes then doing 1) and 2) above will only be so you can test the tunnel while the MPLS router at site 50 is conneced. Once you disconnect it you won't need any of it.


Sorry to overload you with info but it all depends on quite a few things as you can see above.


Jon

Lee Dress Wed, 12/18/2013 - 13:41
User Badges:

Jon,


thank you for your help.  Your ideas gave me many things to think about.

I figured out a way to go that will serve me completely going forward in the future.

here'as what I did,

I nstalled a second router between my 192.168.3.x network and my DMZ 172.16.100.x

I pointed all traffic on it to go straight to my ASA.


My ASA has static rules to point traffic back to the DMZ side of the  MPLS router for the MPLS sites,

and the remote ip address of each tunnel for the tunnelled point to point sites.  (I don't know if the second piece is neccessary or not, but it doesn't seem to be doing any harm)


This should allow me to just change the addresses on the ASA and not worry about MPLS thinking it knows the best route.  all i should need to do is change the address on the ASA, and clear any route cache that may be present.


router1#sho ip route

Gateway of last resort is 172.16.100.2 to network 0.0.0.0


     172.16.0.0/24 is subnetted, 1 subnets

C       172.16.100.0 is directly connected, FastEthernet0/1

S    192.168.0.0/24 [1/0] via 172.16.100.2

S    192.168.50.0/24 [1/0] via 172.16.100.2

C    192.168.3.0/24 is directly connected, FastEthernet0/0

S*   0.0.0.0/0 [1/0] via 172.16.100.2


so my core poits to 3.253, and that device points to the ASA.

so the ASA decides where traffic comes and goes to.


Let me know if you think there will be any issues with my plan.

Thanks again...

Lee

Jon Marshall Wed, 12/18/2013 - 14:09
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


So do you have this -



             ->  new router   -> 

core ->                                  -> ASA -> internet

             ->  AT&T router ->


If so, i'm not entirely following what you are doing but i will write out what i think should happen and see if it matches yours


site 50 = 192.168.50.0/24


You want to bypass the MPLS AT&T and use a VPN tunnel for site 50


1) on core switch/router


ip route 192.168.50.0 255.255.255.0 


this will mean traffic for site 50 goes direct to ASA


2) return traffic from site 50. Is the MPLS router still going to be connected and would site 50 use that as it's return path ?


What is at site 50 ie. is there a core switch that does routing like HQ or not ?


I ask because if site 50 uses the MPLS cloud it obviously won't work but because you have the router you can NAT the source IPs to a different range. Then add a route at site 50 to point this new subnet back to the ASA. Actually you might not need to add a route as i assume any unknown routes in site 50 get sent to the ASA anyway ?


If you do disconnect MPLS at site 50 you may still need to NAT because when the packets are returned via the VPN tunnel the ASA at HQ will think they should be sent to the MPLS AT&T router. 


Remember when the packets are returned to HQ the destination IP will be a subnet attached to the core switch. For normal internet traffic (ie non VPN) to and from HQ the path is -


core switch -> AT&T router -> ASA -> internet


internet -> ASA -> AT&T router -> core switch


and i presume you don't want to change this ?


If you don't then the easiest thing to do is when an internal subnet at HQ wants to connect via VPN it goes via the new router. You can then NAT the source IPs to a new range. You then add a route to the HQ ASA for that new range pointing to the router. So return traffic from the VPN goes via the router and non VPN traffic is unaffected.


Does this make sense ?


Jon

Lee Dress Thu, 12/19/2013 - 06:25
User Badges:

here's my new config


I think you miss the point that I'm unplugging the MPLS router from the 50 site.


My NY Office used to be on MPLS.  When we terminated the MPLS circuit,

I took an ASA, used the same x.x.x.254 gateway address as it's inside address.

built a tunnel back to HQ, and it came right up.  I didn't need to NAT anything or do anything special.

I pointed the route on my core switch to the HQ ASA address and since MPLS didn't know about the network anymore, it lit right up.


to avoid the MPLS network, I'm going to reroute 192.168.50.x to the NEW router. 3.253

disconnect the MPLS circuit from the 50 lan. (50.254)

build the tunnel in the ASA and attach the 50 ASA as the same address as the old MPLS address (50.254)


it should just work. 

no?


once the MPLS circuit at site 50 is disconnected formally via AT&T, it won't matter anyway.  the tunnel will work by default.

I just want to get them on the better circuit sooner, because AT&T Disconnects take 30 days or more to be completed.

Jon Marshall Thu, 12/19/2013 - 06:32
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


I think you miss the point that I'm unplugging the MPLS router from the 50 site.

No, i was more concerned about what heppens when traffic got back to HQ.


I pointed the route on my core switch to the HQ ASA address and since MPLS didn't know about the network anymore, it lit right up.

I may be misunderstanding but if the AT&T MPLS router at HQ doesn't know about site 50 anymore why do you need the new router ie. as you say the router just forwarded the traffic onto your ASA at HQ.


I can only see a need for the new router if the AT&T MPLS router at HQ thinks that site 50 is reachable via MPLS. If it doesn't you don't need it.


Jon

Lee Dress Thu, 12/19/2013 - 06:59
User Badges:

AT&T does still know about site 50.

that's why i need a new route.


and eventually, the at&t mpls network will go away and they will want their router back,

so I need to move off that router anyway.


I appreciate your help sir..

the whole nat thing is just throwing me because I don't need to do that on the other tunnelled sites.


i'm just trying to redirect around the mpls router.

Jon Marshall Thu, 12/19/2013 - 07:14
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


I am honestly not trying to complicate things for you. Can you answer this specific queston ?


Is the new router going to connect to a different interface on the HQ ASA ?


Jon

Lee Dress Thu, 12/19/2013 - 07:21
User Badges:

no.


the dmz address of the ASA is 172.16.100.2

the DMZ address of the MPLS router is 172.16.100.3

the DMZ of the NEW router is 172.16.100.10

Jon Marshall Thu, 12/19/2013 - 07:30
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

edited

Jon Marshall Thu, 12/19/2013 - 07:31
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


Okay, that makes more sense now.


Right the only issue is outbound and return traffic and this may not be an issue for you eg. lets say 192.168.5.0/24 is a subnet on your core switch at HQ. 192.168.5.1 is making the connections -


internet traffic (non VPN)


192.168.5.1 -> core switch -> AT&T router -> HQ ASA -> internet 


internet -> HQ ASA -> AT&T router -> core switch -> 192.168.5.1


the return path goes via the AT&T router because i assume you have a route on the ASA pointing to the AT&T router as next hop for 192.168.5.0/24


VPN traffic from HQ to site 50


192.168.5.1 -> core switch -> new router -> HQ ASA -> internet -> site 50 ASA -> LAN


LAN -> site 50 ASA -> internet -> HQ ASA -> AT&T router -> core switch -> 192.168.5.1


notice that the path back goes through the AT&T router and not the new router because of the route on the HQ ASA. Because your are using the same inside interface on the ASA the VPN tunnel will work as expected but it takes a different outbound and return path.


If this is not an issue for you then it should all work as you expect. If you want the traffic to use the same path both ways you would to need to NAT the 192.168.5.0/24 addresses to a new address or range of addresses as they passed through the new router outbound to site 50 then you can add a route to the HQ ASA pointing to that NAT subnet to go via the new router for the return traffic.


But like i say, it should work either way and it may not be an issue for you,


Jon

Lee Dress Thu, 12/19/2013 - 09:07
User Badges:

first off, thank you again for all of your help here..


right now the route on the core switch points to the AT&T router.


HQ core switch -> AT&T Router (3.254)  -> MPLS ->Remote (50.254)

Remote -> MPLS -> AT&T Router -> HQ core


at the remote site, when I disconnect the MPLS router  (50.254) and replace it with the remote ASA, I will use the same 50.254  so the remote site will still THINK it's the same route. 

everything in that site points to 50.254 as the default gateway.


at that point I will switch my core's routing to go through 3.253, so now the route will look like this:


HQ -> NEW Router (3.253) -> ASA -> VPN Tunnel-> Remote (50.254)

Remote -> VPN Tunnel -> ASA -> NEW Router(3.253) -> HQ


I  changed the route on my core switch to my NY Site (192.168.0.x) last night to make sure  that the routes from a working VPN Tunnel will route properly, and that  one is working fine.


Tracing route to [192.168.0.4]

over a maximum of 30 hops:


  1    <1 ms    11 ms    <1 ms  coresw1 [192.168.3.251]

  2    <1 ms    <1 ms    <1 ms  nj_router [192.168.3.253]

  3    16 ms    23 ms    16 ms  nydc [192.168.0.4]

Jon Marshall Thu, 12/19/2013 - 09:25
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


HQ -> NEW Router (3.253) -> ASA -> VPN Tunnel-> Remote (50.254)

Remote -> VPN Tunnel -> ASA -> NEW Router(3.253) -> HQ

Unless i am missing something it won't.  You need to reread my last post.


The return path will go via the AT&T MPLS router. Have a look at your HQ ASA route table at the moment. Pick a subnet from the core switch and see what route is used by the HQ ASA to send traffic for that subnet. It will be via the AT&T router at HQ.


Adding a route to the core switch for site 50s subnet only affects outbound traffic not return traffic.  The return traffic will get to the HQ ASA firewall and then be sent via the AT&T MPLS router to your core switch.


From my last post, assume 192.168.5.1 is on your HQ core switch then trace it through the network hop by hop to site 50 and then back from site 50.


The issue is not at site 50 it is once traffic gets back to the ASA at HQ. Like i say, it might not be an issue but only you can say. The VPN tunnel will still work because it is only after the traffic has left the firewall at HQ to get back to the core switch that the path is different.


Please have a reread of my last post and make sure you fully understand it. I just want to make sure any solution you implement you fully understand how it is working which will make it much easier later on when you start making other changes.


Edit - unless you are actually proposing to have all  HQ internet traffic (ie VPN and non VPN) go via the new router and you have changed the routes on the ASA for all the internal subnets on the core switch and pointed them to the new router, in which case please just ignore all of the above.


Jon

Lee Dress Thu, 12/19/2013 - 11:46
User Badges:

I tried switching the site over to the tunnel, and it did not work.


the MPLS router is disconnected at the remote site, but traffic stll refuses to return.

it could be maybe an arp cache on the remote site that makes it think that 50.254 is a different hardware address.


I might have to try this over a weekend and see if it clears up.


I'll re-read all of your posts tonight.


Thanks again..

Lee Dress Thu, 12/19/2013 - 13:05
User Badges:

you are right about the problem here, but it's going to take a weekend to fix for a permanent solution


my ASA tells every inside network to go through the AT&T MPLS Router to get back in.

that's confusing the 50 route.

Jon Marshall Thu, 12/19/2013 - 13:08
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


It shouldn't stop the VPN from working though, it just means a different return path. It can be fixed but it's really up to you whether you want/need to do it.


I don't think this is what stopped your test from working.


Jon

Lee Dress Thu, 12/19/2013 - 13:21
User Badges:

I was just informed that the 50 site will be disconnected on January 3rd, so when that's done, the VPN should work right away.


since I have no control over the MPLS Routing, I'm at their mercy.

that's why I'm trying to work "around" their circuit.


I'm just not 100% sure how I should work around it, and if I'll break my other sites.


I was thinking of pointing these inside addresses to the new router's DMZ address, and repointing my core switch to the lan side of the new router.

the only problem is I don't know how the MPLS DMZ routes, so I might just make a total mess.

Jon Marshall Thu, 12/19/2013 - 13:34
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


I was thinking of pointing these inside addresses to the new router's DMZ address, and repointing my core switch to the lan side of the new router.

the only problem is I don't know how the MPLS DMZ routes, so I might just make a total mess.


Outbound from the core switch I suspect it simply routes to the MPLS cloud for remote sites and if it doesn't have a route it sends it to the ASA HQ for internet traffic.


Return traffic from the internet it merely routes it to the core switch.


I wouldn't point all the subnets via the new router because then you are going to be in a bit of a mess. What happens for MPLS traffic from your core site if you bypass the MPLS router ?


I don't think the routing at HQ stopped the tunnel coming up. I think it was more likely, as you say, an issue at site 50 with arp caches etc.


I would leave your core switch pointing to the AT&T router for now. If you want return traffic from site 50s VPN to go via the new router we can NAT outbound on the router but be aware that would mean updating the VPN config on both ASAs because the address range would be different.


I think the new router should only be used for site 50 at the moment otherwise you could break all connectivity.


I would say again though, i do not think anything at HQ caused the tunnel to not work.  That said if you want site 50s return traffic to go via the new router we can setup NAT and update the VPN config on the ASAs.


Jon

Lee Dress Thu, 12/26/2013 - 09:11
User Badges:

Found my problem. 

The NAT rule on my ASA at HQ was wrong. 

everything else was

it was set to allow traffic between site 50 and HQ on the outside interface as opposed to the inside interface.

once I changed the NAT rule for the 50 site, and rerouted my internal traffic to the new router, the whole thing lit up.



I'm writing up the whole procedure so I don't make the same mistake again.

I have 3 more sites I need to switch over when they each get better internet connections installed.

the good thing is that I have a new router installed so I don't have to wait for a site to be disconnected from MPLS to switch them over.


Jon,

thanks again for all of your help. 

Jon Marshall Thu, 12/26/2013 - 09:28
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Lee


No problem, glad you got it all working.


Jon

Actions

This Discussion

Related Content