Routing over Carrier Ethernet w/VPN and failover to DSL w/VPN

[RHITCHCOCK]

I've been banging my brain against this for a bit, and I need some advice from those more experienced.

The diagram I've attached crudely shows how our WAN is currently set up (sorry for my poor diagram skills).

Site A, Site B, and Site C communicate with each other over VPNs on fiber Internet connections.

Site D communicates with Site A over a point-to-point T1, and to all other sites over VPNs on a DSL Internet connection.

Site E communicates with all sites over VPNs on a DSL Internet connection.

Site A, Site B, and Site C have the ability to fail-over to VPNs over DSL if their Fiber connection is down.

Site D can communicate with Site A over VPN over DSL if the T1 goes down.

The benefits of this setup are:

1. Failover is automatic using route tracking at Site A, Site B, and Site C.  If the Fiber at Site B goes down, The ASA changes its default route to the DSL.  Access-list rules are in place to allow traffic through VPN over either the Fiber interface or the DSL interface.  All other ASAs have both the Fiber and DSL IPs as peers, so they can communicate to either.

Likewise, at Site D, if the router senses that the T1 is down, it falls back to a route sending traffic bound for Site A to the ASA5505.

(Obviously, Site E has no failover option.)

2. Sites communicate directly with each other - Site B has a VPN to Site A, Site C, Site D, and Site E.  No routes pass through another Site to reach their destination.

However, communication with Site A is by far the most important factor.  In a failover situation, we are able to sacrifice communication between Site D and Site B, for instance.

We are looking at moving to a carrier ethernet solution.  Basically this just gives us Layer 2 connectivity between some of the sites.  This would replace the Fiber connections in the current diagram.  We would, however, retain the Fiber connection to the Internet at Site A.  We would also retain the DSL lines at the sites that currently have it, for failover.  Site E would remain DSL only, Site D would remain on a T1 (with the DSL for failover).

We are trying to accomplish the following if we move to the carrier ethernet solution:

1. Route all Internet traffic (with the possible exception of Site E) through the Internet connection at Site A.

2. Continue to encrypt internal traffic, even coming over the carrier ethernet.

3. Failover to VPNs over DSL if the carrier ethernet goes down automatically.  If not automatically, with as little manual intervention as possible.

So, the big question is, can I do this with our existing equipment?  And I don't think I can, if for no other reason than I'm going to be out of interfaces on the ASA5510 (in addition to what's on the diagram, there's a DMZ port in use).  So, what if I add another ASA at Site A, specifically for Internet traffic (and the VPN to Site E)?

Let me start with the first thing we are trying to accomplish.  Can I route all the Internet traffic through Site A, assuming the carrier ethernet is up and working at all sites?  If so, how?  Should I encrypt all traffic, send it through a VPN, nice and NAT-exempt, and then route it out a second ASA at Site A?

I realize this is pretty long and involved.  I would appreciate any advice anyone has.  Thanks!

[gatlin007]

That is a nice drawing and I think carrier Ethernet is a good fit for this size and type of network.

The ASA is a good firewall and adequate encryption device; but for the level of fault tolerance you are interested in I think a Firewall Router is a better fit.

A firewall router makes forwarding desicions based on routing information versus the static ACL of a crypto map.

Almost any IOS router with the advanced security feature set would be adequate.  At that point the platform should be determined based on forecasted data throughput.  You'll be very happy with the increased features available in a router versus the ASA.

I would suggest a dynamic routing protcol manifested over GRE/IPSEC tunnels if you'd like the traffic encrypted regardless of the underlying media.  Everything you describe is possible as you'd control the defualt route advertised to the remote sites forcing them to send internet traffic via site A.


Chris

[RHITCHCOCK]

Chris

Thanks.  The diagram is a bit simplified.  We actually have a total of 5 fiber-connected sites, and 2 T1-connected sites (and one unfortunate DSL-connected site).  But it represents the different types of connections we have.

Do you have a suggestion for model of router to look at?  These carrier ethernet connections will only be 10Mb, so it's not as though we'll be pushing too much traffic.

Currently all routes are either connected, static, or EIGRP (on the T1 routers).  Do you have a suggestion on a dynamic routing protocol to move to if we purchase the firewall routers?

Lastly, would what you are suggesting work something like the new attached diagram?  I assume the router can encrypt, NAT exempt, and send all traffic, regardless of whether I'm doing carrier ethernet (using a private ip schema) or routing over the public Internet?  In other words, is the traffic essentially unaware of how it reaches Site A (and the Internet) because all it sees is its local router, then the next hop of the router at Site A, because the traffic in between is all in a VPN tunnel?  This would be different than the ASAs, in that a traceroute between sites now never shows the ASAs as hops.

If the router is able to detect a failed link through the carrier ethernet and automatically build tunnels using the DSL interfaces, I'm starting to see how this would work!

If you have any advice for the questions in the previous paragraphs I would greatly appreciate it.  Thanks for your help so far!

[gatlin007 · Accepted solution]

Robert,

I caution that a sales engineer ought to be consulted for the proper platform.  That said I would think a 1800 or 1900 series would be good for your branches.  Your hub site may be a 2800/3800 or 2900/3900 series router depending on throughput there; I would imagine it would be greater than 10Mbps?  These routers support a VPN accelerator card that offloads the encryption process into hardware vs. software.  A sales engineer will be well versed in all these options.

One critical descsion point to to make sure you get a feature set that supports dynamic routing, GRE and IPSEC.  The Cisco sofware advisor will be helpful here.

http://www.cisco.com/cisco/web/download/index.html

Since you already have EIGRP in your network and your engineers are already already familiar with  this protocol I'd stick with it; make this an EIGRP network.  Utilizing GRE will create tunnel interfaces; you'll be able to adjust the delay attribute on these interfaces to cause one to be preferred over the other.  These routers can NAT or not based on policy and egress interface.  The encryption decision will be based on whether or not a GRE tunnel interfaces is selected for packet forwarding.  The interface selection decision will be based on route metrics in EIGRP.

You are correct; the traffic forwarding decision will be unaware of the underlying media of internet or carrier Ethernet.  It will only be aware of the tunnel interface it forwards over and the delay attribute you select for the tunnel interface.  You will need to adjust the MTU but that's another conversation.

I would think you'd have two tunnels built and ready for use at all times.  One over carrier Ethernet, the other over the internet.  Only the more attractive tunnel would forward traffic while the other waits for a failure situation. 


Chris

[RHITCHCOCK]

Chris

Sounds great.  I've got some crash-courses to take (I'm not currently utilizing GRE for example, so I'll need to do some quick learning), but if I can get everything sorted out this solution should be perfect.  I have a feeling I'll be back on these forums before I'm done... for now you've definitely answered my questions and I'm marking your answer correct.  Thanks very much for your advice!