Current recommendations/best practise for branch to head office connectivity?

[mitchen]

Our branches currently have Cisco 877 routers with ADSL connections and we connect them back to our head office data centres using IPSEC VPN terminating on ASA firewalls (Active/Standby – with the Active ASA in our primary datacentre and Standby the ASA in our “failover datacentre” a few miles away, a leased line connecting both datacentres) We manage all the kit ourselves and use the same ISP for our datacentre leased line connections as for our branch ADSL connections.

This was all implemented several years ago and works reasonably well, the main problems we have experienced being:

• •- ADSL reliability, no “real” SLA against ADSL – although relatively uncommon, it’s not completely unknown for a site to be without connectivity for a week or more.
  
• •- ISP reliability – we chose a “smaller” ISP and perhaps have sometimes suffered as a result (although things have improved on that front recently)
  
• •- ASA stateful failover issues – we have often found that whenever we do have a failover to our secondary ASA, the stateful failover has not worked as it should, we’ve never really got to the bottom of these issues despite extensive troubleshooting by our Cisco partner/Cisco TAC
  

We are now looking at refreshing our network infrastructure and I wanted to find out what current best practise/recommendations were. Obviously cost will be a big factor in anything we do (if not the biggest factor!) but essentially, if we were starting with a blank piece of paper, what would be the best way of doing things:

• •- W have several hundred branches
  
• •- Looking for more reliability/resilience at an individual branch level (though cost obviously becomes an issue here – we currently have a single ADSL connection, moving up to e.g. a leased line would probably not be an option! But what other options are there? EFM might be a more cost-effective alternative to a leased line? Using 3G as a backup to the ADSL? Bonded ADSL?
  
• •- Head office reliability – would a dual ISP solution at the data centres prove more effective/resilient? How could we implement this with Active/Standby ASAs?
  
• •- Rollout – if we were to rollout several hundred new routers to branches, is there any methods to facilitate such a rollout? We have a small IT team and don’t think we’d look forward to sitting there manually configuring each router and dispatching it!
  
• •- MPLS v IPSEC? When we looked at this many years ago, MPLS solutions were far too costly for us, so we figured we could achieve the same results with a hub-and-spoke IPSEC VPN solution. How have things changed nowadays? Would we be better to look at an MPLS type solution (with QoS etc)?
  

I’m just really trying to get an idea of options and what other people are doing before we engage with a Cisco partner to take this further so any feedback would be appreciated!

Thanks.

[Leo Laohoo]

Geez.  This is not going to be an easy topic to address.

I recommend you hire a network consultant or a network integrator.  We try our best to point you to the right direction but there are many ways to go wrong. 

[mitchen]

Ha ha! You're not wrong there! 

But I was really just hoping for some general thoughts and opinions rather than to get a completed design solution from the forum!  

We will, of course, engage with a Cisco partner to take this forward but I was hoping to get some general feedback as to the kind of things people are doing nowadays etc before doing that - so any thoughts would still be welcome....

[Leo Laohoo]

xDSL - How "important" are these branches?  I'm not sure about other countries but xDSL is "best effort" in form of reliability and traffic.  I mean if the link should fail, then chances of you getting it fixed within an day is very, very slim.  xDSL doesn't have an SLA, unlike leased line.

If you are talking about redundancy in regards to sites with xDSL then you'd better think about getting a lease line instead.  Another point of contention is budget.  In my "perfect world", when talking about redundancy for a site, I'd be thinking of a dual router and dual lease line.  Each lease line will NOT be going to the same exchange.  I don't mind going to a single provider.

3G as a backup?  I wouldn't count on it.  When you start to use it, it's not only slow but it's very expensive.  Again, I'd consider a leased line.

MPLS vs IPSec - MPLS is the way to go, no question about it.  You can always push traffic, MPLS or not, down an IPSec tunnel.  The main question is whether or not your router can cope with the traffic.

A few years ago I rolled out about 20 routers all over Australia.  The most difficult process is to arrange a change window and a lucky tech to install the router and sit back (read a magazine) while I go about cutting over the new service.  I also had to make sure the config is "air tight" and no room for a roll-back.  So planning and ironing out the config is equally important.

During the implementation stage, I made it a point to cut over three sites in a week.  This way I don't get stressed plus I don't get confused about the sites.  After every cutover (for two weeks), I'd review the different "mistakes" and lessons.

Otherwise, everything went pretty well (sure, minus several type-o's in my part).  Not every cut-over was perfect, however, I never had a roll-back. 

NOTE:  I know that 20 (or so) sites can't be compared to your 100 or so. 

[Alessio Andreoli]

Even if i do agree with Leo,

i'd say that you should start to choose MPLS over IPSec tunnels for many reasons among the ones:

1) less maintenance

2) more flexibility

3) MPLS VPN are "physically" separated networks - that means you would have your own private network

4) hardware investment

5) Scalability

then you could keep avoiding to consider the rollout of many hundreds of router a big issue. Very skilled engineer can do it in a relatiely short amount of time (everything depends from your deadline)

Head Office reliability - Well, no discussionabout a multihoming configuration if you are working in a HA (High Availability) context. Do note that a multihoming config requires skilled engineer managing the BGP/MPLS or WAN protocols in place!!

EFM might be a very clever idea but you need again some cool network architects doing the right assesment as much as for 3G technology where a risk analysis should take place too .

For "failover datacentre" i did not get wha you mean. Are you backing up a DC with another one or do you have just an ASA in one place and another ASA in another DC?

For the first three points i would say that a network consultant (hopefully CCIE Security) is the one you need.

Hope this helps

Alessio

[mitchen]

thanks for the feedback guys - that's exactly the kind of thing I'm looking for, particularly interested in the MPLS v IPSEC stuff (as I say, when we implemented our current solution well over 5 years ago now we looked at some MPLS solutions but they were so costly we decided to go down the IPSEC route as we could get a "comparable" solution for much less - but one of the things I wanted to understand was how approaches/thoughts on such matters have changed in recent times)

[Karsten Iwen]

MPLS is one of the best solutions if you need a high reliability (*and* you have a provider that knows what he's doing). But of course they are still much more expensive then DSL-lines. Thats the reason most of my customers run business-DSL-lines for the spokes. These have a guaranteed support-time which are normally not as good as with MPLS but still good enough. And the saved money is really a big amount if high bandwidths are needed. The Head-Office typically have leased-lines for the spoke-to hub-traffic and in addition DSL-lines for the "normal" internet-traffic like surfing the inet.

Up to now the customers are quite comfortable with that as the DSLs run really good. At the beginning we were not sure if that's good enough for VoIP, but these services also run fine without major problems.

-- 
Don't stop after you've improved your network! Improve the world by lending money to the working poor:
http://www.kiva.org/invitedby/karsteni

[sean_evershed]

Hi,

All of the customers I have worked for have chosen MPLS. My advice is to always be in charge of your own routing across the WAN links if you choose this solution. I worked for one customer where the carrier was in charge of routing. This meant that they needed to inject new static routes into their MPLS cloud every time a new site was opened or routes were changed at an existing site.

As a result the carrier often forgot to inject the new routes in a timely manner or injected the wrong routes. They also charged an exhorbatent fee for a very simple task. Needless to say this lead to much frustration.

The MPLS vs IPSEC debate will be driven by your security policy. I have worked for a compnay that uses MPLS to connect all their remote sites.  Sensitive customer data was traversing these links. Therefore the security policy mandadated that these links also needed to be protected by IPSEC VPN tunnels. This is despite the fact that we were using a private MPLS cloud.

Cheers

Sean

[Karsten Iwen]

Yes, the routing is a very important point. One customer has a quite "cheap" MPLS-provider and they didn't have the option to run a routing-protocol. The only option was to do a static routing. That really was a nightmare and caused a lot of trouble. For encryption we were also forced to run tunnel-based VPN and were not able to implement GETVPN as we wanted to do it (technically GETVPN had probably worked, but we decided against it because of the missing control over the routing).

-- 
Don't stop after you've improved your network! Improve the world by lending money to the working poor:
http://www.kiva.org/invitedby/karsteni

[mitchen]

Thanks, this is all very useful feedback for me.

As far as security is concerned, there won't be anything particularly sensitive that requires strong encryption or anything so there certainly wouldnt be a need to protect with IPSEC if we moved over to an MPLS solution.

Useful to know about the routing - indeed, I have worked alongside a company with an MPLS solution and found the same sort of thing, we had to wait ages for their service provider to set-up their routing (meanwhile we were patting ourselves on the back for how efficient our side of things had been! )

So, following on from this, for the MPLS solutions you guys have worked with - do the customers provide their own CPE or do you leave it to the service provider to supply the router and "manage" the service?  Our preference has always been to do things ourselves where we can (possibly some self-preservation going on there admittedly!) but most of the service providers I have spoken to on MPLS in the past have always "sold" it as a fully-managed service.   Any thoughts on that side of thing and the pros and cons?

[sean_evershed]

Hi,

If you chose a managed service then you lose control of your network. As a result I have had negative experiences with this type of service.

- On many occasions we knew that a site had failed before the telco did.

- The telco was unable to produce adequate monthly WAN stats, for example how much bandwidth each site was consuming.

- The telco made a mess of the QoS policy across the WAN and so this impacted voice.

- The telco's help desk was very bureaucratic. Therefore it can a long time to get a problem escalated to the Level 3 engineers who can fix complex problems.

I would only use a managed service for two reasons:

1. You have a small network team with a limited skills set.

2. You have remote locations where your company does not have local network staff. It would then be up to the telco to provide the staff necessary to travel to a site to replace a faulty router or investigate network problems.

Cheers

Sean

[mitchen]

Hi Sean,  thanks for the input - couldn't agree more with you on all of that!