Designing for two WAN (non-Internet) links

Unanswered Question

Hi Guys,

I have tried to make the attached diagram as simple as I could. The scenario is:

Till recently we had only one ISP1 giving us intrasite MPLS VPN connectivity. We ran into major support issues with them and have hence decided to bring in a second ISP2 providing us alternative MPLS VPN links to three of our Site offices spread across two continents. We are sitting in say: Site 1.

At all the three sites, the MPLS links terminate on ISP administered routers (1800 ciscos).

Till now, it was simple, the ethernet interface used to come in and terminate on our L3 core swiitches but the problem starts now when second ISP2 will terminate his connections.

Is there a way I can utilise both the links in a load balancing manner rather than keeping one link purely in the passive way?

Which all routing routers will I have to bring into picture? Can GLBP be of help here? I need to minimise my dependence on this I have to involve ISP1 for they take months to even ad a summarised route onto their network when we raise a request to them.

ISP2 will hopefully be more proactive and they are even willing to administer the 'ISP2 Router1' which will eventually sit in our premises.

Pls advise

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Giuseppe Larosa Sun, 03/22/2009 - 13:34

Hello Fahim,

>> Is there a way I can utilise both the links in a load balancing manner rather than keeping one link purely in the passive way?

you need to use a dynamic routing protocol, the same one with both providers then you can get load balancing or fault tolerance.

For example you can use eBGP with maximum-paths 2.

router bgp 65000

maximum-paths 2

neigh isp1-ipaddr remote-as isp1:AS

neigh isp2-ipaddr remote-as isp2:AS

network internal-lan-base-addr mask

this config is needed on your two L3 switches on each site.

you need to avoid to advertise routes received from ISP1 to ISP2

! empty AS path

ip as-path-access-list 1 permit ^$

route-map only-local-routes permit 10

match as-path 1

router bgp 65000

neigh isp1-ipaddr route-map only-local-routes out

neigh isp2-ipaddr route-map only-local-routes out

Other solutions are possible but BGP has the advantage that neighorships are not built automatically.

The suggestion to make easy to use also other routing protocols is to use two different Vlans/ip subnets between the two L3 switches and each ISP router.

Hope to help


Thanks Giuseppe

One associated query, with respect to your last statement where you advise : "The suggestion to make easy to use also other routing protocols is to use two different Vlans/ip subnets between the two L3 switches and each ISP router."

Pls explain this part a bit more as I tried my best to conceptualise it but am not sure if I am able to really understand this. How should I use two different VLANs and what net advantage would I get?

Giuseppe Larosa Tue, 03/24/2009 - 01:11

Hello Fahim,

>> "The suggestion to make easy to use also other routing protocols is to use two different Vlans/ip subnets between the two L3 switches and each ISP router."

simply to use two different vlans in connecting to your L3 switches

vlan 10

isp1 router

l3 switch1 -- l3 switch2

vlan 20

isp2 router

l3 switch1 --- l3 switch2

to perform effettive L3 load balancing and fault tolerance whatever service they use (L3 VPN or L2 VPN) both providers have to present you a router with an interface.

I think BGP is a good choice because other protocols may reflect in some aspects the underlying service type.

Of course you need to setup BGP with both not only the new one.

Hope to help


Giuseppe Larosa Sun, 04/05/2009 - 04:52

Hello Fahim,

if one provider gives you a L3 service and the other one a L2 only service this is a problem for the load-balancing objective.

In fact, you may face the need to compare an eBGP path with an iBGP path and this can lead to a situation where it is easy to achieve redundancy but not to achieve load-balancing.

on the l2 path there is no service provider router to have an eBGP session with but it is like a direct link from your point of view.

so BGP would prefer eBGP path so L3 VPN provider would be preferred.

If using OSPF instead the L2 VPN path would be preferred (because it can provide internal routes while the L3 VPN PE can provide external paths).

if the provider implements correctly OSPF as PE-CE protocol the L2 VPN path can be seen as a backdoor path and the provider needs to add complexity using OSPF sham-links.

With some tuning the two paths can be used with OSPF to provide both redundancy and load balancing but requires stric cooperation with L3 VPN provider to adjust parameters.


Hope to help


Joseph W. Doherty Sun, 03/22/2009 - 13:38

How you might do this is going to depend on what you're doing now for routing, what the VPN providers (both) are willing to do, and the features of your equipment.

For instance, do you run any type of dynamic routing protocol with your original VPN provider, or do you do some type of static routing? How were you going to route to/from the new VPN provider?

If you've been doing static routes, you might just static route to the new VPN provider. Main issue is, if either VPN provider has a path problem to a site, it may be difficult for you to know. (With one provider it didn't matter, either you got through or you didn't, now you'll want to bypass a bad provider's path.)

If both providers will run a dynamic routing protocol with you, you should be able to determine if there's an end-to-end path, but if they offer different routing protocols, you might find it difficult to load share/balance.

There are techniques to bypass the VPN providers, but many of these might require software routers, not L3 switches. For instance, you asked about GLBP, but your diagram shows 3750s, which I don't believe support GLBP. Other features that might be useful might be multi-hop BGP, GRE tunnels, SLA object tracking, but again, without knowing more about your environment, difficult to recommend.

In other words, it's likely your goals could be accomplished, but how will depend much on how you could interact with your MPLS VPN providers and/or using equipment to reduce depending on your providers.

Are you in a position to implement and support a more complex network and/or purchase and deploy additional network equipment?

lamav Sun, 03/22/2009 - 14:21

"For instance, you asked about GLBP, but your diagram shows 3750s, which I don't believe support GLBP."

Yup, you're right. The 3750s do not support GLBP. I had to research that a few months ago.

Thanks for the reply Joseph.

To answer your points and seek further inputs;

Currently, we don't run any dynamic protocol with our original VPN provider. The new one maintains that they are willing to provide connectivity at Site 1 as Layer2. Do you feel I should ask for the end node to be Layer 3 instead?

The new guys are also willing to give us an option of either static routing or eBGP at all three locations.

I do agree with your point though that static routes on both providers will leave me blind to path problems.

If I get both ISP's to allow me do eBGP on both providers, would it take care of both load balancing and and auto redundacy?

Say, I upgrade my two core switches from 3750s to 4500 series of cisco, would I be in a better to design around GLBP and reduce dependency on providers? Pls advise!!

Finally, I am in a position to get the VPN provider (one of them) do things for me but the turnaround time is something which I am really disappointed but given that I have a clear idea of the best way forward, I am willing to take it up.

And yes, I am in a position to implement additional equipment? which you would feel will fit the bill. We have already decided and started to purchase riverbed's for WAN optimisation. If there has to be one more, so be it?

Joseph W. Doherty Tue, 03/24/2009 - 04:25

It's unclear, to me, what your complete physical and logical topology is today. You note you don't do any dynamic routing, but do you do any static routing? You note original provider is willing to provide L2, but what do they provide now, L2 or L3? In other words, do all sites look like the site 1 in your diagram, i.e. you have local device (L2 or L3?) and it connects to provider's router? Does each site have its own subnet or subnets, or does you whole environment share one subnet? If each site has it own subnet or subnets, do you "see" other sites' subnets, or do you only see a transit subnet or subnets? (E.g. each site just has a default route to the WAN?)

It's not really so much a question of whether you can do eBGP with the providers or not, but whether you have a L3 topology and whether you can exchange information (dynamic routing) about that topology with the providers. Having that will allow redundancy with both providers. If they support similar dynamic routing, it often makes it easier to also load share.

For equipment, I wouldn't see a 4500 much of a benefit vs. a 3750. Although the 3750 doesn't support GLBP, I suspect Cisco didn't see the need for it to do so since you can stack them. The 3750 also supports various routing protocols with the advance feature IOSs.

What I had in mind for other equipment, might be full fledged routers (e.g. ISRs). Also unclear whether you would need them, but they offer features not available in both the 3750 or 4500 series.

Since you've noted the original provider is slow to implement change, depending on what they do now, it might be possible to use them as is. You would then "work around" them by some combination of how you have provider 2 build out and what you do with equipment under your control. Another reason I asked about your willingness to buy equipment, you might be able to lessen your dependency on provider 2, too. That might be done by "working around" provider 2 also, or by taking over some of the features that provider 2 supports. For example, instead of the provider managing the WAN router, you do.

One important thing to keep in mind, if you don't have the expertise for doing all this yourself, you might want to retain local consultation. Doing this correctly is too complex for suggestions from this forum unless you have the necessary expertise. If you do, they can provide great ideas that you may not have thought of, if you don't, these forums might be used for a 2nd opinion.

Hi Joseph, having given due thoughts to your statement that said " Doing this correctly is too complex for suggestions from this forum unless you have the necessary expertise", I would still want to know from you as to what are my options to by additional equipment and do a 'workaround' both the ISPs?

I have come across various WAN link load balancers by F5 and Barracuda's but they don't seem to fit the bill.

Do you have something specific in your mind?

Pls advise!!

Joseph W. Doherty Sun, 04/05/2009 - 04:26

"Do you have something specific in your mind? "

No, because I also wrote "It's unclear, to me, what your complete physical and logical topology is today.", which was followed by some questions highlighting some unknown items. However, load balancing via dynamic routing might be part of a solution.

It might be possible something can be done to "workaround" both ISPs, but what that something, or somethings, might be is constrained by your environment. It's hopeful, though, that you can obtain additional equipement, if necessary, since this often makes possible options.

As for " Doing this correctly is too complex for suggestions from this forum unless you have the necessary expertise . . .", that was in context "doing all this yourself".

I think these forums are great for answering specific technical questions and/or addressing concepts, but if you do need a "complex" design and/or implementation, especially to work around your service providers, might be difficult to accomplish based just on forum suggestions.

Mohamed Sobair Tue, 03/24/2009 - 00:04

Hello Fahim,

Whether it is layer-2 or layer-3 VPN service, In order to have loadsharing effective , you will need to connect both ISP1 and ISP2 routers.

You will also need to have both core 3750 connected with both routers at layer-3 for rapid convergence and fast failure detection. Assuming your Cores acting as aggregation switches, you will be having also a direct link between the Cores.

On the LAN Segmnet where your services and hosts reside, I will manually spilit them and run MHSRP Groups for example. I dont know whether its possible or not based on your current setup but its an option.

The Main point here is your WAN link to two different providers. Is there any sort of connectivity between ISP1 and ISP2 router OR this it can still be designable?



Thanks for the advise Mohamed.

When you say I need to connect botht he routersm do I need to connect them using an inbuilt interface, one to one? The two routers are 2800 series so I might need to add an interface HWIC and use Virtual VLAN addressing mechanism to assign a Layer 3 IP.

To asnwer your query, I currently have no connectivity betweem ISP1 and ISP2 except that they both reside in my data center.

My inability to get anything out of ISP1 apart from the core services configured at the starting, is puching me to try out ISP2.

ISP2 is willing to give me administrative access over their router which ISP1 doesn't but I can always pressure them if I am clear of my needs to make the two links work in the most optimal way.

Pls advise!!

Mohamed Sobair Tue, 03/24/2009 - 03:44

It doesnt have to be physically on seperate WIC device. the addressing scheme is the last point we should care about here, but the best choice is to have virtual lan setup.

The Idea is to have the L3 core switches connected with both ISP routers on the LAN segment, this can be achieved by using ur current addressing scheme and have ur l3 interfaces connected. So, Every L3 core 3750 Switch Wan interface will have logically 2 links with both ISP1 and ISP2 lan ports at layer-3.

Having that done, at this point you can do loadhsharing by implementing dynamic routing protocol such as BGP on both ISP routers. I recommend BGP on the WAN in order to isolate it from ur LAN dynamic routing since ur core Switches could be running Eigrp or OSPF for thier connectivity with the upstrream routers.

Pls let me know if u get my idea, I will also forward u a helpfull link that can help u achieve loadsharing with BGP in Such case.



Hi Mohamed

I am a bit stuck at two issues.

1. If one of the provider maintains that he will provide me L2 MPLS at one of the sites, can I make him participate in BGP between CE and PE routers?

2. Just by enabling dual paths from each of my L3 switches and running BGP, would I accomplish load sharing?

3. What all is needed to enable OSPF on my internal side? Is there designing aspect involed.

Would love to recieve that 'helpful' link you've mentioned.


Mohamed Sobair Mon, 04/06/2009 - 01:26

Hi Fahim,

1. IF the provider maintains he will provide L2VPN serice (Overlay VPN Model), then he cant participate in your L3 routing.

2. Enabling Dual Path from your Core Switches besides BGP (Assuming now the LAN Interfaces of ur ISP routers are accessible from each) would achieve Loadsharing, By doing So, you will ensure redundancy and loadsharing from the Core Switches as well as from the ISP routers. Dividing your Networks to be advertised to the rest of the Sites in Segments, and prefer some over another on Each GW router would achieve loadsharing.

3. On your Interneal LAN, you will need to have the CORE Switches part of AREA0 (Backbone) Area, the use of Stub routers and summarizes the Routes at AREAO would limit the number of route exchanged. The number of routers per Area and neighbors is based on lots of factors and based on your previous posted diagram, you dont have to worry about that. Another Note, the Area border router keeps state and copy of each linkstate database for every area, so it requires more CPU and memory resources.

Here is the OSPF deign Guide for ur reference:

Attache bellow is "Loadsharing with BGP" when Multihomed to two different Service providers.




This Discussion