BGP with three ISPs

Unanswered Question
Mar 5th, 2008

Hi. We are connected to three different ISPs. Two in location A and one in location B. Location A and B are connected via dedicated circuit. So there are three routers and three ISPs. Is there an example out there somewhere that allows me to use all three and failover to the alternate site if the Internet goes down locally?


  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Joseph W. Doherty Wed, 03/05/2008 - 17:42

Could you provide more details on both current internal routing and routing between you and the ISPs. Also, the particulars with your "public" presence on the Internet (e.g. your own AS or ASs).

netsec123 Wed, 03/05/2008 - 18:48


We have three AS numbers here -- one from each provider... I want to try to get an idea of the configuration for the three edge routers if the one in site B terminates BOTH an INTERNET connection AND the DS3 to a router in SITE A. The router in Site A is on a segment that ALSO has two routers going out to the other two providers... I had to DRAW THIS to understand. :)

Danilo Dy Wed, 03/05/2008 - 20:48


In your response, you mean to say you don't have your own ASN and you are using your 3 providers ASN - I don't know how you do that. Do you have your own IP Block or you also got them from your 3 providers? This is very tough!

Fix your ASN and IP Block problem first.

Option 1

If you have your own IP Block, you can request for your own ASN from ARIN, RIPE-NCC, APNIC, LACNIC, or AFRINIC.

Option 2

If you have your own IP Block but you can't get your own ASN. Use Private ASN, talk to your 3 providers to agree on the single Private ASN that your will use.



netsec123 Wed, 03/05/2008 - 21:25

Thank you so much! The three ASNs are ours, one TO each ISP... INTERNALLY we are using 11821 but peer with Qwest's 209, Lightpath's 6128, and Verizon's 701... I do not want to put all the info online. :( The issue is that another engineering team started this and left it in an unfinished state, making it more challenging... I'd REALLY appreciate some help here.... :)


Danilo Dy Wed, 03/05/2008 - 22:53


You don't need to put them online, I just want to be clear.

I assume that each router have one ASN and one IP block and they are all not interconnected. Total is 3 ASN, 3 IP blocks, 3 routers.

If you plan to interconnect them together, here's a suggestion.

- For simplicity use only one ASN

- Talk to 3 upstream provider to transit the 3 IP blocks from a single ASN

- Connect R1 to R2 and R2 to R3 using iBGP. You can build IP GRE Tunnel between R1 and R3

Any one or two provider down, your ASN and 3 IP blocks is still reachable from the remaining provider/s.

However, since there is only a single connection between R2 and R3 and R1 to R3 is a logical link. Chances are ,if the connection between R2 and R3 is down, you will encounter split-BGP. To address this, choose one; 1) You can install a low bandwidth link between R1 and R3 to keep the iBGP up, 2) Install analog line to R3 and login to it to shutdown BGP peering to R3 upstream provider if connection between R2 and R3 is down, 3) Setup a monitoring system in R3 location that will monitor the link between R2 and R3 and when its down it will kick up a script to login to R3 to shutdown the BGP peering to R3 upstream provider.

If you want to keep the 3 ASN and advertise the 3 IP block one from each ASN.

- Choose one of the ASN to be the edge between you and your upstream providers

- Talk to 3 ISP to transit the 3 IP blocks. Use one of the ASN as edge.

- Use route-map to advertise the 2 IP blocks using their own ASN behind your edge ASN



Joseph W. Doherty Thu, 03/06/2008 - 05:43

Your requirement is just failover Internet access to/from internal? If so, what's the internal routing to and from your Internet edge? Any hosts visible to the Internet with Internet addresses?

netsec123 Thu, 03/06/2008 - 19:12

Hi and THANKS again!!! YES, hosts need to be accessible from the Internet so I know that they may need to have extra DNS entries placed on public DNS servers with lower weights. BUT, that is phase II -- Right now, I'm just hoping to get the failover working right.. Is there a way I can send configs without posting them for everyone -- there's too much to xxx out....

This really helps!!!

Rick Morris Mon, 03/10/2008 - 13:55

ASN's are not a security risk, all ASN's can be publically seen by any whois search.

i shall read on so I can see what I can add.

Rick Morris Mon, 03/10/2008 - 14:01

hmmm...allow me to play consultant for a minute.

3 sites

Site A - 701

Site B - 6128

Site C - 209

May not be exact but I want to note that all three sites have different Peers.

Your AS is 11821



Address: 372 Danbury rd

City: Wilton

StateProv: CT

PostalCode: 06897

Country: US

ASNumber: 11821


ASHandle: AS11821


RegDate: 1998-12-31

Updated: 2005-07-05

I assume this is you?


Do you have your own IP space?

Does each site have their own IP space?

What is the requirement you are hoping to acheive?

How do all the sites communicate together now?



netsec123 Mon, 03/10/2008 - 20:30

Rick, THANK YOU. Yes, IP space does exist in each location. Hopefully, we are trying to provide site redundancy such that if one site goes down, the other site will provide access to the Internet. Ultimately, the sites should be mirror images of all services for DR... That make sense? I'm happy to share configs as well..

THANKS!!! :)

Rick Morris Tue, 03/11/2008 - 09:21


I think I have a better understanding now.

For what you want to do you will need some form of dynamic routing. BGP is my recommendation. You will run BGP with your peers and run iBGP between your sites.

Now you may get into some complications with route announcements to your peers.

If each site has a block from the ISP you may not be able to announce to any other ISP.

For example, Verizon IP may not have the ability to be routed across Sprint or whoever, they may reject the route. Also, if the IP block is smaller than a /24 it may not get announced beyond the ISP border to the Border Peers, make sense?

You might want to apply for your own IP space and subnet it out and your own ASN.

netsec123 Tue, 03/11/2008 - 19:43

SO, if I am hearing you correctly, all I really need to do is make sure each ISP is announcing a subnet out their line FROM my CPE?

Danilo Dy Tue, 03/11/2008 - 19:57


See if you can do this...let me know the constraint if you can't.

- Decom R3

- Connect R4 to R2 (DS3). You may need a DS3 Module

- Build IP GRE Tunnel between R1 and R4. IP GRE Tunnel is now a standard, non-Cisco routers should support it.

- Create Loopback interfaces in R4, R2, and R3 to be use for BGP

- Run BGP in R4, R2, and R1 using ASN11821

- Advice QWEST, LP, and VZ to allow your prefixes (all prefixes) as they have ACL what to permit from you.

- Peer with QWEST, LP, and VZ using ASN11821 advertising your prefixes



netsec123 Tue, 03/11/2008 - 20:01

Dandy, I think that's an excellent idea. The only issue is that we brought that up and priced the DS3 cards - BIG BUCKS! That's why we did not go with it... :( BUT, it is very reassuring to see we are thinking along the same path.

TF :)

Danilo Dy Wed, 03/12/2008 - 04:14


Any chance to replace DS3 with MetroE or MPLS?

MetroE and MPLS use Ethernet which you may have a spare port in your routers, have you check the migration cost (one-time and recurring)?



netsec123 Wed, 03/12/2008 - 06:53

You're making me cry... :)

I can't replace any hardware here --- unfortunately, it is what it is... I know that makes things more difficult - sorry...

Rick Morris Wed, 03/12/2008 - 05:29

Sort of. For failover and redundancy you need all your routes to be announced to all your providers. However, they may not allow you to route other vendor IP's.

You will also need to have some routing protocol between your sites so you can route through them. I think I saw medan's post about GRE. This is one option, or iBGP between your sites.

netsec123 Wed, 03/12/2008 - 06:52

I would like to use iBGP and I 'thought' I had that configured already... including on the router that did NOT terminate an Internet line from an ISP but DID connect the sites... Does that make sense?

Danilo Dy Wed, 03/12/2008 - 07:11


See if you can do this...let me know the constraint if you can't.

- Both R4, R2, and R1 should have direct physical connection to R3. R4 (DS3), R2 and R1 (direct ethernet or through switch)

- Configure IP GRE between R2 and R4 through R3

- Configure IP GRE between R1 and R4 through R3

- Create Loopback interfaces in R4, R2, and R1 to be use for BGP

- Run BGP/iBGP in R4, R2, and R1 using ASN11821

- Advice QWEST, LP, and VZ to allow your prefixes (all prefixes) as they have ACL what to permit from you.

- Peer with QWEST, LP, and VZ using ASN11821 advertising your prefixes



netsec123 Wed, 03/12/2008 - 07:56

Hi, THANKS. I will try this... I was unaware I needed to do GRE tunnels if BGP routers were not directly connected. I thought if they were in the same iBGP AS routes would propagate [i.e. ALL R1 --> R4 running ASN11821] .. Am I wrong?

Rick Morris Wed, 03/12/2008 - 08:00

I am not sure why the GRE if running iBGP.

If you are announceing your blocks between your routers then you will propigate the routing table. That should be all you need. I am not sure why the GRE is mentioned in the previous post.

netsec123 Wed, 03/12/2008 - 08:22

Yes -- That's why I mentioned in my last post the same thing -- GRE may should [should not?] be needed....



This Discussion