BGP Multihoming design topology

[camdoggle]

Hello,

Currently we have a 50mb pipe with our carrier SONIC. We have signed another contract with another provider here in town (Charter) to multihome our Internet connections in an active/active configuration. We have leased our /24 space through our carrier SONIC. ARIN has already approved our org-ID for an ASN and they will be sending us that once the billing portion is finished.

There a few design considerations I was hoping I could get some insight from the community on.. Before I start, the ultimate goal for us to use BOTH Internet connections in an active/active configuration - utilizing both pipes..

Disclaimer: I have gathered this design from a lot of other posts that have somewhat of a similiar topology with ASA-->3750-->router pair-->CPE--internet.. Please keep an open mind if you think im on the wrong track..

Please see the attached design topology.

Questions related to design:

• What kind of routes should I get from each carrier? I have been told that partial/partial routes plus a default route form each carrier is the way to go. Also, I've heard mention that full routes from both carriers are preferred. My ASR1001's can support ~500k routes. I know the global table is approximately ~337k routes. My goal is to use both pipes and use the best outbound path per carrier.
  

• We will be leasing our /24 space from SONIC. I plan on running OSPF on the DC-Edge-SW1 in conjunction with iBGP - so I can default originate two equal cost routes back to my ASA. My confusion is when the traffic hits DC-Edge-SW1, there will be default equal-cost iBGP routes to both ASR1001's (DC-Edge-RT1 & DC-Edge-RT2). If the switch does not have the BGP table, it will just load-share across both ASR's. When the traffic hits the ASR's, will they know which carrier has the best path and route accordingly?
  

• Should the iBGP connection between both routers be directly connected ? Or will it suffice through the L3 3750 connection? Also, with the limitations on the routes for the ASR1001 at ~500k. If we end up getting full routes from carriers and create a iBGP neighborship between both routers, will this exceed the route limitations on this platform?
  

• On both routes, I will have the network statement 'network 12.231.69.0 mask 255.255.255.0.' This is a leased network from SONIC, and we NAT everything on our ASA to 12.231.69.10. My question is, will this be a problem broadcasting this network from our AS to both carriers AS?
  

• Refer to bgp-design.jpg - is it a requirement that I use our leased public subnet 12.231.69.0/24 for the interfaces from ASA5510 -> 3750 -> ASR1001?
  

Thank's in advance for any assistance/insight you can provide as this is the most advanced topology I have worked with.

[camdoggle]

Not the most popular thread I see

Anyways, i've progressed my configuration and after researching have decided full routes on the ASR1001 platform should work fine. I found a really good white paper on this example below - I will model my configuration after this example.. It is somewhat dated (Feb.2004) I know the

soft-reconfiguration inbound take up some memory and it's not necessarily needed that much anymore..

Example 2: Single Site, Dual Routers, PA-space, Multihoming to Two ISPs

Background: This example is the same as Example 1, except the customer is using two routers, and

connecting to one ISP on each router. The purpose of this example is to show how iBGP is used to

internally propagate BGP routes between the two routers.

Router1

Router2

! Router 1: Dual-router, PA-space, Basic Config ! autonomous-system 900 ! ip route 198.18.1.0 255.255.255.0 Null0 200 ! ip prefix-list ANNOUNCE description Our External Netblocks ip prefix-list ANNOUNCE seq 10 permit 198.18.1.0/24 ! ip prefix-list BOGONS description Bad Routes to Block In ip prefix-list BOGONS seq 10 deny 0.0.0.0/8 le 32 ip prefix-list BOGONS seq 15 deny 10.0.0.0/8 le 32 ip prefix-list BOGONS seq 20 deny 127.0.0.0/8 le 32 ip prefix-list BOGONS seq 25 deny 172.16.0.0/12 le 32 ip prefix-list BOGONS seq 30 deny 192.0.2.0/24 le 32 ip prefix-list BOGONS seq 35 deny 192.168.0.0/16 le 32 ip prefix-list BOGONS seq 40 deny 224.0.0.0/3 le 32 ! Prevent someone else from announcing your own prefix(es) ! back to you, for security: update this with YOUR ! actual announced block(s)! ip prefix-list BOGONS seq 1000 deny 198.18.1.0/24 le 32 ip prefix-list BOGONS seq 9999 permit 0.0.0.0/0 le 27 ! router bgp 900 no synchronization bgp log-neighbor-changes bgp dampening network 198.18.1.0 mask 255.255.255.0 neighbor 1.1.1.1 remote-as 200 neighbor 1.1.1.1 description BGP Transit to ISP-1 neighbor 1.1.1.1 version 4 neighbor 1.1.1.1 send-community neighbor 1.1.1.1 soft-reconfiguration inbound neighbor 1.1.1.1 prefix-list BOGONS in neighbor 1.1.1.1 prefix-list ANNOUNCE out neighbor 1.1.1.1 maximum-prefix 140000 ! define iBGP session neighbor 198.18.1.2 remote-as 900 neighbor 198.18.1.2 description iBGP to Router2 neighbor 198.18.1.2 version 4 neighbor 198.18.1.2 send-community neighbor 198.18.1.2 soft-reconfiguration inbound ! make sure you are using a Loopback0 /32 on each ! router, and propagating this route internally. ! This is important to the stability of your iBGP ! sessions, so they are not tied to a physical ! interface address. neighbor 198.18.1.2 update-source Loopback0

autonomous-system 900 ! ip route 198.18.1.0 255.255.255.0 Null0 200 ! ip prefix-list ANNOUNCE description Our External Netblocks ip prefix-list ANNOUNCE seq 10 permit 198.18.1.0/24 ! ip prefix-list BOGONS description Bad Routes to Block In ip prefix-list BOGONS seq 10 deny 0.0.0.0/8 le 32 ip prefix-list BOGONS seq 15 deny 10.0.0.0/8 le 32 ip prefix-list BOGONS seq 20 deny 127.0.0.0/8 le 32 ip prefix-list BOGONS seq 25 deny 172.16.0.0/12 le 32 ip prefix-list BOGONS seq 30 deny 192.0.2.0/24 le 32 ip prefix-list BOGONS seq 35 deny 192.168.0.0/16 le 32 ip prefix-list BOGONS seq 40 deny 224.0.0.0/3 le 32 ! Prevent someone else from announcing your own prefix(es) ! back to you, for security: update this with YOUR ! actual announced block(s)! ip prefix-list BOGONS seq 1000 deny 198.18.1.0/24 le 32 ip prefix-list BOGONS seq 9999 permit 0.0.0.0/0 le 27 ! router bgp 900 no synchronization bgp log-neighbor-changes bgp dampening network 198.18.1.0 mask 255.255.255.0 ! define BGP session with ISP-2 (ASN 300) neighbor 2.2.2.1 remote-as 300 neighbor 2.2.2.1 description BGP Transit to ISP-2 neighbor 2.2.2.1 version 4 neighbor 2.2.2.1 send-community neighbor 2.2.2.1 soft-reconfiguration inbound neighbor 2.2.2.1 prefix-list BOGONS in neighbor 2.2.2.1 prefix-list ANNOUNCE out neighbor 2.2.2.1 maximum-prefix 140000 ! define iBGP session neighbor 198.18.1.1 remote-as 900 neighbor 198.18.1.1 description iBGP to Router1 neighbor 198.18.1.1 version 4 neighbor 198.18.1.1 send-community neighbor 198.18.1.1 soft-reconfiguration inbound ! make sure you are using a Loopback0 /32 on each ! router, and propagating this route internally. ! This is important to the stability of your iBGP ! sessions, so they are not tied to a physical ! interface address. neighbor 198.18.1.1 update-source Loopback0 ! end

[Ibrahim Jamil]

Hi Cam

pls can share with us this great white papers,  i m interesting, 5 Points from me

thanks

jamil

[eahmed007]

Hi,

I have configured EBGP between to Internet service provider and configured HSRP with IBGP for High-Availability for Local traffic. In this Network Topology, I have two Firewalls behind Two Routers where I configured EBPG Multi-homing.

ISP01                                 ISP02

     EBGP                             EBGP

Router01         IBGP     Router02

                    HSRP

Firewall01           Firewall02

I have one global network block 105.X.X.X/23. My intention is to use both ISP active for 105.X.X.X/24 and 105.X.Y.X/24 while configuring EBGP Multi-homing with HSRP protocol. That means 105.X.X.X/24 will use one ISP for incoming and outgoing traffic and 105.X.Y.X/24 will use other ISP and fail-over happens in case of one ISP goes down.

Can you tell me how I can configure to achieve this Active/Active High-Availability configuring BGP with HSRP protocol and keeping Firewall behind for LAN Network.

I am looking forward to your assistance.

With Regards

Erfan

[camdoggle]

Here you go:

http://openweb.or.kr/wp-content/uploads/2012/03/BGP-MHing-HOWTO-whitepaper.pdf

[Ibrahim Jamil]

Hi Cam

Thanks a lot cam, VERY VERY Informative Docs,thank y mate, another 5 pts

thanks

jamil

[DuncanM2008]

Hi Cam,

I've actually just done very similar for our DC here in the UK, and it's live so I can give you hopefully a bit of assistance. My config actually looks very similar to your in terms of filtering and BGP config.

A couple of questions:

1) Your to BGP edge routers how will they be connected (physically) to the ASA, I'm guessing from your switch model via a VLAN?

2) You've not tweaked local-preference so I'm guessing you want some element of outbound load-balancing, based on the fact each border router will have two copies of the BGP table (1 via it's direct BGP partner, 2nd via the iBGP peer).

Can the kit handle the in excess of 700k routes / path entries that will be in the FIB / RIB?

Any questions you have for me, feel free to fire away as well.

Thanks,

Dunc.

[camdoggle]

Duncan I'm glad you responded and have a similiar topology!

1.) The BGP edge routers will be connected to the L3 3750 switch, the ASA outside interface is also connected to this switch. I'm planning on redistributing default routes from both BGP routers to OSPF which is running on the ASA.. (bgp is not supported on ASA currently)

2.) Correct, we deffinetely want outbound load balancing on these two circuits. However, you raise a good point, on each router we would have two copies of the BGP table.. How do I find out for sure if my ASR1001's will support this? If there is a hardware limitation we will have to re-evalutate and go with carrier/partial routes plus a default from both Carriers, to limit the size of the RIB.

-----------------------

Router#show memory summary

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)

Processor  7F05B1231010   1217482752   262788048   954694704   954652872   954573868

lsmpi_io  7F05B0B2C1A8     6295128     6294212         916         916         916

------------------------

Dunc - did you decide to take full or partial routes from your carrier?

Thanks,

Cam

[DuncanM2008]

Hi Cam,

1) Are you going to use L3 interfaces to connect to the BGP routers?

2) Makes sense, it's worth noting that outbound load-balancing with two BGP tables isn't very finely controlled, like everything with BGP it depends who's better connected. (might be a bit biased)..

I've tried to look into the ASR1001 IPv4 prefix support but get mixed results, bottom end being 500,000 top-end being about 1 million...

Partial routes could be an option if it is 500,000, partial normally works quite well depending on what partial is (usually geographic (US in your case) + carrier customer routes)

You will end up with two copies if you run iBGP, the iBGP peers will exchange there BGP databases (those downloaded from the respective carriers).

Oh for noteworthy reference, your upstream carrier is AT&T. Also the 12.231.69.0/24 prefix falls as part of a larget AT&T aggregate, do you have an agreement from them to announce this to your other carrier?

I did take full routes from both providers in our case, I specced the routers specifically for this purpose, pair of 7206VXR's with NPE-G2 (2GB of RAM), supports around 3M IPv4 routes per router.

Any more questions let me know.

Time for sleep now (UK GMT), will check again in about 8 hours!

Dunc.

[camdoggle]

Hey,

1.) I will be going directly connected interfaces from DC-Edge-RT1 to DC-Edge-RT2. Since this is my iBGP connection between the two ASR's, can I use a private /30?

Also, for the ASR connections to the L3 switch, I will use the 12.231.69.0/24 space. I believe this is a requirement since my Natting takes places on my ASA, and the two OSPF default routes will be 12.231.69.2 & 69.3.

2.) I have tried to pull up the same information about the ASR1001 and full routes, I get the same mixed reports, anywhere from 500k to 1 million. This is one of my biggest hang ups because It will determine if I get partial/full routes from carriers.

We're in the process of setting up a call with at&t to go over this. I guess i'll just ask them for this agreement and that we want to anounce this IP space to our other carrier.

When you cut over to this topology, is there a way to verify that your ASN is listed for your prefix on a public lookinglass NAP? i.e. http://www.bgp4.as/looking-glasses

Thanks,Cam

[DuncanM2008 · Accepted solution]

Hi Cam,

There is one problem with using 2 x /30's between your edge routers and the 3750 -> ASA, when your iBGP session advertises routes to the other peer they will be blackholed / unreachable.

For example say DC-Edge-RT1 (12.231.69.2)  advertises a route for 208.67.0.0/16 to DC-Edge-RT2 via iBGP, DC-Edge-RT2 will know that 208.67.0.0/16 is via 12.231.69.2. To forward a packet to 208.67.0.0 DC-Edge-RT2 will lookup the route to 12.231.69.2 and will find a route via OSPF which is via the 3750.

DC-Edge-RT2 will then spit out a packet destined for 208.67.0.0/16 out the interface to the downstream 3750, this is where the problem occurs, the 3750 doesn't have a clue how to get to 208.67.0.0/16 because unlike the BGP routers it doesn't have the full table. Well it would use the default routes but you have two of those so that's undesirable.

A solution could be (same one I used):

Setup a HSRP group between your two edge routers so they share a virtual address say: 12.231.69.2/29, give RT1 12.231.69.3 for it's physical address and RT2 12.231.69.4 for it's physical address. Then create a Vlan on your 3750 with a Layer 3 SVI with an IP of 12.231.69.5.

Just need one default route then: 0.0.0.0 0.0.0.0 12.231.69.2

iBGP will still do some amount of load-sharing depending on the information in the various BGP tables...

If and when you get this up and running you can view the BGP path (and thus your AS Number) from the following LG:

http://lg.he.net/

Do a bgp route query for your prefix, will show the AS number and AS-PATH

Any questions, holla! 

Oh, just a thought, if there's no requirement for your 3750 to be Layer 3, then you could just create the same HSRP group but put both your Edge routers & your ASA in the same Layer 2 VLAN with no SVI and then set your HSRP group as the Default Gateway for your ASA's WAN interface.....

Dunc.

[Ibrahim Jamil]

HI Cam

pls can y share ur configuration between DC-Edge-RT1 DC-Edge-RT2 .ASA and the 3750

thanks

jamil

[camdoggle]

Hey Dunc,

Your example makes sense. Per the HSRP recommendation, do you have a configuration example for the active/active group setup? For some reason I always think of HSRP as a active/standby mechanism. This achieves the same purpose that I was going for. My objective is to avoid multiple static default routes on the ASA. If a router dies, say 69.2, the static does not get removed..Creating an unreachable path. 

However with your design tip with HSRP, I can keep the ASA with a single static route pointing to the HSRP virtual address. I guess GLBP could work as well.

There is no requirement for a L3 setup on the 3750.. Is there a benefit going L2? Thanks for the link to the LG site, I browsed around and that is powerful tool.

Ibrahim - I will post the configs when I get back to work tomorrow

Thanks,

Cameron

[DuncanM2008]

Hi Cam,

Sadly HSRP will always be active / passive but in this istance the only reason it's used is so you only need a single default gateway / default route.

Unfortunately GLBP would also work in an active passive form, as you're ASA only has a single MAC address it would use the same AVF all the time.

That said though, even with HSRP you'll get a variety of active passive setup for example:

Say you had a packet destined for 210.50.0.0/16 and RT2 had the best route in BGP terms and RT1 was the Active HSRP.

Your ASA would forward the packet to RT1 (HSRP Active), RT1 would then look up 210.50.0.0/16 and as long as iBGP is configured correctly (local-preference) it will then forward the packet via RT2.

It isn't as cut and dry as Active/Active so to speak, but it is a form of load-sharing in outbound direction (depending on how well-connected each BGP Peer is, if one's better connected you might get an unequal bias).

Hope this helps.

Thanks,

[Ibrahim Jamil]

Hi Cam

pls can y share ur configuration between DC-Edge-RT1 DC-Edge-RT2 .ASA and the 3750, i m going with similar setup but with 2 ASAs

thanks

jamil