Data Centre BGP Design

[de1denta]

Hi All,

I'm looking to add a second data centre to our network for DR purposes and I need to design the Internet connectivity so I can failover the public IP addresses to the DR site when required. See diagram.

We currently have 2 internet edge routers which use eBGP to 2 ISPs, iBGP is configured between the edge routers. We have our own ASN and IP prefix and we prepend the ASN over ISP2 to prefer the route via ISP1 

We are looking to add a single internet edge router in the DR data centre and again use eBGP to the 2 ISPs. We will then advertise the primary prefix out from the DR data centre and prepend ASN again to to make this route less prefered then the primary data centre.

For this configuration to work, will I need to use iBGP between the DR data centre router and the Primary? If so will I need to use a dedicated link between the edge routers or can I connect them via Internet. Also any throughts of using a Cisco 3750 as the edge router in the DR data centre? We will only be receiving default routes from the ISPs.

Thanks

[Marwan ALshawi]

Hi

from a high level point of view this is doable design

but there are some points you need to test, discuss and take it into your considerations

i am assuming the DR wil be just a redundant/Back up services that dose not need to communicate with the primary DC and they will be use din the case of the primary DC is down if this is correct then you do not need a link with iBGP between the two DCs

if the services they require some communications, like backup, replications ..etc a link between  those two data center it recommended to avoid routing a larg amount of traffic over the WAN

however, adding a link between the two DCs might lead to advertise the primary DC over the DR link when the eBGP of the primary is down and this will lead to make the primary DC comunicate with the Internet over the inter DC link ( some BGP policies can be configured to overcome this issue )

if you are going to use same BGP AS number or diffrent AS number you will need to make sure that both ISPs accept AS prepedning, the advertisement of the IP range you have over all links and confirm with each ISP regarding which link, to be preferred within thier cloud as some ISP they use some BGP community stings significant within the ISP cloud and their customers and might lead to some sub optimal routing ( discuss it with each ISP )

let me know if you need anymore details about any of the points above

hope this help

if helpful Rate

[Konstantin Dunaev]

if you'd like to use the second DC for  completly separate services, then may be it's better to get an additionl AS for it.

If you'd like to have all DC's in the same AS you'll need to interconnect both DC via iBGP, you can do it withj help of dedicated link or you can look int some kind of GRE tunneling, if it's supported on your hardware.  And you should be carefull in advertising of networks from differemt DC - you want that incoming traffic goes to coreect DC and router.

[de1denta]

Hi,

There will be a dedicated fiber link between the core switches in each DC which will be used for replication traffic etc.

The second data centre will be used for test and dev (seperate dev IP address range) but I would like to failover the primary data centre IP address to the backup data centre in the event of a large failure. All of the servers will be in a VMware enviroment so they will be started up in the backup data centre.

If I need iBGP between data centres, its is advisable to use the dedicated fiber between the intenal core switches (using a dedicated VLAN) or setting up a GRE tunnel over the internet between the edge routers? I dont think I will have buget to have 2 dedicated links between data centres.

To avoid any routing issues, I suppose we could make this a manual process by advertising the live IP address range on the backup data centre router in the event of a failure?

Thanks all

[Marwan ALshawi]

if you are advertising differnt range then this is not a DR case anymore

however if you plaining to faiover the VM environment over the L2 link in this case you can have automated failover rather than manual by using the L2 vlan used over the fiber as L3 interface and exchange routing over it

by the way you talking about a failover from ESXi and VM point of view where the VMs will be moved using DRS/vmotion for example from the main DC to the DR

while you might have a situation where the routers, links or ISPs have an outage which will cause only the Internet link to go down and here where you can take the advantage of runing iBGP between the two DCs and advertise the Primary DC over the DR eBGP peering for backup reasons

hope this help

[Konstantin Dunaev]

I can't say much about ESX vmotion ( i had in the past a little bit experience but so much).

For iBGP I'd suggest to use to dedicated link with  two separate VLANs  to connect both routers and a GRE tunnel as a backup for the dedicated link.

If your dedicated link  is a "flat" and very fast then actually you don't need to think much about the routing, it doesn't really metter how the traffic comes in and out, if for example, the latency between two DC is about 5 ms. 

But I don't recomend to rely on manual process - it's better to sit  and plan a dynamic failover routing schema carefully.

[de1denta]

Hi Guys,

Thanks for your comments, very helpful.

The Virtual Machines will only move to the DR data centre during a complete outage, this will happen using VMware Site Recovery Manager. This means that we will have a small amount of time to move the live IP address range to the DR site whilst the Virtual Machines are coming up. We dont need instant failover.

That is a good point regading an ISP outage in the primary data centre. We wont want to initiate DR just because the ISPs are down if we can help it, so routing the live IP address range via the backup dc and over the fiber link using iBGP to the prmary will be good.

In order to connect the edge routers using iBGP, it is advisable to use the flat fiber connection between the internal core switches by creating dedicated VLANs on the core switches and then directly connecting a port on the core to the edge routers, this will place the iBGP routers in the same broadcast domain and we can peer that way? Or we can connect the edge routers between sites using an IGP over the internal network and GRE tunneling?

Thanks

[Konstantin Dunaev]

willscotty2012 wrote:
In order to connect the edge routers using iBGP, it is advisable to use the flat fiber connection between the internal core switches by creating dedicated VLANs on the core switches and then directly connecting a port on the core to the edge routers, this will place the iBGP routers in the same broadcast domain and we can peer that way? Or we can connect the edge routers between sites using an IGP over the internal network and GRE tunneling?

exactly, it's a good idea to connect the dedicated link to the core switch, but I'd suggest to use 2 separate VLANs for connecting between DC's, first it will be easier to manage the traffic flow (costs, filter and so on), second it's a protect from problem inside of a VLAN.

The iBGP router shouldn't be in same broadcast domai in order to get  "connected". But if you'd like to be 100% sure how the traffic is flow between BGP routers it's better to connect them directly.

[de1denta]

I have just been told that GSS might be a better solution here.

GSS will respond with the primary data centre IP address in its DNS replies during normal operation, and the backup data centre IP address in the event that the primary site is down.

Has anyone had experience/problems with using GSS in a active/standby scenario?

[Marwan ALshawi]

It depends what type if services you use

GSS us commonly used in scenarios like yours but keep on mind that if you have GSS located in one DC you will have same issue reaching the GSS

You need to gave two redundant GSS one in each DC

Two hosts dns records in the ISP for failover one point to the primary GSS and the other failover dns record point to the secondary GSS

Or you can have one GSS in the main DC but by using some routing between the two DCs you have it reachable via both ISPs and DCs

The other option you have two loadbalancers in each DC and use the dns concept described above

Cisco ACE is one option

Hope this help

Sent from Cisco Technical Support iPhone App

[de1denta]

Thanks for the reply. Have you had experience with GSS? Which solution, BGP routing or GSS, in your opinion would be more robust, easier to manage etc?

Thank you

[Marwan ALshawi]

well the is depends on your needs and equipments

if you have loadbalncers in both sites GSS is a god option to go with

if not

routing can do but you need to consider all the failover scenarios, server, VM, ISP, router, Link .etc

and address all of these cases with your design

i believe the easy option is routing by using some BGP polices, however you need to sort out the IP range advertisement issue so that you can smoothly advertise and failover to the DR DC

HTH