BGP Routing Design Considerations

vdadlaney · ‎07-28-2009

Hi, attached is a diagram of the topology that I am looking to build. A summary of the peering relationships are

1. eBGP between R1 and R3

2. eBGP between R2 and R4

3. iBGP between R1 and R2

4. iBGP between R3 and R4

The firewalls are acting as active/passive with the left leg of the design being the active leg always (R1-FW-R3 Leg is Active Leg). There's a state sync dedicated link between the firewall that is not shown on the diagram however that is not used from a routing perspective at all for normal traffic forwarding.

There are some issues that I am facing with this design. They are

1. Since its the same L2 hops between the internal and external switch if the Primary firewall fails than the eBGP on the Primary leg establishes via the redundant backup firewall and this leads to somewhat suboptimal routing because in either direction traffic always comes to the primary switch than goes across the l2 Trunk link to the backup switches and than on its way to the other side. Any suggestions on getting around this. A solution was to peer with Loopbacks and point to the physical next hop of the FW however this causes an issue during normal when everything is working operation because now the second leg sees the BGP next hop as the interface of the second leg switches and hence since the next hop for those switches is configured to go to the physical ip of the firewall it never uses the VRRP. Not sure if there is a fix for this issue.

2. iBGP under normal circumstances is not redistributed into an IGP. So if for ex another routing protocol is being used and say if the Primary leg Internal switch (R3) goes down than since the connection to the customer is still via the Primary Leg External Switch 1 the iBGP learnt route from R2 is never sent across to the the customer. The solution used over here was to originate the route on the external switches however am considering whether it would make sense to use cross eBGP peering between external and internal meaning having a eBGP peering between R1 and R4 and R2 and R3. Are there any downsides to that? I cannot think of any except possibly overhead on the devices with the second eBGP peering? What considerations should be taken into account in deciding if this peering relationship is viable?

3. VRF-Lite is being on the External side to isolate each customer. In addition RT's are being to exchange routes from the customer VRFs to the FW vrfs. This is only on the external side. Does anyone see any downsides to this approach. I haven't seen any issues and had posted in the past a similar question however just want to confirm again.

johnnylingo · ‎07-28-2009

Answer to #2.

If I'm understanding correctly, your concern is that if R3 goes down, R1 will still be HSRP primary and therefor the traffic BGP Hops will be R1 -> R2 -> R4. You want it to just go R2 -> R4 or R1 -> R4.

First off, there is a simple way to do this - just setup a monitor on R1 and tie that to the HSRP priority. If router R3 dies, SLA monitor drops HSRP priority and R2 takes over as primary. Problem solved.

Back to BGP...Personally, I wouldn't worry too much about this, because the traffic has to go through R2 anyway. Doing a peering relationship between R1 and R4 / R2 and R3 may improve convergence times, but results in more sessions to manage and complexity on the configuration. You can help to simply things by using peer-groups and peering with loopback addresses so that the configuration within each AS is the same.

I think there are two questions you should be asking if you haven't done already:

1) How many routes do you expect each router to be receiving?

2) What is the SLA? How fast must re-convergence occur in the event of a failure?

3) Who controls the Firewalls and is running BGP or and IGP on them an option?

johnnylingo · ‎07-28-2009

Another thing I just thought about...

If R1 only peers with R3, it will only go through R2 in the event of a failure.

If you configure R1 to peer with both R3 and R4, it's going to take whichever router has the lowest IP address. You may not want this behavior, and therefore will want to use local pref, metric, or weight to control primary/secondary routes. This further adds to the complexity.

vdadlaney · ‎07-28-2009

Hi John,

Thx for replying. The problem I was trying to overcome on Point # 1 was that I wanted traffic to converge optimally hence if there was a failure on the Primary Firewall than all traffic should only come down the second leg however realized that the link to the customer is still active on R1 hence it will always logically traverse Customer-R1-R2-FW2-R3 and return traffic would logically traverse via Internal-R3-FW2-R2-R1-Customer. I would have liked if it was going along a straight line path however for that to happen we would have to get

a. Multiple Circuits within the same site (Expensive in this day and age)

b. Run Routing with the firewall which isn't an option as of now.

After quite a bit of thinking I came to the realization that the fact that the eBGP peering does not fail when the Primary Firewall fails is actually a blessing in disguise as it helps me with point number 2 as well which I have explained below.

The problem with point number 2 I was trying to solve was I wanted to originate the route that was getting advertised to customers on the Internal Routers (R3 and R4) and they would advertise it to their individual respective eBGP peers R1 and R2. The reason I was trying to do that was to solve a blackhole issue with R3 so in essence what I was doing was creating a static route for the network pointing out via the links to the internal core and than advertising this into eBGP to the external R1 device. However I realized that the links to customers are still connected to R1 hence do I really want to stop advertising that route if the customers because an internal router in my network has an issue. I could solve these issues somehow with conditional advertisement etc but again that will add a bit of complexity and will have to be tested etc. The failure that would happen would be Assume that customer link is only active on R1. Than assume that R3 fails completely. Now that route I want to advertise R1 will learn via iBGP from R2 hence if I am running a Routing Protocol other than BGP with the customer than that route being a iBGP route will not get redistributed normally (unless I use bgp redistribute internal which I do not want to use). Not sure how HSRP comes into play over here. Unless the customer is using static routes pointing towards my networks HSRP would not come into play. Not sure if u meant something else hence can u please elaborate on that.

As you mentioned the added configuration and complexity of creating multiple peering relationships is not absolutely essential since the routes being advertised out are not that many. If there were multiple routes than that could have been explored. The fix that I did for this issue was instead of originating the route on the internal side I put a static route pointing out the interface and next hop to the FW on the external side and than just did a network statement under BGP on the external side. Now the customer will always receive my route as long as the interface is up.

The still did not solve the blackhole situation with R3 when it gets isolated from the Internal Network and its neighbor R4 device. The reason for that is since that eBGP peer between R1 and R3 is up it will continue sending the routes from the vendor to R3 however the routes won't get redistributed into the network so outgoing is fine however incoming traffic from customer to me would still be a problem since the VRRP on the firewall would still be active hence the blachole. Again the fix for this issue would be to

a. run a routing protocol with the firewall

b. run some object tracking to track a prefix on the internal core and if unavailable than point to the other firewall but this would not make the firewall failover. hence i think running a routing protocol with FWs would be optimal.

I hope I got everything. Please let me know if you notice any issues with the situations that I have mentioned above. Thx

johnnylingo · ‎07-28-2009

> The still did not solve the blackhole situation with R3 when it gets isolated from the Internal Network and its neighbor R4 device.

Yes, this is one of the most common design problems with BGP. If the session with the external neighbor is up but the router has lost connectivity to the internal network, how do you stop announcing the route?

The easiest way to do this is move the origin off the networks off the edge routers to the core routers. Then either run iBGP to the cores or redistribute from an interior protocol. This way, if internal connectivity is lost, the edge router ceases announcing to the eBGP peer.