Failover via BGP did not work during carrier maintenance

[shartley]

We have three ISP's connected to two routers. ISP1 has a 3gb circuit to our WAN router 1. ISP2 and ISP3 are connected to WAN router 2 (both 1gb).

We have BGP to our ISP's; iBGP between the two WAN routers 1/2; and OSPF between the four core devices (two WAN routers and two Core switches A/B).

The BGP local pref is set higher (100) for the connection to ISP1 (3gb) than to ISP1/2 (80) and we see all outbound traffic taking ISP1. Inbound the load is shared amongst all three ISP’s.

In WAN router 1 we see the BGP neighbor drop for ISP1 when the vendor started maintenance and then the OSPF neighbors drop; that’s expected. The unexpected part is that all data traffic seemed to stop; even on the ISP2 and ISP3 links via WAN router 2.

The log in WAN router 2 only shows the OSPF neighbor to WAN router 1 drop. It was still peered to the ISP routers via BGP. It still had a OSPF neighbor connection to the Core switch B and that’s an OSPF neighbor to our Core switch A.

I was not able view the route table in WAN router 2 at the time of the outage so I cannot confirm what routes we were able to see at that time.

Our question is could the issue be the design or if not, what configuration components should we focus on?

[Giuseppe Larosa]

Hello Shartley,

>> The unexpected part is that all data traffic seemed to stop; even on the ISP2 and ISP3 links via WAN router

WAN2 router should have injected a default route in OSPF to be used by CoreA and CoreB, If I understand the outage caused WAN1 router to become isolated from the network and also to tear down the eBGP session on the 3Gbps link.

In normal conditions WAN1 router generates a default route in OSPF to be used by CoreA, CoreB

I would focus on this section of configuration if and how OSPF is configured to generate a default route on WAN2 router.

Using O E1 type of default route is preferred in your case, because there are two exit points.

WAN1 router using a lower seed metric (50 for example) may be the preferred exit point mirroring the hierarchy  built with local preference in BGP. WAN2 router should generate a worse default route with an higher seed metric (500 for example)

Hope to help

Giuseppe

[shartley]

WAN router 2 (and WAN router 1) has a"default-information originate" statement in OSPF.

The Core A/B switches also have static routes pointing to the loopback of the WAN routers (equal cost). so in the Core switches we see:

Gateway of last resort is (WAN router loopback) to network 0.0.0.0

Both WAN routers have this statement configured in OSPF:

redistribute static metric-type 1 subnets route-map static_to_ospf

no metrics have been modified in either WAN router (all default) but I understand your recommendation.

[shartley]

In our WAN routers I do not see a default route in OSPF; even though the "default-information originate" is configured in ospf. When I "show IP route" the "gateway of last resort is not set"...when I do "show ip ospf database" I don't see a entry in the type-5 external link states for the default route. Nor do I see it when I "show ip route ospf" or "show ip ospf databse external 0.0.0.0".

I'm not sure why this would be with the ospf "default-information originate" configured?

[shartley]

ignore my comment about the default gateway in the WAN routers. We are receving the full route tables from our ISP's and not just default routing to the vendor router(s).

[Giuseppe Larosa]

Hello Shartely,

if there is no default route in IP routing table of WAN1,WAN2 routers the command default-information originate does not work as it checks if there is a default route in local node IP routing table before creating the LSA type 5 for network 0.0.0.0/0.

The command default-information originate always

would work as it would generate the default route regardless of the presence of a default route in the IP routing table of the local node ( I mean WAN1,2 router).

This is the reason why there are the default static route pointing to WAN router loopback address on core routers.They have been introduced as a fix to the lack of an OSPF default route injected at WAN1,2 routers.

Now, the question becomes that routing is not dynamic in upstream direction and the default route might be not removed from IP routing table when the OSPF neighborship fail.

The static route has an IP next-hop of the loopback of the node, and that route should be provided by OSPF in WAN1 router LSA. However, this should be tested.

If the static route is not removed when WAN1 is isolated there is an issue.

I would suggest   to add the keyword always to the default originate command on WAN1,2 routers under router ospf, and to remove the static default routes pointing to loopbacks on CoreA/CoreB .This should make routing dynamic for sure  in upstream direction.

Hope to help

Giuseppe

[shartley]

Yes, this is very helpful! I need to discuss with some of my peers in other sites and we will probably be making the changes you recommended very soon,