BGP Filtering causing Regional Outage

mn.nathaniel · ‎09-17-2014

Started running out of memory on our Catalyst 6906 due to large BGP route tables. Talked with our ISP (CenturyLink) and they recommended that I use BGP route filtering via the filter-list command. Here is the physical layout:

We have 5x 1Gb/s links to CenturyLink that we use BGP to load balance. We then have a 10Gb/s that we use to communicate to each router. From there, we have multiple links (condensed in the pic) to our firewalls and load balancers.

Here is the BGP output:

Router 1

router bgp 22156 bgp log-neighbor-changes network 209.67.100.0 mask 255.255.252.0 neighbor 64.85.91.129 remote-as 3561 neighbor 64.85.91.129 description Century Link 103 neighbor 64.85.91.129 version 4 neighbor 64.85.91.129 soft-reconfiguration inbound neighbor 64.85.91.129 prefix-list pl-bgp-allow-in in neighbor 64.85.91.129 prefix-list pl-bgp-allow-out out neighbor 64.85.92.97 remote-as 3561 neighbor 64.85.92.97 description Century Link 104 neighbor 64.85.92.97 version 4 neighbor 64.85.92.97 soft-reconfiguration inbound neighbor 64.85.92.97 prefix-list pl-bgp-allow-in in neighbor 64.85.92.97 prefix-list pl-bgp-allow-out out neighbor 192.168.183.66 remote-as 22156 neighbor 192.168.183.66 description router2 neighbor 192.168.183.66 update-source Loopback0 neighbor 192.168.183.66 version 4 neighbor 192.168.183.66 next-hop-self neighbor 192.168.183.66 soft-reconfiguration inbound neighbor 192.168.183.66 prefix-list pl-bgp-allow-in in neighbor 209.67.111.129 remote-as 3561 neighbor 209.67.111.129 version 4 neighbor 209.67.111.129 soft-reconfiguration inbound neighbor 209.67.111.129 prefix-list pl-bgp-allow-in in neighbor 209.67.111.129 prefix-list pl-bgp-allow-out out maximum-paths 8

Router 2

router bgp 22156 bgp log-neighbor-changes network 209.67.100.0 mask 255.255.252.0 neighbor 192.168.183.65 remote-as 22156 neighbor 192.168.183.65 description router1 neighbor 192.168.183.65 update-source Loopback0 neighbor 192.168.183.65 version 4 neighbor 192.168.183.65 next-hop-self neighbor 192.168.183.65 soft-reconfiguration inbound neighbor 192.168.183.65 prefix-list pl-bgp-allow-in in neighbor 209.67.68.97 remote-as 3561 neighbor 209.67.68.97 description Century Link 102 neighbor 209.67.68.97 version 4 neighbor 209.67.68.97 soft-reconfiguration inbound neighbor 209.67.68.97 prefix-list pl-bgp-allow-in in neighbor 209.67.68.97 prefix-list pl-bgp-allow-out out neighbor 209.67.111.133 remote-as 3561 neighbor 209.67.111.133 version 4 neighbor 209.67.111.133 soft-reconfiguration inbound neighbor 209.67.111.133 prefix-list pl-bgp-allow-in in neighbor 209.67.111.133 prefix-list pl-bgp-allow-out out maximum-paths 8

Here are the prefix-lists which are same on both routers:

ip prefix-list pl-bgp-allow-in deny 0.0.0.0/8 le 32 ip prefix-list pl-bgp-allow-in deny 10.0.0.0/8 le 32 ip prefix-list pl-bgp-allow-in deny 169.254.0.0/16 le 32 ip prefix-list pl-bgp-allow-in deny 172.16.0.0/12 le 32 ip prefix-list pl-bgp-allow-in deny 192.168.0.0/16 le 32 ip prefix-list pl-bgp-allow-in deny 224.0.0.0/3 le 32 ip prefix-list pl-bgp-allow-in permit 0.0.0.0/0 ge 1

ip prefix-list pl-bgp-allow-out permit 209.67.100.0/24

This is the code I added per looking up the commands from Cisco:

Router 1

ip as-path access-list 80 permit ^$ ip as-path access-list 85 permit ^3561_[0-9]*$ ip as-path access-list 86 permit ^22156_[0-9]*$

router bgp 22156 neighbor 192.168.183.66 filter-list 80 out neighbor 192.168.183.66 filter-list 86 in neighbor 64.85.91.129 filter-list 80 out neighbor 64.85.91.129 filter-list 85 in neighbor 64.85.92.97 filter-list 80 out neighbor 64.85.92.97 filter-list 85 in neighbor 209.67.111.129 filter-list 80 out neighbor 209.67.111.129 filter-list 85 in

Router 2

ip as-path access-list 80 permit ^$ ip as-path access-list 85 permit ^3561_[0-9]*$ ip as-path access-list 86 permit ^22156_[0-9]*$

router bgp 22156 neighbor 192.168.183.65 filter-list 80 out neighbor 192.168.183.65 filter-list 86 in neighbor 209.67.68.97 filter-list 80 out neighbor 209.67.68.97 filter-list 85 in neighbor 209.67.111.133 filter-list 80 out neighbor 209.67.111.133 filter-list 85 in

When I added that, everything seemed to do exactly what I wanted. We saw our BGP route routes go from 500K+ down to 107k. We saw RAM utilization get cut in half. I did all sorts of connectivity tests and everything seemed to be good. Commited it and walked about.

2 hours later I start getting calls and alerts. Out clients in the SE US can't connect to us. Other clients are just fine, and our monitoring show us as up, except for our Florida and Virginia sites. I do some more tests and find I can ping some places in the SE, and not others. Did not have a lot of time to troubleshoot, so I rolled back the BGP and within 10 minutes the issue went away.

I know that I did not get enough troubleshooting in, but can anyone review what I have and see if it is something blatant that I am missing? I just want to cut down the BGP routes as we are running out of memory and from what I understand, the commands I put in there should only get the routes that are directly connected to our BGP neighbors.

Thanks!

Peter Paluch · ‎09-18-2014

Hello,

This is an interesting issue. Unfortunately, because of the short time you had to do any sensible troubleshooting, we only have very limited information so there's little to hang on right now.

I am thinking - I see you have the soft-reconfiguration inbound configured for your BGP neighbors. Are you absolutely sure you need it? This feature is an ancient workaround for an old BGP deficiency (the inability to ask a neighbor to resend its routes after an inbound filter has changed) that causes your router to maintain an unfiltered database of all BGP routes your neighbor sent you in addition to the prefixes your inbound policy has decided to accept. As a result, doing any kind of aggressive inbound filtering with the Soft Reconfiguration activated is not really going to save your memory because the unfiltered prefixes still have to be kept in your router's RAM. All modern BGP implementations have solved this problem with additional BGP message called Route Refresh (see RFC 2918 for more details, and search for Route Refresh on Cisco Support Community too - I have discussed this feature often in the past) that has been supported for many years. I am sure your routers support the Route Refresh feature (if you need to be absolutely sure, please check the output of show ip bgp neighbor .

So while not directly answering your question about what could have gone wrong with your changes, I am posing another question: do you believe you could deactivate the Soft Reconfiguration for your BGP peers? It should alone lower the memory footprint of your BGP process - and if you are receiving a full BGP feed, memory savings should be very noticeable.

Best regards,
Peter

mn.nathaniel · ‎09-18-2014

Peter,

Thanks for the response. I have verified that our routers do support the Route Refresh and we are running BGP v4 on all our neighbors. This was more than likely a left-over command from previous iterations and I was just not aware enough to catch that. Thanks for that. I will verify that I can turn that off and post back shortly.

And yes, while it does not provide the answer for the weird routing, it could provide an overall solution without the need to do filtering. If I can make the change and it reduces my memory footprint, I will mark yours as the answer and call it a day on it.