BGP Multihoming & single Router, load sharing and automatic fail over

[gasparmenendez]

hi folks,

after been waiting for a while finally I'm ready to have BGP with my 2 ISP's...right now I have this configured on my ASR:

router bgp 2625
 bgp log-neighbor-changes
 network 170.X.X0.0 mask 255.255.252.0
 neighbor 208.Y.Y.181 remote-as 35498
 neighbor 208.Y.Y.181 password 7 151E66
 neighbor 208.Y.Y.181 route-map L3 out
!
ip prefix-list L3 seq 10 permit 170.X.X0.0/22 le 27
ip prefix-list L3 seq 15 deny 0.0.0.0/0
!
route-map L3 permit 10
 match ip address prefix-list L3
!


Contencion1001-X#sh ip bgp summary
BGP router identifier 208.Y.Y.182, local AS number 2625
BGP table version is 2508974, main routing table version 2508974
655803 network entries using 162639144 bytes of memory
655803 path entries using 78696360 bytes of memory
103801/103751 BGP path/bestpath attribute entries using 25742648 bytes of memory
89989 BGP AS-PATH entries using 4634324 bytes of memory
4619 BGP community entries using 575242 bytes of memory
48 BGP extended community entries using 1344 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 272289062 total bytes of memory
BGP activity 684995/29188 prefixes, 705406/49603 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
208.Y.Y.181  4         35498  563553    3349  2508734    0    0 1d01h      655802
Contencion1001-X#

and it's working fine, but as you can see I still don't establish BGP with my second ISP. What I want to achieve is the following (after establish BGP session with my second ISP):

want to split my whole public network (/22) in two /23 to make load-sharing
want to pass 170.X.X0.0/23 inbound and outbound traffic via LVL3
want to pass 170.X.X2.0/23 inbound and outbound traffic via MTC

both routes must be active at the same time
in case of failure of any one of the ISP's, traffic must be automatically routed via the other one

things to be considered:

I have a whole block of public IP addresses 170.80.240.0 /22
ISP LVL3 is sending me the full routing table and ISP MTC only default-route

I found this post: https://supportforums.cisco.com/t5/wan-routing-and-switching/bgp-multihoming-with-2-isp-on-single-router/td-p/2722860which is very similar to what I want to do and tried to adjust to my specific scenario, but since my network is in production I need to be very carefull before make any changes This is what I think I need to set up:

router bgp 2625
 bgp log-neighbor-changes
 no synchronization
 no auto-summary
 network 170.X.X0.0 mask 255.255.254.0
 network 170.X.X2.0 mask 255.255.254.0
 neighbor 208.Y.Y.181 remote-as 35498
 neighbor 208.Y.Y.181 desc LVL3
 neighbor 208.Y.Y.181 password ********
 neighbor 208.Y.Y.181 route-map L3 out
 neighbor 208.Y.Y.181 route-map UPSTREAM1 out
 neighbor 208.Y.Y.181 prefix-list FULL in
 neighbor 187.Z.Z.112 remote-as 1933
 neighbor 187.Z.Z.112 desc MTC
 neighbor 187.Z.Z.112 password ********
 neighbor 187.Z.Z.112 route-map MT out
 neighbor 187.Z.Z.112 route-map UPSTREAM2 out
 neighbor 187.Z.Z.112 prefix-list DEFAULT-ONLY in
!
!
interface TenGigabitEthernet0/0/0
 description *** ISP Level3 ***
 ip address 208.Y.Y.182 255.255.255.252
!
interface TenGigabitEthernet0/0/1
 description *** ISP Metrocarrier ***
 ip address 187.Z.Z.113 255.255.255.254
!
interface TenGigabitEthernet0/1/0
 description *** ASA 5580 ***
 ip address 170.X.X0.1 255.255.255.240
 ip policy route-map OUTBOUND-INTERNET
!
!
ip prefix-list DEFAULT-ONLY seq 10 permit 0.0.0.0/0 =======> should I set another one for the FULL table ????
!
!
ip prefix-list L3 seq 10 permit 170.X.X0.0/22 le 27
ip prefix-list L3 seq 15 deny 0.0.0.0/0
!
route-map L3 permit 10
 match ip address prefix-list L3
!
!
ip prefix-list MT seq 10 permit 170.X.X0.0/22 le 27
ip prefix-list MT seq 15 deny 0.0.0.0/0
!
route-map MT permit 10
 match ip address prefix-list MT
!
!
ip access-list extended LVL3
 permit ip 170.X.X0.0 0.0.1.255 any
ip access-list extended MTC
 permit ip 170.X.X2.0 0.0.1.255 any
 
!
ip sla 1
 icmp-echo 208.Y.Y.181 source-interface TenGigabitEthernet0/0/0
 frequency 30
ip sla schedule 1 life forever start-time now
ip sla 2
 icmp-echo 187.Z.Z.112 source-interface TenGigabitEthernet0/0/1
 frequency 30
ip sla schedule 2 life forever start-time now
!
track 1 ip sla 1 reachability
track 2 ip sla 2 reachability
!
route-map OUTBOUND-INTERNET permit 10
 description send 170.X.X0.0/23 out Level3
 match address LVL3
 set ip next-hop verify-availability 208.Y.Y.181 10 track 1
!
route-map OUTBOUND-INTERNET permit 20
 description send 170.X.X2.0/23 out Metrocarrier
 match address MTC
 set ip next-hop verify-availability 187.Z.Z.112 20 track 2
!
ip access-list standard bgp-up1
 permit 170.X.X0.0 0.0.1.255
!
ip access-list standard bgp-up2
 permit 170.X.X2.0 0.0.1.255
!
route-map UPSTREAM1 permit 20
 match address bgp-up1
route-map UPSTREAM1 permit 40
 match address bgp-up2
 set as-path prepend 47117
!
route-map UPSTREAM2 permit 20
 match address bgp-up2
route-map UPSTREAM2 permit 40
 match address bgp-up1
 set as-path prepend 47117
!

Can anybody help me please and tell me what I need to modify in order to achieve my gold??

Thanks in advance.

[Georg Pauwen]

Hello,

I might be missing something here, but where is your second ISP ? I see only one neighbor configured...

[gasparmenendez]

sorry George, right now I have configured BGP only with one ISP, because I'm not sure if once I establish the BGP session with the second ISP it will affect my internet connection. I'm aware that the first thing I need to do is to run BGP with the second ISP, but is there any chance that all my traffic goes down ???

[Georg Pauwen]

Hello,

are you going to receive BGP routes from your ISPs or are you using static routing ?

[gasparmenendez]

I'm going to receive routes from both ISP's: ISP LVL3 is sending me the full routing table and ISP MTC only default-route...

Thanks.

[Richard Burts]

I would think that there is very small chance that all of your traffic goes down (with a very significant configuration mistake it might happen but that is not likely). It is more likely that traffic may not flow in quite the way that you intend when you bring up BGP with the second ISP. I do have a couple of comments about the suggested config that you posted:

- in that config you have two route maps configured for each BGP neighbor. that will not work. You need to work out the route map logic so that there is a single route map for each neighbor.

- when you bring up BGP with a second ISP one thing you must protect against is becoming a transit network between the ISPs. This would happen if you learn routes from one neighbor and then advertise those routes to the other neighbor. This is the default behavior when you are running EBGP with two ISP. So you need to implement logic so that your outbound advertisements are only advertising your own networks and not advertising routes that you learned from the other ISP.

HTH

Rick

[gasparmenendez]

sorry folks, I've been out of the office the last couple of days...right now I'm back and first thing was bring up BGP with my second ISP, but is not working as I think it should...take a look:

Contencion1001-X#sh ip bgp summary
BGP router identifier 208.X.X.182, local AS number 2625
BGP table version is 5682942, main routing table version 5682942
655785 network entries using 162634680 bytes of memory
655785 path entries using 78694200 bytes of memory
103837/103808 BGP path/bestpath attribute entries using 25751576 bytes of memory
90064 BGP AS-PATH entries using 4602384 bytes of memory
4596 BGP community entries using 574078 bytes of memory
50 BGP extended community entries using 1428 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 272258346 total bytes of memory
BGP activity 833221/177421 prefixes, 954470/298685 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd     MsgSent   TblVer         InQ OutQ Up/Down  State/PfxRcd
187.Z.Z.112     4        139      27         31            5682936    0    0      00:23:23        1
208.X.X.181     4        349    1572095   19816    5682942    0    0        6d07h      655782
Contencion1001-X#

as you can see there are not prefixes through 187.Z.Z.112, so I need to know: even so, can I do load-sharing and force traffic through 187.Z.Z.112 ??? He're is my ASR config regarding BGP:

router bgp 2625
 bgp log-neighbor-changes
 network 170.Z.Z0.0 mask 255.255.254.0
 network 170.Z.Z2.0 mask 255.255.254.0
 neighbor 187.Y.Y.112 remote-as 139
 neighbor 187.Y.Y.112 password 7 060B5A0847
 neighbor 187.Y.Y.112 route-map MC out
 neighbor 208.X.X.181 remote-as 349
 neighbor 208.X.X.181 password 7 151E58C3A6066
 neighbor 208.X.X.181 route-map L3 out
!
ip prefix-list L3 seq 10 permit 170.X.X.0/22 le 27
ip prefix-list L3 seq 15 deny 0.0.0.0/0
!
ip prefix-list MC seq 10 permit 170.X.X.0/22 le 27
ip prefix-list MC seq 15 deny 0.0.0.0/0
!
route-map MC permit 10
 match ip address prefix-list MC
!
route-map L3 permit 10
 match ip address prefix-list L3

Thanks!!

update: a moment ago I did shutdown interface connected to ISP Level3 and all traffic goes down...nothing passed through my second ISP

[Richard Burts]

The output that you posted shows that you did form two neighbor relationships. And shows that you received 655782 routes from one neighbor (who is advertising the entire routing table) and received 1 route from the other neighbor (which would be just the default route). So on the receiving end I think you are ok and that the problem is in what you advertise to the neighbor. As a start it would be helpful if you would post the output of the command show ip bgp neighbor 187.Z.Z.112 advertised. This would show what (if anything) you are advertising to them. I suspect that it will show that you are not advertising any of your routes to them.

It also occurs to me that you are advertising half of your address space to L3 and half to MC. If you shut down the link to L3 then the hosts in that address space will not work. Are you sure that there are active devices attempting to access the Internet from the part of your addresses advertised to MC?

HTH

Rick

[Georg Pauwen]

Hello,

in addition to Richard's post, what do you have in your routing table ?

network 170.Z.Z0.0 mask 255.255.254.0
network 170.Z.Z2.0 mask 255.255.254.0

Are these networks actually in your routing table ?

[Richard Burts]

Georg makes a very good point. When you use a network statement in BGP that includes a mask then BGP looks in the routing table for an entry that is an exact match for the network (exact including matching the mask) and if it does not find a matching entry then BGP does not advertise the route. To take care of this in configuring BGP people will frequently configure a static route for the network and mask (pointing the static route to null 0).

HTH

Rick

[gasparmenendez]

I'm using two static routes pointing to Null 0:

ip route 170.Z.Z0.0 255.255.254.0 Null0

ip route 170.Z.Z2.0 255.255.254.0 Null0

thanks.

[gasparmenendez]

here's my routing table:

Contencion1001-X#sh ip route | i 170.Z.Z   
S        170.Z.Z0.0/23 is directly connected, Null0
C        170.Y.Y.0/28 is directly connected, TenGigabitEthernet0/1/0
L        170.Y.Y.1/32 is directly connected, TenGigabitEthernet0/1/0
S        170.Y.Y.16/28 [1/0] via 170.Y.Y.2
C        170.Y.Y.32/29 is directly connected, GigabitEthernet0/0/1
L        170.Y.Y.33/32 is directly connected, GigabitEthernet0/0/1
C        170.Y.Y.40/29 is directly connected, GigabitEthernet0/0/2
L        170.Y.Y.41/32 is directly connected, GigabitEthernet0/0/2
S        170.Y.Y.48/28 [1/0] via 10.170.18.2
C        170.Y.Y.64/30 is directly connected, GigabitEthernet0/0/4.65
L        170.Y.Y.65/32 is directly connected, GigabitEthernet0/0/4.65
C        170.Y.Y.68/30 is directly connected, GigabitEthernet0/0/4.69
L        170.Y.Y.69/32 is directly connected, GigabitEthernet0/0/4.69
C        170.Y.Y.72/30 is directly connected, GigabitEthernet0/0/4.44
L        170.Y.Y.73/32 is directly connected, GigabitEthernet0/0/4.44
S        170.Y.Y.80/28 [1/0] via 170.Y.Y.2
C        170.Y.Y.96/30 is directly connected, GigabitEthernet0/0/4.20
L        170.Y.Y.97/32 is directly connected, GigabitEthernet0/0/4.20
S        170.Y.Y.100/30 [1/0] via 10.147.24.2
S        170.Y.Y.108/30 [1/0] via 10.147.89.2
S        170.Y.Y.112/30 [1/0] via 10.147.21.2
S        170.Y.Y.116/30 [1/0] via 10.147.197.2
S        170.Y.Y.120/30 [1/0] via 10.147.27.2
S        170.Y.Y.124/30 [1/0] via 10.147.29.2
S        170.Y.Y.128/30 [1/0] via 10.147.26.2
S        170.Y.Y.132/30 [1/0] via 10.147.28.2
S        170.Y.Y.136/30 [1/0] via 10.147.25.2
C        170.Y.Y.140/30 is directly connected, GigabitEthernet0/0/4.75
L        170.Y.Y.141/32 is directly connected, GigabitEthernet0/0/4.75
S        170.Y.Y.144/30 [1/0] via 10.147.20.2
S        170.Y.Y.148/30 [1/0] via 10.147.31.2
S        170.Y.Y.152/29 [1/0] via 10.147.22.2
S        170.Z.Z2.0/23 is directly connected, Null0
B        170.Z.Z4.0/22 [20/15184] via 208.X.X.181, 01:37:02
Contencion1001-X#

I can see both route on the table....the only thing that comes to my attention is the last route (B        170.Z.Z4.0/22 [20/15184] via 208.X.X.181, 01:37:02), the Z4 part....

[gasparmenendez]

I'm indeed advertising my network to 187.Z.Z.112:

Contencion1001-X#sh ip bgp neighbors 187.Z.Z.112 advertised-routes
BGP table version is 7410037, local router ID is 208.X.X.182
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  170.Z.Z0.0/23  0.0.0.0                  0         32768 i
 *>  170.Z.Z2.0/23  0.0.0.0                  0         32768 i

Total number of prefixes 2
Contencion1001-X#

and besides that, even when I did split my network in two halves, I'm still advertising the whole network to both ISP's, please take a look to my actual conf.:

router bgp 2625
 bgp log-neighbor-changes
 network 170.Z.Z0.0 mask 255.255.254.0
 network 170.Z.Z2.0 mask 255.255.254.0
 neighbor 187.Y.Y.112 remote-as 139
  neighbor 187.Y.Y.112 route-map MC out
 neighbor 208.X.X.181 remote-as 349
  neighbor 208.X.X.181 route-map L3 out
!
ip prefix-list L3 seq 10 permit 170.X.X.0/22 le 27
ip prefix-list L3 seq 15 deny 0.0.0.0/0
!
ip prefix-list MC seq 10 permit 170.X.X.0/22 le 27
ip prefix-list MC seq 15 deny 0.0.0.0/0
!
route-map MC permit 10
 match ip address prefix-list MC
!
route-map L3 permit 10
 match ip address prefix-list L3

please note that both prefix-list includes the whole network, even when in BGP I did split the block.

[Georg Pauwen]

The placeholders are confusing actually. Make sure you haven't made a typo somewhere. Your null routes and your prefix lists should match:

170.80.240.0/23

170.80.240.0/22 le 27

and 

170.80.242.0/23

170.80.242.0/22 le 27

respectively.

Also, looking at one of your static routes, what are the placeholders Y.Y for :

S        170.Y.Y.108/30 [1/0] via 10.147.89.2

Make sure that is actually an IP address that falls in the 170.80.240.0/22 range...

[Georg Pauwen]

Hello,

I have recreated this in GNS3, and it seems that the only thing you are missing is 'redistribute static' and 'redistribute connected' under the BGP process (or manually advertise all networks you want to send out).

Other than that, I cannot see anything wrong with your configuration...