Help with BGP

Answered Question
Apr 5th, 2010

Hi All,

I have a router with a single physical interface connecting to the outside world and talking BGP to two ISPs.

Here's the BGP config:

router bgp 65100
no synchronization
bgp log-neighbor-changes
network NETWORK mask 255.255.255.128
neighbor x.x.x.x remote-as 11830 (link to secondary connection)
neighbor x.x.x.x route-map METRIC in
neighbor y.y.y.y remote-as 11830 (link to primary connection)
neighbor y.y.y.y route-map METRIC in
maximum-paths 2
no auto-summary

According to the IP routing table, the router has two default gateways (one for each BGP), doing load balancing:

B*   0.0.0.0/0 [20/0] via y.y.y.29, 1w1d
                        [20/0] via x.x.x.5, 14:03:48

According to the BGP table, the preferred path should be y.y.y.129

   Network          Next Hop            Metric LocPrf Weight Path
*  0.0.0.0          x.x.x.5                  0             0 11830 i
*>                     y.y.y.29                 0             0 11830 i

The problem that I'm having is that if we disconnect the second link (connection to x.x.x.5), everything stops working.
If I do a traceroute from the local network, we get out to the Internet using x.x.x.5, but according to the BGP table
the preferred path should be through y.y.y.29 (which is supposed to be the primary link).

My goal is to have all traffic going out via y.y.y.29 and if it fails to go over x.x.x.5 (and then come back to the primary
connnection when the primary link comes back).

How can I check what's going on?

Federico.

I have this problem too.
0 votes
Correct Answer by ranjit123 about 6 years 8 months ago

Hi!,

Along with Timers you can use weight on the primary link so that your internal traffic will prefer this primary link.

If the primary link goes down traffic will automatically be diverted through the second link.

If the Primary link comes back up again traffic will automatically choose the preimary link with the secondary as back-up

Regards,

Ranjit

Correct Answer by Jon Marshall about 6 years 8 months ago

coto.fusionet wrote:

Yes.


We had a problem with a fiber and we physically disconnect the link to one BGP peer.
When we did this, all Internet traffic stopped working.
As soon as we connected the link back again (both BGP links up), then everything went back to normal.


We were surprised since, when disconnecting one BGP connection, the other one should assume the traffic.

I am going to do the test again as soon as possible to check the traceroute (when the link is down) and let you know.
The BGP configuration seems fine.

Federico.

Federico

It might also be worth checking with the ISP to see what they see when you disconnect one of the links. I'm assuming that they dynamically receive your networks via EBGP advertisements because you do have a network statement in your BGP config but if they were using statics at their end for some of your networks then their routers might not realise your end has gone down so outbound traffic from you goes the right way but return traffic does not.

Also how long are you waiting after disconnection. Bear in mind BGP is not the quickest to failover and you have not modified your BGP timers so you could be looking at minutes before the ISP end updated and removed the route to your networks back via the disconnected link.

Jon

Correct Answer by Jon Marshall about 6 years 8 months ago

Federico

It seems that the router first try to reach the x.x.x.5, then y.y.y.29, but then again x.x.x.5
Is this a problem with the BGP peer y.y.y.29?

What does the maximum-paths 2 under BGP accomplishes? Load balancing?

Shouldn't it prefer the route mark as ''best'' under BGP?


By default even if there are multiple equal cost paths to the same destination BGP will only use one of them. By adding the maximum-paths 2 command you have told BGP it can use 2 equal cost paths. Hence the reason it tries x.x.x.5 then y.y.y.29 then x.x.x.5 etc..

If you want it to always use just one of the default-routes then you should look to remove the maximum-paths command but only if you are sure it is not being used for some other destinations other than the default-route.

It will then only use one of the paths. As they are both equal cost though it could choose either so you will need to make sure it picks the y.y.y.29. You can modify the weight for this if you want.

Jon

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (3 ratings)
Loading.
Jon Marshall Mon, 04/05/2010 - 10:22

Federico

You have 2 ISPs, are they both using the same AS number ??? ie. 11830

We need to see the route-map configuration. Also can you post the output of "sh ip route | include 0.0.0.0"

Also, if you want to prefer one route over the other why have you configured "maximum-paths 2" under the BGP config ?

Jon

Federico Coto F... Mon, 04/05/2010 - 10:32

I'm sorry, is the same ISP with two BGP peers.

The route-map configuration is just this:

route-map METRIC permit 10
match ip address 101
access-list 101 permit ip 201.193.214.0 0.0.0.127 any

Output of sh ip route | include 0.0.0.0:

r-internet-ice#sh ip route | include 0.0.0.0
Gateway of last resort is y.y.y.29 to network 0.0.0.0
B*   0.0.0.0/0 [20/0] via y.y.y.29, 1w

Also,
When I do a trace from the Internet router, this is what I get:

r-internet-ice#trace 4.2.2.2

Type escape sequence to abort.
Tracing the route to 4.2.2.2

  1 x.x.x.5 [AS 11830] 8 msec
    y.y.y.29 [AS 11830] 0 msec
    x.x.x.5 [AS 11830] 4 msec
  2 x.x.x.97 [AS 11830] 0 msec 0 msec 4 msec
  3 144.223.245.133 [AS 11830] 40 msec 40 msec 40 msec
  4 144.232.2.241 [AS 11830] 40 msec 44 msec 40 msec
  5 4.68.62.77 [AS 11830] 56 msec 56 msec 56 msec
  6 4.69.138.126 [AS 11830] 64 msec 56 msec 68 msec
  7 4.69.140.142 [AS 11830] 56 msec 68 msec 56 msec
  8 4.68.103.34 [AS 11830] 56 msec
    4.68.103.2 [AS 11830] 56 msec
    4.68.103.34 [AS 11830] 56 msec
  9 4.2.2.2 [AS 11830] 60 msec 56 msec 56 msec

It seems that the router first try to reach the x.x.x.5, then y.y.y.29, but then again x.x.x.5
Is this a problem with the BGP peer y.y.y.29?

What does the maximum-paths 2 under BGP accomplishes? Load balancing?

Shouldn't it prefer the route mark as ''best'' under BGP?

Federico.

Federico Coto F... Mon, 04/05/2010 - 10:37

I've mentioned it was a single physical interface, sorry its two interfaces on the router.

So, the scenario is like this:

Single router having two BGP peer with the same ISP (using two outside interfaces)

Federico.

Federico Coto F... Mon, 04/05/2010 - 10:43

Also Jon, sorry for not posting the question correctly from start:

This is what we want:

We want to use both links to both BGP peers corresponding to the same ISP in load-balancing.

Since both links are 30M, we want to balance the traffic across both links.

But why if I disconnect the link to one BGP peer, everything stops working?

Sorry for the confusion, I hope its clear now.

Federico.

Jon Marshall Mon, 04/05/2010 - 10:48

Federico

How are you "disconnecting" the link to x.x.x.5 as you only have one physical link ?

Also what is the physical link, is it ethernet ?

Jon

Correct Answer
Jon Marshall Mon, 04/05/2010 - 10:41

Federico

It seems that the router first try to reach the x.x.x.5, then y.y.y.29, but then again x.x.x.5
Is this a problem with the BGP peer y.y.y.29?

What does the maximum-paths 2 under BGP accomplishes? Load balancing?

Shouldn't it prefer the route mark as ''best'' under BGP?


By default even if there are multiple equal cost paths to the same destination BGP will only use one of them. By adding the maximum-paths 2 command you have told BGP it can use 2 equal cost paths. Hence the reason it tries x.x.x.5 then y.y.y.29 then x.x.x.5 etc..

If you want it to always use just one of the default-routes then you should look to remove the maximum-paths command but only if you are sure it is not being used for some other destinations other than the default-route.

It will then only use one of the paths. As they are both equal cost though it could choose either so you will need to make sure it picks the y.y.y.29. You can modify the weight for this if you want.

Jon

Federico Coto F... Mon, 04/05/2010 - 10:51

The router has two physical connections to the ISP (I corrected earlier).
Each interface handles a BGP peer.
Both connections are Ethernet.

As a summary Jon,
We need to load-balance traffic across both links (seems to be happening now)...
But we need the either link to assume all traffic if one disconnects (this is not happening)...

Thank you for your help.

Jon Marshall Mon, 04/05/2010 - 10:58

Right, that makes more sense

So when you disconnect x.x.x.5 are you physically shutting the interface down ?

And once you have done this what does a traceroute show ie. do you see it going out via y.y.y.29 ?

Your traceroute from before is normal if you have 2 equal cost paths in the routing table ie. traceroute will use them both.

Also you are advertising just one network to both ISPs ?

Jon

Federico Coto F... Mon, 04/05/2010 - 11:03

Yes.


We had a problem with a fiber and we physically disconnect the link to one BGP peer.
When we did this, all Internet traffic stopped working.
As soon as we connected the link back again (both BGP links up), then everything went back to normal.


We were surprised since, when disconnecting one BGP connection, the other one should assume the traffic.

I am going to do the test again as soon as possible to check the traceroute (when the link is down) and let you know.
The BGP configuration seems fine.

Federico.

Correct Answer
Jon Marshall Mon, 04/05/2010 - 11:18

coto.fusionet wrote:

Yes.


We had a problem with a fiber and we physically disconnect the link to one BGP peer.
When we did this, all Internet traffic stopped working.
As soon as we connected the link back again (both BGP links up), then everything went back to normal.


We were surprised since, when disconnecting one BGP connection, the other one should assume the traffic.

I am going to do the test again as soon as possible to check the traceroute (when the link is down) and let you know.
The BGP configuration seems fine.

Federico.

Federico

It might also be worth checking with the ISP to see what they see when you disconnect one of the links. I'm assuming that they dynamically receive your networks via EBGP advertisements because you do have a network statement in your BGP config but if they were using statics at their end for some of your networks then their routers might not realise your end has gone down so outbound traffic from you goes the right way but return traffic does not.

Also how long are you waiting after disconnection. Bear in mind BGP is not the quickest to failover and you have not modified your BGP timers so you could be looking at minutes before the ISP end updated and removed the route to your networks back via the disconnected link.

Jon

Federico Coto F... Mon, 04/05/2010 - 12:31

Jon,

We are advertising just one public range in BGP to the ISP.

What are good BGP timers?

Federico.

Jon Marshall Mon, 04/05/2010 - 12:56

coto.fusionet wrote:

Jon,

We are advertising just one public range in BGP to the ISP.

What are good BGP timers?

Federico.

Federico


I have used timers of 10 & 30 rather than the default of 60 & 180 and they worked fine.

Jon

Correct Answer
ranjit123 Thu, 04/08/2010 - 04:03

Hi!,

Along with Timers you can use weight on the primary link so that your internal traffic will prefer this primary link.

If the primary link goes down traffic will automatically be diverted through the second link.

If the Primary link comes back up again traffic will automatically choose the preimary link with the secondary as back-up

Regards,

Ranjit

Actions

This Discussion