BGP route-reflectors and MPLS - suboptimal path.

[Konstantin Dunaev]

Hello everybody,

I'm quite lost and need some good advices about my network topology.

Please have a look at a  picture in the attachment.

We have 4 routers physicaly  connected in a ring, three of them have a eBGP session with a upsteam ISP.

Two RC-RR's are route-reflectors and all other routers have  BGP sessions with them using   Loopbacks IP as source. 

Because of speed and price the connection RC-E001 <--> RC-RR1 is a backup and OSPF and BGP metric are set accordingly.

The internal routing are working as expected.  All routers are MPLS "P" routers, but only  Loopbacks IP are label-switched, it means that only traffic to a Loopback follow label-path, other traffic should use normal routing table.

The problem is followin: Traffic to the Internet  from the router RC-E001 follows the path RC-E002 ---> RC-RR2 ---> RC-RR1,

but it should just go to the router RC-E002 and then directly to the Internet.  All external prefixes on RC-E001 have RC-RR1 as a next-hop (higher local-preference)

Traceroute on RC-E001 shows following:

RC-E001#traceroute 8.8.8.8    

  1 RC-E002 [MPLS: Label 202 Exp 0] 16 msec 20 msec 60 msec

  2 RC-RR2 [MPLS: Label 79 Exp 0] 20 msec 16 msec 20 msec

  3 RC-RR1 [AS UPSTREAM] 20 msec 16 msec 20 msec

  4 UPSTREAM [AS UPSTREAM] 20 msec 16 msec 20 msec

  5 ....

I understand that RC-E001 tries to reach  the BGP next-hop via MPLS label-path, bacause all Loopbacks should use MPLS Label path-switching, but I don't want that the traffic goes in such sub-optimal way.

What have I configured wrong and what should I do to force  the traffic  from RC-E001 goes out direct  from RC-E002?

Best regards,

Konstantin

[Konstantin Dunaev]

Hi Matthew,

of course RC-RR1 and RC-RR2 do have a iBGP session , as I said, they are our Route-refclectors and all other routers are clients of them.

I wanted to set hierachical route-reflectors because I don't want to set a third RR router, beacuse it would need to set a nes BGP session, it means more prefisex on each router and so on.

[maayre]

On second read over your earlier discussions regarding the BGP local-preference scheme.

Best common practise would dicatate that routers in  the same BGP AS should never have routes with different attributes. Ie  you should never set local preference in the middle of your network for a  specific route and some routes have the old route and some have the  changed route.

As you are aware you should also have an iBGP full mesh or at least route-reflectors.

So what you should have is

RR1 iBGP to RR2, E001, E002 (last two are RRClients)

RR2 iBGP to RR1, E001, E002 (last two are RRClients)

Then  all you have to do is on the network borders (which in this case are  RR1 & RR2) just configure the local-preference on the route as it is  learnt inbound (whether from eBGP or redistribution of a static route).

This way all routers will have "congruent routing information" which is what's expected in terms of best common practises.

I'm not too sure about the other background of what  you're trying to do here but for example if you also intended some kind  of load balancing scheme that could be done too but I don't think it's  in the scope of what you were asking.

Anyway sorry for all the spam, let us know how you go after this one =)

HTH,

Matt

[Konstantin Dunaev]

Matthew, you're right about iBGP sessions and BGP attributes, I'm  changing our  local-preference only on our eBGP sessions and all BGP attributes stay the same over all network on all routers.

The BGP sessions with Route-reflectors are configured with "weigth", so I can choose the primary and secondary path.

I just want to reach an optimal path, if you see the firsth picture, you can see that  traffic to external prefixes goes the whole network and left it only at RC-RR1 instead of RC-E002.

[Kishore Chennupati]

Hi All,

Please allow me to jump in here with your permission.

Konstatin,

Before I go into solution. let me ask you a design question and this will also have the solution within it for your problem.

Q. What happens if the link between RC-E002 <->RC-RR2   and RC-E001 <->RC-RR1 goes down?

   You are toast. Although  RC-E002 has ebgp with ISP it wont pass the default route to RC-E001 because there is no iBGP between RC-E002 and RC-E001. so literally RC-E001 will not be able to get to the internet. You see what I mean here.

Now, the solution to your problem

Make RC-E002 as a RR with a LP=200 and rest of routers its client. Now, if the above scenario occurs then RC-E001 will still be able to get to the internet.

This will also fix your original problem. Since RC-E002 is the RR with an LP =200 and in a full mesh , it will send that to the other 3 routers(RC-RR1, RC-RR2, RC-E001). Now , RC-E001 will ignore the LP=150 from RC-RR1 and route via RC-E002.

Even if you are using MPLS here it will still be the same as it will just create an LSP between  RC-E001 and RC-E002 between the loopbacks and follow that path.

In case RC-E002 dies, then RC-E001 will go to the internet via RC-RR1.

or if the link between RC-E002 and RC-E001 dies then RC-E001 will  take the path from RC-RR1<->RC-RR2<->RC-E002 to the internet.

Hope this answers your question. If any thing unclear, please let me know.

Regards,

Kishore

Message was edited by: Kishore Chennupati

[Konstantin Dunaev]

Hi Kishore,

thank you for jumping in

to your question - I'm agree actually, but these links are completely separate, in different locations, different media and different HW. But I see what you mean.

To your suggestion, I'm agree that RC-E001 need somehow a BGP session with RC-E002, but as I said I don't want to put RC-E002 on the same route-reflector level as our main route-reflector router - RC-RR1 and RR-RC2, that is why I was thinking about second level of RR. But I'm not sure if I can/may set an iBGP session to a route-reflectors from different levels.

[Kishore Chennupati]

Hi Konstantin,

You can setup different levels of RR's. You can have RC-E001 and RC-E002 in one level and have RC-RR2 and RC-RR1 in another level and then have the RR's talk to each other but to solve your original problem you would still need RC-E002 as a RR.

Please see this below which has a diagram of how to connect different RR's within the same AS.

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800c95bb.shtml#routereflectors

HTH

Kishore

Edit: Actually looking at your topo RC-E001 might have to be also made an RR -client and also an RR if you want to want multiple clusters. otherwise if RC-E002 dies then the RC-E001 wont be able to go out as it wil become isolated. A RR client in a cluster can have only RR within the same cluster. But if it wants to talk to differnet clusters then it needs to be a RR as well. 

Honestly, having multiple clusters in your topology might not be beneficial and it will make it more messy in my opinion. Please stick to the solution i provided in my prev post and have a flat network

Message was edited by: Kishore Chennupati

[Konstantin Dunaev]

exactly, RC-E002 will be RR router but on the higher  level of RR topology like this:

the question is:  is it possible or is it allowed to set RC-E001 as route-reflector client of  RR routers RC-E002 and RC-RR1 which are on different levels? I'm not quite sure how the BGP information will be  exchange in this case.

[Kishore Chennupati]

The answer is NO. I explained that in my prev post in the Edit: message

Also  in your diagram you have RC-E002 as a RR-client and RC-E001 also as a client to this client. You can't have a RR client to another RR client.

What you can do though is to make RC-E002 an RR and then make RC-E001 its client and also and RR. Then you can have a RR <->RR relationship between RC-E001 and RC-RR1 and this is allowed. Does it make sense?

I had edited my prev post which talks about this concept of communication between clusters.

Let me give you another link which might help

http://blog.sazza.de/?cat=22

HTH

Kishore

[maayre]

Kishore,

A RR can be a client of another RR, this is called "hierarchical RRs" and is used to scale in very large SP networks.

In an abstract sense there is no such a thing as a RR, just a feature you can use to turn off the iBGP split horizon rule per neighbor.

[Kishore Chennupati]

Matt,

A RR can be a client of another RR, this is called "hierarchical RRs" and is used to scale in very large SP networks.

Where did I mention that it cannot. please point the post to me

Regards

Kishore

[Konstantin Dunaev]

I agree with you to some extent, but why do want to have RC-E001 as route-reflector and RC-E002 as it client and not other way around?

I should clarify that RC-E002 is truly backbone router and RC-E001 is let say more or less a "stub router".

Kishore Chennupati wrote:
Also  in your diagram you have RC-E002 as a RR-client and RC-E001 also as a client to this client. You can't have a RR client to another RR client.

I think Matt  points to this post, actually I'm disagree with this statements as well. I din't use a hierarchical RR  topology in a proctice but pretty all MPLS topologies somehow refers to hierarchical RR topology.

[Kishore Chennupati]

Konstantin,

Please read my prev post carefully my friend. Below is what I said.

" What you can do though is to make RC-E002 an RR and then make RC-E001 its client and also and RR. Then you can have a RR <->RR relationship between RC-E001 and RC-RR1 and this is allowed. Does it make sense?"

[Konstantin Dunaev]

Konstantin,
Please read my prev post carefully my friend. Below is what I said.
" What you can do though is to make RC-E002 an RR and then make RC-E001 its client and also and RR. Then you can have a RR <->RR relationship between RC-E001 and RC-RR1 and this is allowed. Does it make sense?"

Kishore, 

that is exactly the point which I'm not quite sure I correctly understand you.
 
You mean I can set RC-E002 as RR router and in the same time it stays as 
route-reflector client for RC-RR1 and RC-RR2? Do I understand you correctly?

Then I configure RC-E001 as a route-reflector client for RC-E002?


But how can I configure "RR relatioship" between RC-RR1 and RC-E001? Do you meanto  configure 
a simple iBGP session between them? I don't think it's a good idea - we don't get a full-mesh there 
if  RC-E002 (RR router for RC-E001) failed. 

[Kishore Chennupati]

Konstantin,

You can use it but there are certain considerations you need to take into account while using clusters etc which I explained in my prev posts. I am happy to be corrected if I am wrong.

[Konstantin Dunaev]

it's a veery good link, thank you.

it shows pretty the same topology as I would like to get, without RR-BGP session between RC-E001 and RC-RR1.

Actually now I understand why many ISP's are trying to separte Internet and MPLS backbones, at leaset on the logical-BGP level