Can BGP play nicely with OSPF

Answered Question
Sep 1st, 2009
User Badges:

We have a somewhat diverse network. On the "legacy" side (consisting of a dark and leased fiber "MAN", PPP circuits, and a gig fiber based LAN) we have OSPF. On the "New" side we have MPLS as a replacement for our Frame Relay infrastructure. Our telco only supports BGP as a routing protocol so we have BGP across the MPLS. The problem we're having is maintaining a consistent routing table across the BGP/OSPF boundaries. We do not redistribute OSPF into BGP. We control BGP announcements with network statements and prepending. We do however redistribute BGP into OSPF. The tricky part is that we have 4 "shared" entry points into the BGP and OSPF networks. We have 4 DS3 connections coming from the MPLS cloud going to 4 of our major data centers. We also have 4 leased fiber connections going to each of those major data centers. The leased fiber is a rung so the fiber entry point to the data centers is one VLAN. We use 3845s for the DS3 connections. The WAN side has BGP and the LAN side has OSPF. The BGP side uses one AS # for the whole shebang. We've not been able to come up with a consistently stable way to get the BGP routes into OSPF and the OSPF routes into BGP. What typically happens is what I've started calling the "Self Serving Routing Loop". Router A gets a route through OSPF and in turn, via a network statement, injects that route ingo BGP. The MPLS network, via BGP, then sends that route to Router B which dutifully picks it up and drops it into OSPF. OSPF then updates Router A which starts the process all over again. Since Router A thinks Router B is destination for the route through BGP and Router B thinks Router A is a good destination through OSPF, we wind up with a routing loop.


My question is, what is the preferred way of keeping a consistent routing table between OSPF and BGP when there are 4 shared entry points into each network? Using eBGP for the MPLS and iBGP for the LAN/MAN isn't an option due to the costs of upgrading 80+ devices to Advanced Enterprise. I apologize if this my explanation is confusing.

Correct Answer by Edison Ortiz about 7 years 8 months ago

Terry,


You got the concept, nice job.


BTW, no need to include the metric-type 2 on the redistribution, it's done by default.


OSPF into BGP does not need 'subnets' you need 'subnets' from BGP into OSPF.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4.9 (8 ratings)
Loading.
johnspaulding Tue, 09/01/2009 - 05:58
User Badges:

If you could make a diagram and draw out the BGP/OSPF boudary points that would help. What you would want to do is deny duplicate routes when redistributing. Its fine if your not doing mutual redistribution but you also have to be carefull about what your advertising and you Administrative distance on the devices.


Typically when redistributing from B-->O you would use a route-map permitting the specific routes that you need (either ACL, tag value,etc) and just bring those into the network. If you have Dual redistribution points than you would typically deny those tagged routes from the other device that it doing redistribution. If you could draw up a diagram this would be helpful. thanks

trodecke Tue, 09/01/2009 - 06:47
User Badges:

Thanks. We've got a far too detailed map that I'll try to simplify and post. If I'm understanding what you're saying, I think that's where we're having a problem. We do still need the duplicate routes as the BGP network is supposed to be a backup connection in case the OSPF network connections fail at the data center. In other words, Router A still needs to somehow or another know that Router B's subnets are available via BGP just in case the leased fiber connection at Router B's location fails. We've come up with a configuration that uses a route-map to limit what can come in dynamically through BGP and a static route with a higher metric for all the subnets behind the other redistribution routers. We haven't implemented it yet as we wanted to make sure that this change will be the final change and we're not overlooking anything.


I'll see if I can get the diagram posted later this afternoon. Thanks.

johnspaulding Tue, 09/01/2009 - 06:53
User Badges:

Awesome...If you could also include the configurations that your planning on using I could help you out with this as well. I will be waiting for the digram and your response. thanks

Edison Ortiz Tue, 09/01/2009 - 06:56
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

When you redistribute a route from BGP into OSPF a tag from the BGP AS is inserted into the route.


You can create a route-map to match on that tag and deny the redistribution from OSPF back into BGP hence avoiding the "Self Serving Routing Loop".


This design will work better if all BGP speaking routers had their own AS #. Having their own AS # will help you determine what router redistributed that route from BGP into OSPF.


Route Tagging is the most scalable solution to avoid routing loops in a complex network as yours.




__



Edison.

trodecke Tue, 09/01/2009 - 07:23
User Badges:

Is this Edison Ortiz from NYC that either is or was a Novell SysOp?


More on topic, since we use the same AS everywhere, if I understand you correctly, then we couldn't use the default tag. Is there a way to apply a tag in either in the BGP config or the network statements so that we can create a route map statement based off of that?

Edison Ortiz Tue, 09/01/2009 - 07:28
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

Oh my, Terry Rodecker - Okie boy :) How are you man?


You can apply a tag on redistribution from BGP into OSPF.


router ospf x

redistribute bgp xxx subnets tag xxx


Then this redistributed route is unique within the OSPF domain and be able to block on other BGP routers doing the redistribution back into BGP.



Good to hear from you man..


trodecke Tue, 09/01/2009 - 07:35
User Badges:

Same here Edison. E-mail me at terry dot rodecker at gmail dot com.


I apologize but this is coming to me not as fast as I'd like, if we add custom tags to the BGP redistributed routes we can block those routes from getting into the OSPF database on other routers using a route map statement? Is there a way to add tags to only some of the routes getting redistributed? In other words, tag the non-mpls routes coming in from one of the other 3845s but not the routes coming in from the MPLS only connected sites?


Boy, I really need to get that diagram posted. I'm getting confused talking about it and it's my network. ;)

Edison Ortiz Tue, 09/01/2009 - 07:42
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

I apologize but this is coming to me not as fast as I'd like,


It's Ok :)


if we add custom tags to the BGP redistributed routes we can block those routes from getting into the OSPF database on other routers using a route map statement?


You won't block those routes from entering your OSPF database but you will be able to color them with a tags. With this design, a BGP router will bring those routes from BGP into OSPF but another BGP router won't take these routes back into BGP causing this loop. You want these routes to remain in OSPF, not advertised back into BGP.


Is there a way to add tags to only some of the routes getting redistributed?


Yes, you can - with route-maps:


route-map SET-TAG permit 10

match ....

set tag xxx


route-map SET-TAG permit 20



router ospf x

redistribute bgp xxx subnets route-map SET-TAG





Jon Marshall Tue, 09/01/2009 - 08:15
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Edison


Apologies for butting in but the OP is not redistributing OSPF into BGP, he is using network statements so how will the tags help him ?


Jon

trodecke Tue, 09/01/2009 - 08:31
User Badges:

Initially we were redistributing OSPF into BGP but as you can imagine, we had less than stellar results as soon as we brought up the second shared connection. That's when we went to network statements. We still had issues so we added prepending using a route-map statement - any "local" routes go out with just the AS added to it, any "backup" routes get the AS prepended a certain number of times depending on if that router is the main backup, the secondary backup, or the tertiary backup for those routes. It would be nice if we could simply go back to automatically redistributing OSPF into BGP and vice versa. Over the years I've found that while you gain a large measure of control over things doing it manually, you also gain a large potential for royally messing things up. We're human and humans make mistakes. ;)


At this point in time, I'm still open to any and all ideas. To control the issue now we simply only have one DS3 currently enabled. We'll manually enable any other DS3 if we need to fail to that site. As bad as that is it's preferable to not knowing if a route to a remote site is going to be working or not when we come to work in the morning.

Jon Marshall Tue, 09/01/2009 - 08:44
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Terry it would be helpful if you could provide a small example of what is happening in terms of route choice and what you want to happen ie. router A sees BGP as the best path, why does router B see OSPF as the best path.


I know when i was doing something similiar ie. multiple entry points with BGP and EIGRP I ended up modifying the weight in BGP but that was to make sure BGP was used over EIGRP so it may not be applicable. Like i say, an example would be helpful.


Having said that it may also be worth waiting for Edison to come back as i may well have misunderstood what he is proposing and i've learnt it's never a good idea to underestimate Edison !!


Jon

trodecke Tue, 09/01/2009 - 08:47
User Badges:

I've just about got the picture ready to go. I think when you see that, you'll have a better understanding of what I mean by the self serving routing loop. Thanks!

trodecke Tue, 09/01/2009 - 09:25
User Badges:

OK, here's the diagram.


We started out with just the leased fiber connections on the back side of locations A thru D. Unfortunately, our leased fiber provider has a history of outages so management asked that we provide a good sized backup pipe. That's where the MPLS connections came into play. They're primarily there to provide connectivity to the MPLS connected sites (remote locations). However, if the leased fiber from Location A should go down, we need all traffic from Locations B thru D to fail over to the MPLS cloud to get to location A. The same is true for the other major locations, the MPLS connection needs to be a backup for the leased fiber. Note that the leased fiber and the DS3 connections do not terminate in the same device. The L3 switches are a combination of 6509s, 3750s, and a 4948. There are multiple subnets at each of the major locations A thru D.


Also, there's a multi-point GRE VPN tunnel from each of the remote locations to a 2811 at one major location. We use the floating static route method to handle traffic to and from the remote locations if their primary connection ever goes down. The remote location has a default route pointing down the tunnel. The VPN router has a set of static routes, pointing back to the remote location's local subnets with the IP address of the remote router's tunnel as the next hop. These static routes have a metric of 111 so that OSPF is preferred. If the route disapears from OSPF the VPN router injects it's static route. This part has worked flawlessly over the years.


Please let me know if you have any other questions about what we've got and/or what we're trying to do. Thanks.



Edison Ortiz Tue, 09/01/2009 - 09:55
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

Jon,


Picture a remote location with 172.16.1.0/24 connected to the MPLS Cloud. There are multiple BGP routers at the DC connected to the MPLS.


Router A picks the 172.16.1.0/24 and redistribute this route into OSPF.


Router B has a network statement for this route (I'm assuming based on the route-loop Terry has experienced) and advertises this route back to BGP.


Router A may use this route as Best Path based on the BGP attributes hence causing the loop.


We need to eliminate Router B from advertising 172.16.1.0/24 back into BGP and the most scalable way of avoiding this would be with redistribution with route-tagging.


I guess you haven't come across those redistribution nightmares from the INE labs on your CCIE studies yet?


:)



trodecke Tue, 09/01/2009 - 10:06
User Badges:

Thanks Edison, that's pretty much what is happening. I'm looking for something on Cisco's site talking about the tagging with some examples. I can see the route map coming in (the redistribution statement in OSPF) but I can't quite figure out how we're going to get other BGP routers to not insert those routes back into BGP.

Edison Ortiz Tue, 09/01/2009 - 10:17
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

A quick search didn't turn up a clear example on tagging, I'll have to dig further. The best I could find is explained here:


http://www.cisco.com/en/US/tech/tk365/technologies_white_paper09186a0080531fd2.shtml#tags



Basically, the idea is this:


1) All routers will redistribute from BGP into OSPF.


2) During redistribution, those routers will apply an unique tag (you mentioned you only wanted to tag routes from the MPLS - are there any other routes? more on that after you reply with more info).


Router A


router ospf x

redistribute bgp xx subnets tag 1


Router B


router ospf x

redistribute bgp xx subnets tag 2


Router C


router ospf x

redistribute bgp xx subnets tag 3



3) You want to redistribute OSPF routes that were not learned from the MPLS. Those OSPF routes aren't tagged so:


route-map OSPF->BGP deny 10

match tag 1

route-map OSPF->BGP deny 20

match tag 2

route-map OSPF->BGP deny 30

match tag 3

route-map OSPF->BGP permit 40


router bgp xx

redistribute ospf xx route-map OSPF->BGP



Jon Marshall Tue, 09/01/2009 - 11:09
User Badges:
  • Super Blue, 32500 points or more
  • Hall of Fame,

    Founding Member

  • Cisco Designated VIP,

    2017 LAN, WAN

Edison


You see, that's why i never underestimate you :-)


Thanks


Jon

Edison Ortiz Tue, 09/01/2009 - 09:45
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

Redistribution will be part of the new design.


You need redistribution from OSPF into BGP to dynamically process the tag from OSPF. You can't do that while explicitly entering the network statements under BGP.

johnspaulding Tue, 09/01/2009 - 10:19
User Badges:

You have alot of options you could do with this design as far as redundancy + load balancing too. Depends on the design you want. You could mutual redistribute BGP & OSPF on the routers with a well planed out route-map policy for tagging.

trodecke Tue, 09/01/2009 - 10:40
User Badges:

I agree and I believe this is the direction that we're going to go. I just need to start fleshing out some of the details of the overall plan and see what, if any, obstacles I come up with.


To Edison - We haven't really had a problem yet where a BGP only connection was involved in a routing loop. What we've had issues with is an OSPF only connection getting involved in a routing loop when Router A inserts the route into BGP which updates Router B which takes that route and drops it back into OSPF, etc. I'm doing my thinking now around what the route map statements need to be.


Thanks for all your help on this, all three of you. You've been very helpful in getting me to think about alternative solutions to the very manualy intensive route-map/static list config I had been thinking about.

Edison Ortiz Tue, 09/01/2009 - 10:49
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

Terry,


I fully understand your scenario. I actually finished doing a design just like that one and worked quite well.


BTW, unlike Novell forums, we do rate helpful posts here - so how about it ? :)


___


Edison.

trodecke Tue, 09/01/2009 - 12:52
User Badges:

OK, here's what we've come up with. Again, it's bare bones but I believe it gets the point across. Does this somewhat match what you were trying to tell me? Do you think it will work? Are there any improvements you would make on this? Thanks.


************** Router A ****************

router ospf 1

redistribute bgp XXXXX subnets metric 120 metric-type 2 tag 1111

network 10.X.X.X 0.0.255.255 area 0


router bgp XXXXX

redistribute ospf 1 subnets route-map OSPF-BGP

neighbor A.A.A.A remote-as YYYYY

neighbor A.A.A.A soft-reconfiguration inbound

neighbor A.A.A.A route-map myroutes out

no auto-summary


route-map myroutes permit 10

match ip address prefix-list mine

route-map myroutes permit 20

set as-path prepend XXXX XXXXX XXXXX


route-map OSPF-BGP deny 10

match tag 1111

route-map OSPF-BGP deny 20

match tag 2222

route-map OSPF-BGP deny 30

match tag 3333

route-map OSPF-BGP deny 40

match tag 4444

route-map OSPF-BGP permit 50


ip prefix-list mine seq 10 permit 1.1.1.1

ip prefix-list mine seq 20 permit 11.11.11.11




************** Router B ****************

router ospf 1

redistribute bgp XXXXX subnets metric 120 metric-type 2 tag 2222

network 10.Y.Y.Y 0.0.0.255 area 0


router bgp XXXXX

redistribute ospf 1 subnets route-map OSPF-BGP

neighbor B.B.B.B remote-as YYYYY

neighbor B.B.B.B soft-reconfiguration inbound

neighbor B.B.B.B route-map myroutes out

no auto-summary


route-map myroutes permit 10

match ip address prefix-list mine

route-map myroutes permit 20

set as-path prepend XXXX XXXXX XXXXX


route-map OSPF-BGP deny 10

match tag 1111

route-map OSPF-BGP deny 20

match tag 2222

route-map OSPF-BGP deny 30

match tag 3333

route-map OSPF-BGP deny 40

match tag 4444

route-map OSPF-BGP permit 50


ip prefix-list mine seq 10 permit 2.2.2.2

ip prefix-list mine seq 20 permit 22.22.22.22




************** Router C ****************

router ospf 1

redistribute bgp XXXXX subnets metric 120 metric-type 2 tag 3333

network 10.Z.Z.Z 0.0.0.255 area 0


router bgp XXXXX

redistribute ospf 1 subnets route-map OSPF-BGP

neighbor C.C.C.C remote-as YYYYY

neighbor C.C.C.C soft-reconfiguration inbound

neighbor C.C.C.C route-map myroutes out

no auto-summary


route-map myroutes permit 10

match ip address prefix-list mine

route-map myroutes permit 20

set as-path prepend XXXX XXXXX XXXXX


route-map OSPF-BGP deny 10

match tag 1111

route-map OSPF-BGP deny 20

match tag 2222

route-map OSPF-BGP deny 30

match tag 3333

route-map OSPF-BGP deny 40

match tag 4444

route-map OSPF-BGP permit 50


ip prefix-list mine seq 10 permit 3.3.3.3

ip prefix-list mine seq 20 permit 33.33.33.33




************** Router D ****************

router ospf 1

redistribute bgp XXXXX subnets metric 120 metric-type 2 tag 4444

network 10.1.1.0 0.0.255.255 area 0


router bgp XXXXX

redistribute ospf 1 subnets route-map OSPF-BGP

neighbor D.D.D.D remote-as YYYYY

neighbor D.D.D.D soft-reconfiguration inbound

neighbor D.D.D.D route-map myroutes out

no auto-summary


route-map myroutes permit 10

match ip address prefix-list mine

route-map myroutes permit 20

set as-path prepend XXXX XXXXX XXXXX


route-map OSPF-BGP deny 10

match tag 1111

route-map OSPF-BGP deny 20

match tag 2222

route-map OSPF-BGP deny 30

match tag 3333

route-map OSPF-BGP deny 40

match tag 4444

route-map OSPF-BGP permit 50


ip prefix-list mine seq 10 permit 4.4.4.4

ip prefix-list mine seq 20 permit 44.44.44.44



BTW - The BGP and OSPF only routers will be configured normally, with either regular BGP or regular OSPF statements. Thanks.

Correct Answer
Edison Ortiz Tue, 09/01/2009 - 13:10
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

Terry,


You got the concept, nice job.


BTW, no need to include the metric-type 2 on the redistribution, it's done by default.


OSPF into BGP does not need 'subnets' you need 'subnets' from BGP into OSPF.

trodecke Tue, 09/01/2009 - 13:20
User Badges:

Thanks Edison. I was typing into notepad as I don't have router next to me to play with right now. I'm a complete newb on BGP. We've got someone working for us that previously worked for an ISP and is very familiar with BGP. He would have caught my boo boo. :)


Again, thanks for all your help!

Edison Ortiz Tue, 09/01/2009 - 13:22
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

Glad to be of help man and don't be a stranger.


I also check the semi-private forum from time to time...


__


Edison.

johnspaulding Tue, 09/01/2009 - 13:10
User Badges:

Yeah that looks pretty solid..So the end result would be any routes being redistributed FROM BGP into OSPF will not be redistributed BACK into BGP to its neighbors.I would check this route-map out real quick:


route-map myroutes permit 10

match ip address prefix-list mine

route-map myroutes permit 20

set as-path prepend XXXX XXXXX XXXXX


You need to put the as-path prepend under the myroutes permit 10 statement or else your going to allow all routes outbound to your neighbors. so like this


route-map myroutes permit 10

match ip address prefix-list mine

set as-path prepend XXXX XXXXX XXXXX

Edison Ortiz Tue, 09/01/2009 - 13:15
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

John,


I think the logic from Terry is that he does not want to prepend his own routes while prepending what's not his (everything else).



johnspaulding Tue, 09/01/2009 - 13:29
User Badges:

Edi,


Thanks for clearing that up. Im trying to read through all of these post that I got lost :)

Edison Ortiz Tue, 09/01/2009 - 13:33
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

John,


Not a problem and thanks for your contribution. I find it easier to read a long thread by hitting the Outline link from the OP's post.




trodecke Tue, 09/01/2009 - 13:16
User Badges:

Thanks. That makes it trickier though as I'd have to enter in all the OSPF routes that aren't local to that location rather than just entering in the routes that are local to it. Is there a way to do it somewhat reverse of what you've entered? Basically, what we want to do (and what we have now) is for the OSPF routes, the 3845s will populate BGP with their local subnets with a single AS prepended but multiple ASs prepended for all other routes. We do it with a route-map statement now but that route-map statement will get very wicked if we're redistributing OSPF into BGP rather than using the network statements that we have now. We've got something in the neighborhood of 1000 subnets in our routing table. Not huge but big enough that I don't really want to list them individually in a prefix list. ;)

Edison Ortiz Tue, 09/01/2009 - 13:20
User Badges:
  • Super Bronze, 10000 points or more
  • Hall of Fame,

    Founding Member

Another thing Terry, you may want to consider using E1 vs E2.


E1 takes into account the cost of the link so you can have routes drawn to the closest exit point.


You will have multiple OSPF routes for the same subnet with the new design. Each WAN edge router will be redistributing about the same BGP routes from the remotes into OSPF so a lot of load-balancing will take place.


__


Edison.

johnspaulding Tue, 09/01/2009 - 10:52
User Badges:

No problem..I wouldnt mind seeing the final results and what you decide to do.


heres just a tip for the route-map policy


Denying the taged routes from the other ASBR routers.


route-map B-->O deny 10 (RTRA)

match tag 2110

route-map B-->O deny 20 (RTRB)

match tag 3110

route-map B-->O deny 30 (RTRC)

match tag 4110

route-map B-->O permit 40


- I think you know were im going with this


You can either use a prefix-list or access-list or nothing to permit the rest of the untagged routes.


When redistributing the BGP routes into OSPF you can do this: Might been stated above but oh well.


router ospf 1

redistribute bgp 1 subnets route-map B-->O tag 1110


verify the tag value


show ip ospf database - Check the LSA type 5 external routes and they should have a tag value...But you also need to check you metric value and what going on there. Just my two cents. Let me know if you have any other questions.





Actions

This Discussion