Thanks for the answers, guys, but I think I didnt make a few things clear enough.

The set up is as follows:

There are only 4 routers in the whole topology, 2 edge routers and 2 core routers that sit behind them.

Topology is a basic square:

Edge 1 is directly connected to core 1; Edge 2 is directly connected to core 2. Edge 1 is directly conneced to edge 2. Core 1 is directly connected to core 2.

My question did not concern how to go about creating an iBGP neighborship between edge 1 and edge 2. Thats easy. Either I can use the direct link between the edge routers and assign a /30 and be done with it. Or I can use loopbacks and make sure each router knows how to get to each others loopback. (Edison, didnt know that was a best practice. Why is that?).

My question involved a posible OSPF neighborship between edge 1 and edge 2, in addition to the iBGP. Take note that the iBGP MUST be there for 2 reasons:

1.) The client demands it. :-)

2.) Edge 1 is primary connection to Internet. If that connection fails, traffic can get rerouted through the iBGP crosslink to edge 2. But mostly it is because of item 1, so that part is not up for debate.

The route logic is extremely simple:

Each edge router runs eBGP with their respective ISP neighbor, and OSPF with the core router that sits behind it. There is no redistribution of BGP into OSPF or OSPF into BGP. OSPF on the edge route (Which makes it an ASBR), does a default info orig to advertise a default into OSPF. OSPF also advertises the /30 direct links between them.

Also, the edge routers are accepting full internet routes but NO default route. The default is advertised statically on each edge router.

So, again, the question is this:

With an iBGP link between edge 1 and edge 2, is there any need, any value, any earthly reason for also having an OSPF neighborship between edge 1 and edge 2?

The only reason I can think for having an OSPF neighborship between them is to have the default advertised from edge 2 to edge 1 in the event that edge 1's BGP connection fails and it needs a default route. It is not learning one through iBGP because it is statically configred and it points out to the internet interface. So, it will have to learn it through OSPF (def info orig - just like core 2 does), or one can configure a FLOATING static default on edge 1 pointing to ede 2, and vise versa for that matter.

Thanks

And what happens in your square topology if connection between an edge and core fails? Assuming edge OSPF peers with core, and default is toward Internet, how does edge then know where internal routes are? Could be done with statics, but easier (I think) with OSPF. However, for internal, would be better if second edge to edge link connected to other core. Otherwise, outbound traffic (I believe) would only use one outbound link if single edge to core link fails. (Even better, would be both, but again this assumes you want traffic that's outbound on one edge router to use other edge router if it has "better" eBGP route.)

[edit]

"With an iBGP link between edge 1 and edge 2, is there any need, any value, any earthly reason for also having an OSPF neighborship between edge 1 and edge 2? "

Yes, especially if you keep to "square" topology. Besides thinking failure of external edge to Internet path, you must also think about failure of edge to core link and traffic flow in both directions.

However, again, instead of using 2nd link from edges to connect between them, might be better to connect both cores to both edges. Then there would be little need to OSPF directly peer them, although having connection between them can be used for "better" outbound path selection since initial edge, for outbound internal traffic, will be only selected by default route.

Hello Joe,

>> Take note that the iBGP MUST be there for 2 reasons:

1.) The client demands it. :-)

2.) Edge 1 is primary connection to Internet. If that connection fails, traffic can get rerouted through the iBGP crosslink to edge 2.

The topology is a square not a full-mesh.

Even if your customer didn't mention it one key target is fault-tolerance: defined as the capability of the network to survive to a single link failure or a single node failure.

This is the fault-tolerance at level 1 and this is implicitly expected nowdays.

So the following measures are needed:

the iBGP session have to terminated using loopbacks that are advertised in your IGP (OSPF).

All links have to run OSPF.

But this is enough only to keep live the iBGP session and not for point 2:

If that connection fails, traffic can get rerouted through the iBGP crosslink to edge 2.

if the direct link between edge1 and edge2 fails the iBGP session is rerouted via core1, core2.

But this means that if edge1 needs to reroute user traffic to edge2 via the U path (because direct link has failed) the packets are sent to core1:

now if core1 has no BGP knowledge it will send the packet according to OSPF default route including in some cases sending the packet back to edge1 and you have a routing loop!

So an optimal design requires OSPF enabled everywhere, iBGP sessions using loopbacks addresses and a full mesh of iBGP sessions between edge1,edge2,core1,core2.

in this case if the edge1-edge2 link fails or simply because edge2 has a better path for a specific destination iBGP rerouting will work well.

To be noted: if the edge routers receive full BGP tables you need to verify if the core devices are able to host 300,000 routes in their routing table and in their CEF tables (also very important)

As I wrote in my first post MPLS technology can help solve this possible issue eliminating the need for an iBGP full mesh but still the iBGP session between edge1 and edge2 has to be using loopbacks and OSPF and MPLS have to enabled on all links.

As you see the complexity of the solution increases and the original customer requirement has been integrated with fault-tolerance.

All this to have a good chance to be called again for future projects by the same customer.

Hope to help

Giuseppe

"Take note that the iBGP MUST be there for 2 reasons: "

I must not be explaining myself well (I apologize if true). For instance, I didn't think I expressed, nor intended, for there not to be iBGP, at least between the two eBGP WAN edge routers. On this point, besides 2nd OP's post's "requirements", I've tried to note, each eBGP WAN edge router would be able to send outbound traffic to its WAN edge peer if the peer had a "better" (shorter) AS path for a route. I.e., there's an advantage if the two WAN edge routers share their BGP routes, even w/o the "requirements" from the OP's 2nd post.

"The topology is a square not a full-mesh. "

Yes, as we now know from the OP's 2nd post. I did note, now knowing this, I thought having dual connections between the core and WAN edge routers might be better than just a square (NB: you raise an important point on this issue - further on), and those connections with the WAN edge to edge connection even better yet, but also didn't actually recommend a full mesh (i.e. didn't recommend core to core), since I don't know what the topology of the other side of the cores is.

"So the following measures are needed:

the iBGP session have to terminated using loopbacks that are advertised in your IGP (OSPF).

All links have to run OSPF. "

Not sure I fully subscribe to "have to", assuming I'm not mistaken, but whether I'm mistaken or not, would agree it's a really, really good idea.

"But this is enough only to keep live the iBGP session and not for point 2:

If that connection fails, traffic can get rerouted through the iBGP crosslink to edge 2.

if the direct link between edge1 and edge2 fails the iBGP session is rerouted via core1, core2.

But this means that if edge1 needs to reroute user traffic to edge2 via the U path (because direct link has failed) the packets are sent to core1:

now if core1 has no BGP knowledge it will send the packet according to OSPF default route including in some cases sending the packet back to edge1 and you have a routing loop!

So an optimal design requires OSPF enabled everywhere, iBGP sessions using loopbacks addresses and a full mesh of iBGP sessions between edge1,edge2,core1,core2.

in this case if the edge1-edge2 link fails or simply because edge2 has a better path for a specific destination iBGP rerouting will work well. "

An excellent point! I was writing on the issue of Internet return traffic, how OSPF would ease one WAN edge router still "knowing" internal routes in the OP's square topology if it lost its connection to the core router, but Giuseppe raises a new issue, the danger of routing via a core if there's no direct edge to edge path and with an OSPF default route being advertized from the WAN edge routers.

This is indeed a valid and important point, and running iBGP between edge and core routers would be one method to handle it as Giuseppe describes, but it may not be necessary. For instance, in OP it's noted "default-information-originate" but w/o "always" keyword, shouldn't default be withdrawn if and when, as Giuseppe also describes, (pt. 2) "connection fails"?

Another approach, might be if dual links between cores and edge are used, place WAN pairs into same subnet so that the two WAN edge routers don't need to transit (logically) across cores at L3.

Giuseppe/Joseph:

Thank you.

Im glad you mentioned fault tolerance, because its the first thing on my mind.

Lets go into fault scenarios, but please read what I wrote carefully. I know its a bit long and I really do appreciate it, and remember this first:

1.) Remember that the edge is advertising ONLY a default route (read original post), to the OSPF routers, so no worries about the core handling the full Internet table.

2.) Also core 1 and edge 1 is the primary path, with core 2 and edge 2 having OSPF cost bumped up to force it to be a secondary path. So all traffic, in a normal situation, wll always go to core 1 and edge 1 and out to the Internet.

FAILURE SCENARIOS:

Lets assume I have an iBGP neighbor established between edge 1 and 2 AND an OSPF neighborship as well.

1. edge 1 internet link fails:

what happens: edge 1 loses routing table. edge 1 also loses default route, which is statically configured, with the next hop being the internet link. So, edge 1 will no longer advertise the default to core 1 (default-information originate), and core 1 will be forced to use its higher cost default route learned from the crosslink to core 2, and core 2 will default to edge 2, and edge 2 defaults to the internet.

2. edge router itself fails.

What happens: the exact same thing with regard to core 1 losing its default route and defaulting to core 2.

3. The link between edge 1 and edge 2 fails.

What happens: NOTHING at all. The primary traffic path is hard-coded through core 1 and edge 1. edge 1 will lose its iBGP and OSPF neighborships with edge 2, but big deal, it doesnt cause any outage. In the scenario you described regarding this link failure, the internet link AND the crosslink between edge 1 and edge would have to have failed for the traffic to be forced down the "U" to core 1...but in that case, core 1 loses default to edge 1 and will default to core 2 anyway (THIS WAS YOUR POINT, JOSEPH abouth the default route being withdrawn...).

4. Link between edge 1 and core 1 fails:

What happens: core 1 loses default again and fails over as in case 1 and 2.

5. Core 1 itself fails:

what happens: core 1 and 2 are part of a VRRP group. Souser traffic defaults to core 2, then edge 2, the out to the Internet.

I wont go trough failover scenarios for core 2 and edge 2 because they are the secondary path. Nothing will happen in terms of effecting traffic.

Did I get all this right? if not, please be specific about where I went wrong.

So, to summarize and put this issue to rest, OSPF between edge 1 and edge2 is really ONLY to create the iBGP session because I am sourcing the loopbacks, right?

Were the iBGP neighborship established ona physical /30 link, you of course would not need OSPF or statics - it would be a directrly conencted neighbor, right?

[EDIT] I just thought of something. If my faiilover scenrios are correct, take note that the crosslink between edge 1 and edge 2 never gets used! ANY failure on the primary path will cause failover at core 1 to core 2. There is no instance in which a failure in the primary pth will cause core 1 to STILL send traffic to edge 1 and put edg 1 in a situation where it needs to use its crosslink to edge 2. In theory, we dont need the physical crosslink between edge 1 and edge 2, and we dont need iBGP or OSPF.

Do I have a point??

[EDIT]

Hello Joe,

this is a very interesting discussion as often happens.

>> Also, the edge routers are accepting full internet routes but NO default route. The default is advertised statically on each edge router.

This means you have a default static route pointing to eBGP next-hop/outgoing interface.

the iBGP session is not used if the two Internet full tables are exactly the same and this happens only if the ISP is only one.

If the edge routers connect to two different ISPs each one provides its own full table but they are different in some parts!

So there would be a potential for some prefixes to have a best route via edge2.

Anyway, it looks like the ISP here is only one.

There is still a fault event to be considered here:

the BGP session can be turned down by a protection feature or by a configuration error without having a fault on the link.

(I mean on the service provider side: everything is possible)

example of protection feature:

maximum-prefixes I saw it many times in an internet exchange point where my customer connects.

So if the eBGP session is down edge1 will try to use the iBGP session.

Hope to help

Giuseppe

Giuseppe:

Youre right, it is one ISP.

And you know, I thought of that failure scenario after I submitted my response and I knew that would be a case in which the iBGP link would be used, but I didnt want to do another "edit" and confuse matters even more. lol

So, as I said befre, except for the case in which we have the kind of BGP session failure that we just noted, that edge router crosslink would never be used, right? and iBGP and OSPF are not needed either on that crosslink, right?

"4. Link between edge 1 and core 1 fails:

What happens: core 1 loses default again and fails over as in case 1 and 2.

5. Core 1 itself fails:

what happens: core 1 and 2 are part of a VRRP group. Souser traffic defaults to core 2, then edge 2, the out to the Internet. "

But is that what you want? Since you've defined edge 1 as primary path (perhaps more bandwidth than edge 2 - otherwise why not use both edge 1 and edge 2), and it (edge 1) still has a valid link to Internet, and since you have cross link, shouldn't traffic transit edge 2 for traffic from edge 1 and core2?

"Were the iBGP neighborship established ona physical /30 link, you of course would not need OSPF or statics - it would be a directrly conencted neighbor, right? "

You could use the peer address on the cross link, but you risk losing the iBGP peer if that link fails. If you use a loopback, iBGP would remain up as long the peer could be reached via either the direct cross link or via the core routers. Using a loopback, though, requires we make the two edge routers aware of it, and this is easier done with OSPF.

Another issue, even if you use the cross link peer IP address, iBGP, by default (unless you use next-hop-self), passes along the next hop address for the iBGP routes. This too would require both edge routers to "know" the other edge router's eBGP subnet, which could be communicated via OSPF.

Your thinking of single points of failure, but what about double points of failure? Although unlikely, suppose primary eBGP fails and secondary core-edge fails, how do you best get core 1 to transit edge 1 to/from edge 2? With OSPF, core 1 would "know" of edge 2's defualt (via edge 1) and edge 2 would know of core 1's internal routes (also via edge 1).