cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1348
Views
10
Helpful
23
Replies

Design Question - Redundancy between 2 Routers (CEs)

4everlearning
Level 1
Level 1

Hello Guys,

I have a question regarding the link redundancy between 2 CEs, let's say we have the following scenario 

 

 CE1 _ __ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _  10G link  running EBGP _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _  PE1

   _                                                                                                                                                                                         _ 

  One 10G link L2 tunk, and Three 1G links L3 between CEs                                                                                        _

   _                                                                                                                                                                                          _   

  CE2 _ __ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _  10G link  running EBGP_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _  PE2

 

2 CEs connected to 2 PEs with 10G for each PE-CE connectivity using EBGP (L3 connectivity). now between the CEs there is One 10G Link used as L2 trunk and Three 1G Links used as L3 Trunk. 

Now let's say the link between PE1-CE1 goes down, all the traffic will be shifted to the link between PE2-CE2, now when CE2 receives the traffic from EBGP, will it use the L2 trunk or L3 trunk to send the traffic to CE1 using the L3 links through iBGP ?

 

Just to give an example, let's say there is an X device and Y device behind the CEs:

The X device is connected to the CEs using L2 connectivity (Multiple VRRP groups with CE1 being the Active for 50% and CE2 being the activity for the other 50%). Also we have the Y device which is connected to the CEs using L3 connectivity (OSPF), and then the CEs connect to the PEs using EBGP.

so now let's assume the ebgp link between CE1-PE1 goes, down, the traffic will be shifted and sent through CE2-PE2, now when CE2 receives the traffic, would CE2 send the traffic, in which it is acting as the Master for, directly out to internal network and only the rest of the traffic (the 50% where CE1 is the master for) will be sent through the ibgp link ?

or would the ibgp link carry all the traffic to CE1 anyway, and then CE2 will send it's concerned traffic to internal network and CE1 will send it's concerned traffic to internal network. 

We are using Cisco asr 1006 as CEs and we are connected to the same provider 2 CEs to 2 PEs. 

I would really appreciate if someone gives be a better idea from design point of view on how traffic moves between the links between CEs in case there is a failure between one of the CE links to external devices (PEs, or X and Y devices in our example).

 

Many thanks for the help. 

2 Accepted Solutions

Accepted Solutions

so let me make sure i understand things right, outbound traffic has nothing to do with inbound traffic in this case, so even if internal network use CE1 as Master for outbound traffic, CE2 can still send inbound traffic to the internal network and does not have to send it through CE1, right ? 

Basically yes as long as CE2 has directly connected interfaces or sees the OSPF routes as the preferred routes to non connected networks then traffic should be sent direct.

Quick way to check is to look at the IP routing table on CE2 to see what it thinks are the best routes for the internal networks.

But that is only for inbound traffic.

In terms of whether you have enough bandwidth it comes down to what Joseph was saying about the outbound traffic. If CE1 is the VRRP master and the CE1 to PE1 link fails then traffic will have to traverse the link to CE2.

Again as Joseph was saying with HSRP/GLBP you can track the CE to PE interface and if it fails then you can switch the active router to the other CE so all traffic is sent direct.

I had a very quick look at VRRP Cisco document and it suggests you can track an interface but i have never done it with VRRP so can't say for sure how it works.

In terms of your OSPF routing we don't have enough information to say what would happen. For example are you receiving BGP routes from the PEs and then redistributing them into OSPF or doing something different ?

So as far as i can see you need to take into account outbound traffic but your inbound traffic should not be an issue.

But like i say worth checking the IP routing table to be sure.

Jon

View solution in original post

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Ah, just saw all the new posts.  When you mentioned VRRP, I got in my mind outbound traffic, totally overlooked you were asking about inbound.

 

However, Jon picked up on that, and as he notes, inbound traffic shouldn't transit the other CE, it should go directly to the destination (this is implied by using a FHRP [oh, and VRRP isn't exactly like HSRP, although both are FHRPs]).

 

As whether your links between the CE are enough, well again, inbound shouldn't transit the other CE.  Outbound, because VRRP might still split traffic, and that's why I mentioned router redirection, tracking and IGP route injection, all to avoid the need for outbound traffic to transit the CE without an active path to its own eBGP peer.

 

Also keep in mind, on ingress your bandwidth limitation will be your single remaining CE-PE link.  For egress, you've lost half your outbound bandwidth.  So any traffic that would normally transit the CE with the failed path to its PE is going to go to the remaining CE.  Again, whether some or all of the traffic starts with the failed path CE depends on your internal routing and/or FHRP setup.

 

As you're using next hop self, even though iBGP is only peering on the L3 Etherchannel (for redundancy, I would suggest peering iBGP on loopbacks), I believe the L2 trunk would be used if your IGP sees it as the better path to the other CE's loopback.  This because generally IGPs, such as your OSPF, have a better AD than iBGP.  (If you don't use loopbacks, I believe next hop would be the interface IP, and your IGP would then see the L2 as the better path.)

 

NB: I don't work much with BGP, so I'm not 100% sure what iBGP shows as next hop for peering on the interface with and without an loopback (assuming the latter is also in your IGP).

View solution in original post

23 Replies 23

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

iBGP, usually relies on the local routing to determine how traffic gets to its next hop.  It also depends on what IPs you use to peer the two CE routers on and what you advertize as the next hop.  For example, you could peer your two CE routers on interface IPs on the same subnet, or you might peer them on loopback IPs.  The next hop could be the PE's interface IP or it might be the CE loopback.  How were you going to setup your iBGP?

You also mention VRRP.  I know HSRP and GLBP support tracking, not sure about VRRP.  But if VRRP does, with failure of a CE path to the PE, ideally you would want to shift your gateway to the CE that has the active path outbound (to avoid bouncing traffic off the CE w/o an outbound path).

 

Or if VRRP supports duplicate IPs on the same subnet as HSRPv2 and GLBP do, you can have the CE router issue redirects to the other CE's virtual gateway.

You mention OSPF.  If you have other OSPF routers behind your CEs, how you inject routes, can be changed with loss of the a CE<>PE path.  If that's done, all traffic should flow just to the CE with the active outbound path.

 

Worst case would be, half your traffic is still sent to CE w/o path to its PE.  It then needs to send that traffic to its peer CE.  This would be automatic with iBGP, but path used depends on your configuration.  You could insure a path, or you can use "default" interior routed path.

 

Best case, CE that loses its path to its PE no longer receives any outbound interior traffic.

Thanks for your reply. Actually im making an audit on existing network and i am trying to provide a list of best practises to be applied.

we are using Cisco asr 1006 to connect to PEs of one provider and here is the scenario:

1) Between PEs and CEs we have 1+1 10G links used for some services and then another 3+3 aggregated 1G ports used for other services, and between PEs and CEs we are running EBGP peering directly with interfaces (not loopback)

2) Between the 2 CEs we have 2 aggregated 10G L2 trunk, and 3 aggregated L3 ports running iBGP.

3) the iBGP peering on the L3 etherchannel IPs which are IPs on the same subnet (not loopback)

4) iBGP is using next-hop-self 

5) CEs advertising exactly the same networks to PE1 and PE2. 

6) internal network connectivity as i explained we have some L2 connectivity (which is the majority) using VRRP multiple group with CE1 being the active for some routes and CE2 being the active for the others (VRRP is exactly like HSRP it's just not Cisco proprietary). 

for the sites that run OSPF with internal network, there is a separate aggregated 1G links (etherchannel) that run OSPF between CEs and use the same area with internal network.

 

Now my first observation is

1) is 3+3 aggregated 1G links enough for ibgp connectivity between CEs ? 

2) isn't 2+2 aggregated 10G links too much for L2 trunk between CEs ?

 

Thank you,

 

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Ah, just saw all the new posts.  When you mentioned VRRP, I got in my mind outbound traffic, totally overlooked you were asking about inbound.

 

However, Jon picked up on that, and as he notes, inbound traffic shouldn't transit the other CE, it should go directly to the destination (this is implied by using a FHRP [oh, and VRRP isn't exactly like HSRP, although both are FHRPs]).

 

As whether your links between the CE are enough, well again, inbound shouldn't transit the other CE.  Outbound, because VRRP might still split traffic, and that's why I mentioned router redirection, tracking and IGP route injection, all to avoid the need for outbound traffic to transit the CE without an active path to its own eBGP peer.

 

Also keep in mind, on ingress your bandwidth limitation will be your single remaining CE-PE link.  For egress, you've lost half your outbound bandwidth.  So any traffic that would normally transit the CE with the failed path to its PE is going to go to the remaining CE.  Again, whether some or all of the traffic starts with the failed path CE depends on your internal routing and/or FHRP setup.

 

As you're using next hop self, even though iBGP is only peering on the L3 Etherchannel (for redundancy, I would suggest peering iBGP on loopbacks), I believe the L2 trunk would be used if your IGP sees it as the better path to the other CE's loopback.  This because generally IGPs, such as your OSPF, have a better AD than iBGP.  (If you don't use loopbacks, I believe next hop would be the interface IP, and your IGP would then see the L2 as the better path.)

 

NB: I don't work much with BGP, so I'm not 100% sure what iBGP shows as next hop for peering on the interface with and without an loopback (assuming the latter is also in your IGP).

Jon Marshall
Hall of Fame
Hall of Fame

I think i may be interpreting this slightly differently than Joseph but you seem to be talking about traffic coming from the PEs going to the internal network.

If so VRRP is irrelevant because it only comes into play for traffic from the internal network out via the CEs to the the PEs.

For inbound traffic CE2 should just forward the traffic directly to the internal devices, assuming it has routes to the internal network. Nothing to do with VRRP or BGP.

For outbound traffic Joseph has explained what happens with VRRP and it 's not clear what happens with OSPF without more details.

One other thing. Not sure what you mean by a L3 trunk ?

Jon

Dear Jon,

Thanks for your reply, kindly check my previous post, i appreciate your feedback and input as well. 

 

Thanks,

 

I did and you still seem to be talking about inbound not outbound traffic so the same applies.

Only traffic going outbound is relevant in terms of either BGP routes or VRRP unless you are running VRRP between the CEs and PEs which you aren't by the sounds of it.

That said as you haven't clarified i'll leave this thread to Joseph as he seems to understand what you are asking better than me.

Which happens quite often smiley

Jon

Thanks Jon, 

Regarding the inbound and outbound traffic. 

most of the outbound traffic will be going out using VRRP, and there is 2+2 10G L2 trunk between the CEs so i believe the existing links between the CEs will be enough to carry the outbound traffic in case of any failure. Even for the devices that run ospf between the internal network and CEs, there is a L3 aggregated links running  ospf on the same area.

 

Now my main confusion is on the inbound traffic:

to summarize my confusion, let's say the link between PE1 and CE1 fails, and it was carrying 3 gig of traffic, now those 3 gig will be shifted to the link between PE2 and CE2 (let's assume the link between PE2 and CE2 was already carrying 2gig traffic), so the 3 gig will be added to the 2 gig which will be a total of 5 gig traffic going through PE2 and CE2 link.

Now let's assume 3.5 gig of those 5 gig traffic will be forwarded to internal network through CE 1 (since it is the active VRRP for them, and 1.5 gig will be forwarded to internal network through CE2 since it is the active VRRP for them).

so when CE2 receives the 5 gig traffic will it forward the 1.5 gig traffic directly to internal network and then only the 3.5 gig traffic will go through the ibgp link between the 2 CEs to CE1 ? or would all 5 gig traffic go through ibgp link between the 2 CEs to CE1 ? even though only 3.5  gig will be forwarded by CE1 to internal network ?

 

Now let's assume 3.5 gig of those 5 gig traffic will be forwarded to internal network through CE 1 (since it is the active VRRP for them, and 1.5 gig will be forwarded to internal network through CE2 since it is the active VRRP for them).

This is the bit i am trying to clarify.

Why would the traffic from CE2 be sent to CE1 to get to the internal network ie. CE2 is connected to the same internal network so it should just send it direct unless you are saying CE2 will have most of the internal network routes pointing to CE1 ? 

So -

1) VRRP is only relevant to the internal network ie. you don't have VRRP between the PEs and CEs so it doesn't matter which CE is the VRRP master

2) BGP routes. If CE2 sees the best path to the internal network via CE1 then yes it would send traffic across the link but i would have thought CE2 would have either directly connected or OSPF routes pointing directly to the internal network and not via CE1.

It may be that i am misunderstanding how you have this setup but i can't see from what you have described why CE2 would not simply use it's LAN interface for any inbound traffic.

Jon

Thanks Jon, 

Things are starting to make sense to me now, i just never thought of things from design point of view. 

btw, CE2 has directly connected/ospf to internal network. 

so let me make sure i understand things right, outbound traffic has nothing to do with inbound traffic in this case, so even if internal network use CE1 as Master for outbound traffic, CE2 can still send inbound traffic to the internal network and does not have to send it through CE1, right ? 

 

if this is the case, then is 3+3 1 G aggregated L3 etherchannel enough between the CEs even though we are running 10G links with the PEs? since both CEs have connectivity to internal networks and they can send traffic directly without going through iBGP link ?

 

 

 

so let me make sure i understand things right, outbound traffic has nothing to do with inbound traffic in this case, so even if internal network use CE1 as Master for outbound traffic, CE2 can still send inbound traffic to the internal network and does not have to send it through CE1, right ? 

Basically yes as long as CE2 has directly connected interfaces or sees the OSPF routes as the preferred routes to non connected networks then traffic should be sent direct.

Quick way to check is to look at the IP routing table on CE2 to see what it thinks are the best routes for the internal networks.

But that is only for inbound traffic.

In terms of whether you have enough bandwidth it comes down to what Joseph was saying about the outbound traffic. If CE1 is the VRRP master and the CE1 to PE1 link fails then traffic will have to traverse the link to CE2.

Again as Joseph was saying with HSRP/GLBP you can track the CE to PE interface and if it fails then you can switch the active router to the other CE so all traffic is sent direct.

I had a very quick look at VRRP Cisco document and it suggests you can track an interface but i have never done it with VRRP so can't say for sure how it works.

In terms of your OSPF routing we don't have enough information to say what would happen. For example are you receiving BGP routes from the PEs and then redistributing them into OSPF or doing something different ?

So as far as i can see you need to take into account outbound traffic but your inbound traffic should not be an issue.

But like i say worth checking the IP routing table to be sure.

Jon

Many thanks Jon, now it's clear to me. 

I would double check the OSPF routing but i beleive we are redistribution BGP routes into OSPF, but i would cross check and let you know.

"In terms of whether you have enough bandwidth it comes down to what Joseph was saying about the outbound traffic. If CE1 is the VRRP master and the CE1 to PE1 link fails then traffic will have to traverse the link to CE2." 

in this scenario, the traffic will traverse to CE2 through the L2 trunk between CEs (Not the L3 aggregated links, right ? since VRRP is L2 technology. 


 

 

in this scenario, the traffic will traverse to CE2 through the L2 trunk between CEs (Not the L3 aggregated links, right ? since VRRP is L2 technology. 

Correct, the traffic should use the L2 trunk not the L3 links <-- this is incorrect. It assumed VRRP failover but there is no tracking so there is no failover. The traffic would use the L3 links.

Edit - just seen your comment about checking the BGP configuration so rather than confuse the issue i will wait until you confirm one way or the other.

Jon

Hi Jon,

No confusion at all, you and joseph have been very helpful and i really was confused before this discussion. 

I am out of town and will check the BGP configuration right after the weekend. 

but out of topic for now and just for my knowledge, your previous post stated an interesting point regarding the internal bgp redistribution. 

in our case, if there was redistribution between BGP and OSPF, is there a need to redistribute internal as well?

Since both CEs are adjacent  through OSPF, and both have connectivity to internal network; 

In case CE1-PE1 fails, the traffic will be sent through CE2-PE2 link and then redistributed to OSPF on CE2, and CE2 should be able to send them to internal network or to CE1 through OSPF (not iBGP) since OSPF routes have better AD than internal iBGP routes. 

Am i right here, or we still have to configure redistribute internal BGP if that is the case.

 

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Redistribution requirements depends on what each routing protocol needs to "know" about the other.

 

In many setups like yours, BGP would just generate an OSPF default, so any OSPF taffic with an unknown destination IP would come to the CEs.

 

Conversely, the OSPF routes are picked up (not redistributed) by BGP although aggregation of those routes is often done.  (NB: BGP might pick up internal routes using static route statements to null.)

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: