On flying through my studying this weekend i came across a very interesting (well i think it was interesting) and i thought i would share the scenario for everyone to try to see if they get the answer to it spot on. Scenario
PE A connects to CE A
PE B connects to CE B
CE A connects via LAN to CE B
BGP is the routing protocol for the vpn and CEA&B are in the same VPN. There is an IBGP connection between CEA & B also.
Configure this scenario to avoid routing loops. What would you do and why is my question here. The CE routers are in AS 87 and the SP environment is AS 65001.
Really a teser. I do not see the reason why a scenario like this will be, more especially since the CEs are connected on the same LAN.
However, AS-Overide will definitely be needed for communication across the SP backbone. Weights would need to be configured to influence the traffic path and ensure a loop free network.
Robert, what do you think?
well there are two ways of doing this If as you say you use as-override then you will have to use soo to stop the loop happening. Now if you do go down this scenario break the ibgp connection between the 2 CEs - Q: can CE1 ping CE2?? Answer is absolutely 100% NO. This is my dillema - i have always done as-override and soo in this scenario - however the only way to get this work under failure scenarios as well is to use allowas-in on the CE routers. Now im left with 5 years of experience being doubted 1 day before i go to sit the CCIE SP lab. Im well confused.
I'm a little lost in your scenario, so let me try and clarify things.
I will agree with you that if you use AS-override, you should use SOO. However, I was thinking that the 2 CEs are the only CEs in the vpn. If so, then the SOO is not useful. Once you use SOO, the 2 CEs cannot use the MPLS core to communicate, as routes from one CE are not readvertised back to the second CE. Therefore AS-Overide and SOO is not an option.
Like you mentioned, you could use allowas-in on the CEs with extra configuration on the PEs. But you could still have the possibility of routing loops.
I think to prevent your routing loops, weight needs to be configured. This way, you can manually set on each router the preferred path, and the alternate path.
In summary, I think you have two options. Either AS-Override on the PE, or Allowas-in on the CEs. But in either case, you will need to configure weights to influence the paths.
What do you think?
hmmm. you seem to be thinking that as-override and soo is only used for traffic coming into a dual homed site (from what i read) - SOO is used to stop traffic from a dual homed site coming back into itself (if im not mistaken). Using weight will not help as this doesnt stop any loops (ever) its all down to the as-path issue that you strip when using as-override. Also using allowas-in doesnt require any more config on the PE this command is a CE only command in this scenario.
This to me is a really weird one as it kinda goes against good design practices. You would tend to say that "how would the ibgp connection go down in the first place" - but hey it could easuly go down if the customer thought "yeah we can upgrade this switch as the traffic will go through the SP vpn". WRONG!! I really fail to understand how weight in BGP stops a routing loop my friend, maybe im missing something - i hope i am!.
This is a really good question. Thanks for getting me to focus on it. So here's some thoughts that may or may not be useful.
I'm not sure that either approach is "best practice" from a technical standpoint. I think it all just depends.
Here's some pros/cons I can think of. Maybe others have some additional to throw in.
By using override/soo, the provider maintains some measure of control against customer misconfiguration and certainly eliminates loops. Intra-site traffic utilizes IGP routes rather than VPN/EGP.
The downside is the behavior you are seeing. So, the IBGP peering, loopback to loopback must be maintained. Customer must have some level of redundancy in their network to ensure that there is an alternate, non-vpn path.
Allow-as doesn't have the failover issue.
However, by using allow-as on these CE, you will prefer the VPN path over the LAN path since each CE is learning the other CE's local networks via eBGP. This may not be desired behavior either.
Also, AS prepend used elsewhere by the customer may break AS path limit configured for allow-as on CEs and thus allow loops.
Other thoughts, anyone?
Good answer although - "However, by using allow-as on these CE, you will prefer the VPN path over the LAN path since each CE is learning the other CE's local networks via eBGP. This may not be desired behavior either" This isnt true im afraid as the BGP process will install the ibgp learned router i.e. the CE's loopback due to the shorter AS path and origin being ibgp.
The control manner is exactly correct - an SP would want to protect themselves so would use SOO. However the day will come when a cusomter will break that IBGP connection and then realise that they cant use the SP connection - oh man im glad i left the SP environment because i dont want to try to explain to a customer that we (SP) are intetionally breaking their network.
THanks all for responding - tho it still doesnt solve my problem - Tomorrow i MAY face an issue like this in the LAB. What to do? Dont Know!!
well the answer is: this is how BGP is designed to work! If you split an SP network (f.e. by killing all route reflectors) you can not expect to route traffic across other ASs to get internal reachability. This is against the interior BGP design. You simply can only achieve, what has been designed into a protocol.
(Side note: you could also ask to filter one single network from an OSPF database within a single area - this is not what the design allows for)
Second: in case you have iBGP you (usually) also have an IGP, which should be responsible for internal routing (thus AD 200 for iBGP).
So the proper configs would be f.e. with AS100 in the SP and AS 65001 for the customer:
ip vrf CE11
route-target export 100:101
route-target import 100:101
ip vrf forwarding CE11
ip address 10.1.11.1 255.255.255.252
router bgp 100
address-family ipv4 vrf CE11
neighbor 10.1.11.2 remote-as 65001
neighbor 10.1.11.2 version 4
neighbor 10.1.11.2 activate
neighbor 10.1.11.2 as-override
neighbor 10.1.11.2 advertisement-interval 5
neighbor 10.1.11.2 filter-list 1 in
neighbor 10.1.11.2 route-map CE11sooIN in
maximum-path ibgp 2
network 10.1.11.0 mask 255.255.255.252
ip as-path-list 1 permit ^65001(_65001)*$
route-map CE11sooIN permit 10
set extcommunity soo 100:11
!-----------------Configuration of CE11 -------------
ip address 126.96.36.199 255.255.255.255
description interface to PE1
ip address 10.1.11.2 255.255.255.252
description interface to CE12
ip address 192.168.1.1 255.255.255.0
router ospf 10
network 188.8.131.52 0.0.0.0 area 0
network 192.168.1.1 0.0.0.0 area 0
router bgp 65001
neighbor 10.1.11.1 remote-as 100
neighbor 184.108.40.206 remote-as 65001
neighbor 220.127.116.11 update-source Loopback0
network 18.104.22.168 mask 255.255.255.255
PE2 and CE12 would be configured accordingly with IP addresses adjusted.
The better approach from a BGP prespective is to have a separate AS for each customer location. Then you do not even need as-override. Just plain good old BGP design between multiple ASes.
My 2 cents
Right, but since we're not talking about separate customer locations, only separate CE routers with a lan connection between them, does the the multiple AS approach make sense? It sounds like you are suggesting a eBGP peering between the CEs rather than an iBGP - but I'm probably misunderstanding.
Also, I was working on the assumption that with the allow-as approach, the CE would prefer VPN/eBGP on Admin distance alone. I need to lab this up as I'm concerned that the iBGP peering Rob was seeing was over the VPN rather than the LAN.
Rob's concern with customers is a big one - at least from my perspective. I want to echo that concern. So often MPLS is marketed as a superior replacement for L2 WAN networks. And it is not at all uncommon for customers to want to use the VPN as an additional backup path between CEs. The issue is not a technical one, since you could always design around it. But rather a customer expectation issue. We can argue that Sales shouldn't market that way, but when you come in after the MPLS network is already purchased, or you don't work for the carrier, the pain is very real.
A few things here folks -
1. Martin you are going down the exact same path as me - this is the way it is designed to work so there ya go. However a customer will NEVER see it this way as they expect to be able to have inter site communication over the mpls vpn afterall what are they paying for. Also i do not see any reason for your input filter in your confiuration - to me it makes no sense at all.
2. Michael you have hit the nail on the head - I labbed this up and thought oh ok BUT why isnt the EBGP link being preferred to reach CE1's loopback from CE2?? I then realised that despite EBGP being AD 20 that the AS-path and origin is better for the IBGP connection.
Now on a par here - i could argue both points - why would a customer have this stupid setup where they could have a failure in their LAN and cause several other lans connected to each CE to stop communicating with each other - the answer is simple - because the can and because they are the customer. Dumb as it may be but hell ive been there far too many times.
2. Now as an SP i would never ever allow the customer to control the routing BUT in this case you actually have got to allow them to do it with allowas-in.
not my cup of tea i must admit but hey - if i get hit with this scenario tomorrow i will promise you the proctors ear will be well used ;-) AND i will prolly go for the as-override and the SOO method as that is what Cisco say is the correct way to do it - albeit in my findings it is the total WRONG way to do it.
Thank you all fo rthis discussion. It is nice to see that it is not just me who thinks WTF?
Hello Rob again,
Firstly, that was a typo, I meant to type "Like you mentioned, you could use allowas-in on the CEs with NO extra configuration on the PEs".
Then to the question at hand.
From the customer perspective, he has two routers that need to communicate across the SP backbone.
From the providers perspective also, there are 2 CEs that he is given an MPLS VPN service to. And these 2 CEs have a backdoor connection. The LAN connection from the SP perspective can be seen as a backdoor link.
Looking at it from these perspective, I will not consider the scenario as a multihome site. Hence, to achieve the sets of objective above, SOO is not an option. Once you use SOO, the scenario is that of multihome site, in which the CEs do not need the SP core to communicate under any circumstances.
Therefore, I will ask the question, how is possible to use AS-Override, and still avoid routing loops. To answer these, we might want to see what happens with AS-override without SOO. I think the main problem here is that routes from the site can be readvertised back to the site. And this readvertised routes would be preferred over any other learnt route, as eBGP has the best AD (20).
I think another key point to note is that BGP advertises only the best route and active route in the routing table. I thought of weights, since it is locally significant, as a way of controlling BGP path decision. If weights for the varous routes and neighbors can be configured, in such a way to mirror primary and floating routes, then I think there might not be a routing loop. From the scenario, each route can have a maximum of two paths for a prefix. The weights should therefore be configured in such a way that the primary and secondary path are consistent on all the routers.
It does not sound optimal to me, and I have not tested it, so I do not guarantee that it will work. It is just an idea. If you can lab it up, it will be appreciated.
Sounds like you going for the lab tomorrow. If so, I wish you the best.
I think you should check the following thread, Paresh explained well the routing loop issues with the Allowas-in.
In case the link does not open, thread title is BGP redundancy, under WAN Routing and Switching, last update on Jan 27.
Maybe the lesson to take to the lab is that there limitations to either approach.
I'm thinking that if you get a requirement that you think should use AS pre-pending, you now know there's a dependency where allow-as is used if you go that route. And if there's a requirement for failover, you gotta use allow-as. Dunno, just hoping there's a silver lining here.
Anyway, good luck tomorrow! Of course, if you pass, I'll need to get off my backside and schedule my next attempt. ;-)
the eBGP suggestion is for different locations of the customer. ONE location should be setup with iBGP between CEs. My point was just, that a separate AS per customer location does remove the requirement for as-override and thus SOO in the SP network - hence simplifying the SP BGP.
Second AD does not play a role in BGP path selection - only after best path is selected (external or internal) to decide whether to insert the best path into RIB or not. Side note: this is the reason for RIB failre indication in the BGP table - best path was selected but could not be inserted into routing table.
I totally agree to the comment that wrong customer expectation might be an issue - but hey, did you ever implement QoS? WOW there are some rumors and misconceptions around QoS, which make the ones about MPLS VPN a piece of cake ;-)
(SP technician: "What should be prioritized?" Customer: "Oh, I thought you could tell me!")
@Rob: the AS filter just is to avoid eBGP multihop "testing" by customers across different sites. This could also lead to suboptimal routing ;-)
And a final note: bad design does create problems. If the requirement exists, that LAN connectivity in a site should be backupped by the MPLS VPN, then BGP is the wrong choice. We are talking about backdoor links and that might be an approach as well.
Working on the idea of not using SOO, I think filtering based on communities can be used to ensure there will be no routing loops. Lets say for CE1, all the locally injected routes into BGP are tagged with community 1:1 and for CE2 they are tagged with 2:2. CE1 is connected to PE1 and CE2 is connected to PE2.
On PE1, have a route-map to filter all routes with community 1:1 to CE1. Also, PE2 should filter out routes with 2:2 to CE2. With this scenario, I do not expect a routing loop. I assume though that CE1 and CE2 do not have any other BGP sessions.
Any comments on this?
BGP AS path processing would still prevent CE-CE connectivity through the MPLS VPN. Additionally you would need allowas-in on the CE and then routing loops could occur ... so you need additionally filters on the CEs and the whole thing gets messy and complicated.
Basically one trys to implement something which was not intended by the initial BGP design.
I am sure it can be done ... turn off standard BGP loop detection in PE and CE and configure filtering manually through communities and route-maps.
Or place very CE in its own AS and use LocPref for proper path selection.
In any case it gets painful from an operation point of view.
new idea ... keep the iBGP session between CE routers up and running through the MPLS VPN.
So I would assume floating statics for the Loopbacks of the CE routers would do the trick.
ip address 10.0.0.1 255.255.255.255
ip address 10.1.1.1 255.255.255.0
description to PE1
ip address 10.1.2.1 255.255.255.252
ip route 10.0.0.2 255.255.255.255 10.1.1.2
ip route 10.0.0.2 255.255.255.255 10.1.2.2 190
router ospf 10
network 10.0.0.1 0.0.0.0 area 0
network 10.1.1.1 0.0.0.0 area 0
router bgp 65000
neighbor 10.0.0.2 remote-as 65000
neighbor 10.0.0.2 update-source Loopback0
redistribute ospf 10
PE and other CE along the lines of the config above.
This should do the trick.
No worries. I thought I was misreading. Just wanted to be sure. :-)
I see where I was wrong on the AD issue. AD won't come into play until after the BGP decision process. doh!
I'm with you on your point about QoS for both provider and the customer. Especially when the customer won't pay a premium for a well thought out SLA. Also known as, How does my gold-level EF/Exp 3 RTP traffic get priority over your gold-level EF/Exp 3 RTP traffic in the SP core, etc. When all traffic starts to look the same, nobody gets priority.
Not to digress anymore on the WTF discussion, but I do spend a fair amount of my day working at educating previously L2VPN customers to the pros/cons of L3 VPNs, particularly QoS and backup links. I usually get called after something breaks or the network is asked to perform "un-natural acts" So this is a bit of a sore spot for me.
Anyway, I need to play in the lab with some of the ideas posted here. Clearly, I need to do some add'l homework.
So far I have:
Floating static for iBGP over the VPN. (If iBGP is a requirement)
Any others, anyone?
I see your problems Michael. People build a Toyota and you are then in charge to let it look and drive like a Ferrari ...
So your challenge is: "Pimp my network!" ;-)
Have you ever tried to get a job at MTV?
Have fun! Martin