Hub & Spoke VRF routing question

[fsebera]

I have a Hub and Spoke MPLS VPN topology, meaning spoke to spoke traffic MUST pass through the hub. : I need to have 2 Hubs: HUB1 and HUB2 HUB1 will be primary for spoke-A, secondary for spoke-B HUB2 will be primary for spoke-B, secondary for spoke-A HUB1 and HUB2 will have an eBGP peering link external to the MPLS cloud for failover purposes. All sites will have unique AS #'s. : HUB1 will connect to PE1 HUB2 will connect to PE2 Spoke-A and Spoke-B will connect to PE3 : In a Hub and Spoke topology I know I need an ingress & egress vrf on each HUB CE-PE peering pair as this is what forces spoke-to-spoke traffic to flow through the hub. : If I want PRIMARY traffic flows of Spoke-A to HUB1 & Spoke-B to HUB2, do I need one (1) additional vrf (total of 3 vrf's) on the HUB CE-PE peering pair to support the "Load Sharing" topology or 2 additional vrfs (total of 4) per peering pair (HUB PE-CE) or can I do this with just the two vrfs (1 ingress vrf & 1 egress vrf) already configured? : Examples would be great!!!! :) Thanks again! Frank

[Vaibhava Varma · Accepted solution]

Hi Frank

This is a very interesting scenario of Hub and Spoke MPLS VPN requirement along with the the need to provide the HUB servcies in alternate redundant way..

I would put forward my understanding which can be a possible solution to this scenario. Lets look at the two requirements separately:

1. HUB & Spoke Services

We will only require 2 VRFs on the HUB-PEs viz one Hub_Ingress VRF to collect the routes from the remote spoke PE and send it towards the HUB and one Hub_Egress VRF to send the Hub_Routes as well as remote Spoke Routes from the HUB back to the MP-iBGP cloud. Having said that we would need two logical L3 peering between HUB PE-CE . One peering to be part of Hub_Ingress VRF and other Peering to be part of Hub_Egress VRF..We will have two eBGP neighbourship between HUB PE-CE ie one per VRF..

Also an important point to consider here is that because of default AS-Path Loop Prevention mechanism of BGP on the HUB-PEs under HUB_Egress VRF BGP Neighbourship config we need to add " allowas-in " command so that the spokes routes sent via HUB_Ingress VRF are not dropped when coming back to the HUB PE.

2. Alternate Redundant HUB Services to Spokes

To achive the deisgn such that Spoke 1 is services by HUB1 as Primary and HUB2 as Secondary and vice-versa for Spoke 2 we would need to design the RTs in such a way that on the remote spoke PE we can macth on the unique RT value and change the local preference of the HUB_PE1 routes under Spoke1_VRF to Higher

Value so that Spoke1 prefers HUB1 PE as Primary and similarly we need to macth on the unique RT value and change the local preference of the HUB_PE1 routes under Spoke1_VRF to Higher Value so that Spoke1 prefers HUB1 PE as Primary

So from the above points the VRF Design would be as below

HUB-PE1

VRF: HUB1_Ingress

         rd 64513:1

         route-target import 64513:200

         route-target import 64513:300

VRF: HUB1_Egress

         rd 64513:2

         route-target export 64513:100

HUB-PE2

VRF: HUB2_Ingress

         rd 64513:3

         route-target import 64513:200

         route-target import 64513:300

VRF: HUB1_Egress

         rd 64513:4

         route-target export 64513:400

Spoke-PE

VRF: Spoke1

         rd 64513:5

         route-target import 64513:100

         route-target import 64513:400

         route-target export 64513:200

VRF: Spoke2

         rd 64513:6

         route-target import 64513:100

         route-target import 64513:400

         route-target export 64513:300

Also on Spoke PE we need to apply inbound import-map to change the LP to higher value as discussed above

SPoke PE

ip vrf Spoke1

import map HUB1

!

ip vrf Spoke2

import map HUB2

!

route-map HUB1 permit 10

match extcommunity HUB1

set local-preference 500

!

route-map HUB1 permit 20

!

route-map HUB2 permit 10

match extcommunity HUB2

set local-preference 500

!

route-map HUB2 permit 20

!

ip extcommunity-list standard HUB1 permit rt 64513:100

ip extcommunity-list standard HUB2 permit rt 64513:400

Hope this provides some insight in your traffic requirements.

Regards

Varma

[fsebera]

Hi Vaibhava,

Thank you for your guidence, it is MUST appreciated

After I applied your suggestions, the spoke routes are now flapping in/out of the routing tables every 20 seconds or so.

Also on HUB1 I do not get any routes from HUB2 and HUB2 does not see any routers from HUB1.

Wondering if I need import/export between Hub1 & Hub2?

Tks

Frank

[Vaibhava Varma]

Hi Frank

Yes if we need routes exchange between the two HUB Sites then we would need to import the HUB_Egress VRF Export RTs into each other's HUB_Ingress VRF..

When the spoke routes are flapping on the HUB sites is it just the route-flap or also the BGP session flap ?

Can you share the Network Toplogy Diagram at the HUB sites to check for any possible feedback routing ?

Can you share the sh ip bgp output for a particular spoke route before and after flap ?

Regards

Varma

[fsebera]

Hi Varma,

:Sorry about the formatting, Ciscos site is not very friendly, I have pasted and repasted and still jumbles my text into one big mess.

:

No bgp neighbors are flapping at on any CE, PE or P routers.

My backup link between CE-HUB1 and CE-HUB2 is manually shut at this time to prevent additional confusion. I can send the topology map (Visio) and configs for Hubs, spokes and PE1, PE2 and PE3 (.txt) but cannot send til after 6pm EST, but here are the sh ip bgp nei between Hub1 and PE1 and futher down is the sh ip bgp before and after flap.

:

I think I may need to implement soo to prevent loops but not sure where to implement.

:

As reference:

HUB1  (WAN1) is 172.16.10 and 172.16.1

HUB2  (WAN2) is 172.16.20 and 172.16.2

Spoke1 WAN3) is 172.16.30 and 172.16.3.

Spoke2 (WAN4) is 172.16.40 and 172.16.4.

:

:

:

Here I have 5 less routes as compared with next output, about 30 seconds later

:

WAN1#sh ip bgp nei 172.17.10.2 received-routes

BGP table version is 391772, local router ID is 172.16.10.2

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,              

                  r RIB-failure, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete  

Network          Next Hop            Metric LocPrf Weight Path

*> 172.16.2.0/24    172.17.10.2                            0 65535 20 ?

*> 172.16.20.0/24  172.17.10.2                            0 65535 20 i

*> 172.16.20.1/32  172.17.10.2                            0 65535 20 ?

*> 172.16.20.2/32  172.17.10.2                            0 65535 20 ?

*> 172.16.20.252/30 172.17.10.2                            0 65535 20 ?

*  172.17.10.0/30  172.17.10.2              0            0 65535 ?

*> 172.17.20.0/30  172.17.10.2                            0 65535 20 ?

*> 172.17.26.0/30  172.17.10.2                            0 65535 ?

*> 172.17.30.0/30  172.17.10.2                            0 65535 ?

*> 172.17.40.0/30  172.17.10.2                            0 65535 ?

*> 172.18.250.0/30  172.17.10.2                            0 65535 20 ?

Total number of prefixes 11

:

:

:

Now I have 5 additional routes

:

WAN1#sh ip bgp nei 172.17.10.2 received-routes

BGP table version is 392283, local router ID is 172.16.10.2

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,             

                    r RIB-failure, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete  

Network          Next Hop            Metric LocPrf Weight Path

*> 172.16.2.0/24    172.17.10.2                            0 65535 20 ?

*> 172.16.3.0/24    172.17.10.2                            0 65535 30 ?

*> 172.16.4.0/24    172.17.10.2                            0 65535 20 65535 40 ?

*> 172.16.20.0/24  172.17.10.2                            0 65535 20 i

*> 172.16.20.1/32  172.17.10.2                            0 65535 20 ?

*> 172.16.20.2/32  172.17.10.2                            0 65535 20 ?

*> 172.16.20.252/30 172.17.10.2                            0 65535 20 ?

*> 172.16.30.2/32  172.17.10.2                            0 65535 30 ?

*> 172.16.30.3/32  172.17.10.2                            0 65535 30 ?

*> 172.16.40.2/32  172.17.10.2                            0 65535 20 65535 40 ?

*  172.17.10.0/30  172.17.10.2              0            0 65535 ?

*> 172.17.20.0/30  172.17.10.2                            0 65535 20 ?

*> 172.17.26.0/30  172.17.10.2                            0 65535 ?

*> 172.17.30.0/30  172.17.10.2                            0 65535 ?

*> 172.17.40.0/30  172.17.10.2                            0 65535 ?

*> 172.18.250.0/30  172.17.10.2                            0 65535 20 ?

Total number of prefixes 16

:

:

And here is SPOKE3 to PE3 bgp received routes :

WAN3#sh ip bgp sum

BGP router identifier 172.16.30.2, local AS number 30

BGP table version is 70541, main routing table version 70541

23 network entries using 2691 bytes of memory

23 path entries using 1196 bytes of memory

16/9 BGP path/bestpath attribute entries using 1984 bytes of memory

7 BGP AS-PATH entries using 168 bytes of memory

0 BGP route-map cache entries using 0 bytes of memory

9 BGP filter-list cache entries using 108 bytes of memory

BGP using 6147 total bytes of memory

BGP activity 192/148 prefixes, 34763/34718 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent  TblVer  InQ OutQ Up/Down  State/PfxRcd

172.17.30.2    4 65535  115057    4393    70541    0    0 00:37:26      20

:

:

Here shows Spoke3 (WAN3) just received two additional routes

:

:

WAN3#sh ip bgp sum

BGP router identifier 172.16.30.2, local AS number 30

BGP table version is 70763, main routing table version 70763

25 network entries using 2925 bytes of memory

25 path entries using 1300 bytes of memory

16/9 BGP path/bestpath attribute entries using 1984 bytes of memory

7 BGP AS-PATH entries using 168 bytes of memory

0 BGP route-map cache entries using 0 bytes of memory

9 BGP filter-list cache entries using 108 bytes of memory

BGP using 6485 total bytes of memory

BGP activity 192/148 prefixes, 34875/34828 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent  TblVer  InQ OutQ Up/Down  State/PfxRcd

172.17.30.2    4 65535  115281    4393    70541    0    0 00:37:32      22

:

:

WAN3#sh ip bgp  

BGP table version is 70923, local router ID is 172.16.30.2  

Network          Next Hop            Metric LocPrf Weight Path

*> 0.0.0.0          172.17.30.2                            0 65535 20 i

*> 172.16.1.0/24    172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.2.0/24    172.17.30.2                            0 65535 20 ?

*> 172.16.3.0/24    0.0.0.0                  0        32768 ?

*> 172.16.4.0/24    172.17.30.2                            0 65535 20 65535 40 ?

*> 172.16.10.0/24  172.17.30.2                            0 65535 20 65535 10 i

*> 172.16.10.1/32  172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.10.2/32  172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.10.3/32  172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.10.252/30 172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.20.0/24  172.17.30.2                            0 65535 20 i

*> 172.16.20.1/32  172.17.30.2                            0 65535 20 ?

*> 172.16.20.2/32  172.17.30.2                            0 65535 20 ?

*> 172.16.20.252/30 172.17.30.2                            0 65535 20 ?

*> 172.16.30.2/32  0.0.0.0                  0        32768 ?

*> 172.16.30.3/32  0.0.0.0                  0        32768 ?

*> 172.17.10.0/30  172.17.30.2                            0 65535 20 65535 10 ?

*> 172.17.16.0/30  172.17.30.2                            0 65535 20 65535 ?

*> 172.17.20.0/30  172.17.30.2                            0 65535 20 ?

*> 172.17.26.0/30  172.17.30.2                            0 65535 ?

r> 172.17.30.0/30  172.17.30.2              0            0 65535 ?

*> 172.17.40.0/30  172.17.30.2                            0 65535 20 65535 ?

*> 172.18.250.0/30  172.17.30.2                            0 65535 20 ?

r> 192.168.1.0      172.17.30.2                            0 65535 20 ?

:

:

:

Now there are gone again, about 30 seconds later

:

WAN3#sh ip bgp  

Network          Next Hop            Metric LocPrf Weight Path

*> 0.0.0.0          172.17.30.2                            0 65535 20 i

*> 172.16.1.0/24    172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.2.0/24    172.17.30.2                            0 65535 20 ?

*> 172.16.3.0/24    0.0.0.0                  0        32768 ?

*> 172.16.10.0/24  172.17.30.2                            0 65535 20 65535 10 i

*> 172.16.10.1/32  172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.10.2/32  172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.10.3/32  172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.10.252/30 172.17.30.2                            0 65535 20 65535 10 ?

*> 172.16.20.0/24  172.17.30.2                            0 65535 20 i

*> 172.16.20.1/32  172.17.30.2                            0 65535 20 ?

*> 172.16.20.2/32  172.17.30.2                            0 65535 20 ?

*> 172.16.20.252/30 172.17.30.2                            0 65535 20 ?

*> 172.16.30.2/32  0.0.0.0                  0        32768 ?

*> 172.16.30.3/32  0.0.0.0                  0        32768 ?

*> 172.17.10.0/30  172.17.30.2                            0 65535 20 65535 10 ?

*> 172.17.16.0/30  172.17.30.2                            0 65535 20 65535 ?

*> 172.17.20.0/30  172.17.30.2                            0 65535 20 ?

*> 172.17.26.0/30  172.17.30.2                            0 65535 ?

r> 172.17.30.0/30  172.17.30.2              0            0 65535 ?

*> 172.17.40.0/30  172.17.30.2                            0 65535 20 65535 ?

*> 172.18.250.0/30  172.17.30.2                            0 65535 20 ?

r> 192.168.1.0      172.17.30.2                            0 65535 20 ?

WAN3#

[Vaibhava Varma]

Hi Frank

Thanks for sharing the outputs. It seems the spokes routes which are learnt across mp-ibgp cloud are being lost at HUB Location and at Spoke 1, the Spoke 2 routes are flapping which are again learnt via HUB across MP-iBGP..

SoO would have to be considered only after looking at the network topology but in my view since at the HUB we have the dual homed CEs it needs to be applied there only but as the HUB_Ingress VRF is not exporting anything out and also there are two separate VRFs on the HUBs for Ingress/Egress without any route-leaking there should not be an issue. We can analyze more on the backup link between the the CE-HUB1 and CE-HUB2 as potential feedback routing point and can apply SoO at the two HUBs Egress VRF BGP neighbourship..

But even when backdoor link is shut there is route-flap happening..I have experenced similar issues of route-flap in test lab with overloaded BGP process killing the route cpu and causing route-flaps but at that time I did notice BGP session flap also sometimes..

Regards

Varma

[fsebera]

[fsebera]

Hi Varma,

As you may or may not be aware, I have a dual stacked environment.

It gets complicated pretty quickly. But never-the-less it's strange that IPv4 is having an

issue and IPv6 ALWAYS works flawlessly everytime, end-to-end.

Not a single problem with IPv6. For some strange reason IPv4 is failing.

Here are the configs for WAN1, WAN3 and WAN4, PE1, PE3 and P1.

Also, if you need the other configs on WAN2, PE2 and/or P2 Please let me know.

:

:

:

Here are PE1 cpu levels

:

PE1#sh proc cpu | e 0.00%

CPU utilization for five seconds: 76%/6%; one minute: 39%; five minutes: 32%

PID Runtime(ms)    Invoked      uSecs  5Sec  1Min  5Min TTY Process  

91      419316      800553       523  8.71%  4.40%  3.57%  0 BGP I/O         

113        688    1381679          0  1.03%  0.87%  0.84%  0 Ethernet Msec Ti

135        3648      339340       10  0.23%  0.19%  0.18%  0 IPAM Manager   

139      378092      616470      613  7.83%  3.94%  3.20%  0 IP Input       

181        952      41367         23  0.07%  0.06%  0.07%  0 TCP Timer       

186      712312      371930     1915 17.59%  8.46%  6.88%  0 IP RIB Update   

224      169056      345685      489  3.75%  2.09%  1.81%  0 BGP Scanner     

323      827348    1183355       699 21.03% 10.33%  8.30%  0 BGP Router     

356        624      43647         14  0.07%  0.03%  0.02%  0 e1t1 Framer back

389      354060      623692      567  8.55%  4.46%  3.71%  0 BGP Task     

PE1#

:

PE1#sh proc cpu | e 0.00%

CPU utilization for five seconds: 3%/0%; one minute: 37%; five minutes: 32%

PID Runtime(ms)    Invoked      uSecs  5Sec  1Min  5Min TTY Process  

2        408        2185        186  0.07%  0.03%  0.02%  0 Load Meter     

113        688    1383069         0  0.71%  0.86%  0.84%  0 Ethernet Msec Ti

135        3648      339677      10  0.15%  0.19%  0.18%  0 IPAM Manager   

150      14640        1560     9384  1.51%  0.28%  0.07%  0 Exec           

181        952      41401        22  0.07%  0.06%  0.07%  0 TCP Timer       

380          56      109115       0  0.07%  0.06%  0.07%  0 FR Broadcast Out

PE1#

:

:

See attached notepad file for configs - This input function on Cisco's web jumbles all the text into a big unreatable mess.

Regards

Frank

[fsebera]

Hey Varma,

I think I have solved this issue.

:

I noticed on PE3, the local spoke router WAN4, 172.16.4.0/24 prefix owner was flapping between PE1 and WAN4. Every 30 seconds, the prefix would get updated by either PE1 or WAN4. Each time a new source (owner) of the prefix was provided, either PE1 (10.0.0.1) or WAN4 (172.17.40.1).

:

:

Simple solution was to set the eBGP prefixes received from WAN4 (on PE3) to a local preference higher than the prefixes coming in from HUB1 and HUB2.

:

:

EXAMPLE:

router bgp 65535

:

  add ipv4 vrf REG4

     nei 172.17.40.1 route-map SPOKE4 in

exit

:

ip access-list standard SPOKE4

  permit 172.16.4.0

:

route-map SPOKE4 permit 10

  match ip address SPOKE4

  set local-preference 700

:

:

NOTE: Even through I did not add a local preference for WAN3 (172.16.3.0/24) this route is now also stable.

No idea why, nor no idea why IPv6 never had an issue.

Thanks again for your support!!!!!!

Frank

[Vaibhava Varma]

Hi Frank

Gr8 to hear that you found a fix for the route-flap issue..Did you implement the IPv6 Toplogy same as the IPv4 ie Hub and Spoke with redundant HUBs. I can see from your notes that its a Dual Stack Network for IPv6..Did you use 6PE or 6VPE for IPv6 network ?

I did think over what might have caused the route-flap and my understanding is as below:

Lets take the Subnet 172.16.4.0/24 being advertised from Spoke 2 to PE3. PE3 accepts this as an eBGP update best route under Spoke2_VRF and further readvertises it to the MP-iBGP cloud and being received at PE2 and selected as best route on the HUB_Ingress VRF and sent towards HUB2. Now HUB2 accepts this as an eBGP best route and further readvertises it to PE2 which again accepts is on HUB_Egress VRF becuase of the "allowas-in" feature being enabled and readvertises it to the MP-iBGP cloud. Now PE3 receives this update under Spoke2_VRF and set the Local Preference to 500 because of the import map we used under Spoke2_VRF "HUB2". Now since this MP-iBGP route had higher LP it removes the eBGP route learnt from Spoke2 and consequently the MP-iBGP loses that and then PE2 and the chain kicks off and this Subnet goes in a loop and starts flapping.

To solve this issue your solution is perfect on per prefix basis. What I was thinking on the VRF config level that we change the import MAPs in Spoke1_VRF and Spoke2_VRF to make the LP of the HUB1 routes on Spoke1 being learnt from PE2 to be lower and same ways on PE1 set the LP of the HUB2 routes being learnt from PE1 to be lower value.

This way on PE3 the local routes would always be preferred being eBGP and not contended based on LP value.

Modified route-maps would be:

SPoke PE

ip vrf Spoke1

import map HUB1

!

ip vrf Spoke2

import map HUB2

!

route-map HUB1 permit 10

match extcommunity HUB2

set local-preference 50

!

route-map HUB1 permit 20

!

route-map HUB2 permit 10

match extcommunity HUB1

set local-preference 50

!

route-map HUB2 permit 20

!

ip extcommunity-list standard HUB1 permit rt 64513:100

ip extcommunity-list standard HUB2 permit rt 64513:400

Regards

Varma

[fsebera]

Hey Varma,

Yes I like your idea better as it is a cleaner solution, setting the looping prefixes to a lower LP, thus will prevent the looping in the first place.

:

As far as IPv6, yes everything is the same except for the addresses of course eg. ":" instead of ".".

:

I now think I understand why IPv6 is not flapping. When I run the command "sh ipv6 route" on any of the dual stacked boxes, I just happen to notice (just yesterday) none of the IPv6 routes have dates or timestamps associated with them as IPv4 does. Purly an oversite on my part - guess I was focusing on too many other things!

:

Best Regards

Frank

:

:

:

Hi Varma,

UPDATE . . . . . . .several hours after configurations updated and implemented

:

:

Turns out, selecting a lower LP on PE3 for spoke routes coming from the hubs (MP-iBGP) causes the original effect of not load sharing the spokes-to-spoke traffic between the hubs.  So I am back to setting the LP for local routes from spokes to 700 in ingress (MP-eBGP) and the spoke routes from the hubs (MP-iBGP) to 500. This works very well and is totally stable. No flapping.

:

Also, before playing around with the LP for IPv4 prefixes, as I mentioned a while ago, IPv6 never had problems, BUT now that I changed the LP for IPv4, IPv6 decided to choose a single hub for all traffic. After implementing a very similar fix for IPv6 routes, all is good!! My guess is it's a special IOS feature!!!

:

This is how the good traffic is flowing: (there is no bad traffic)

WAN3 uses HUB2 as the primary path to reach WAN4

WAN4 used HUB1 as the primary path to reach WAN3

WAN3 uses HUB2 for 0/0 and ::0

WAN4 uses HUB1 for 0/0 and ::0