Hi MikhailovskyVV,

Thanks for the replying.

Your question's answers are as follows :

• do you have a requirement for symmertic routing?
  

Yes.Our Remote site's LAN should reach to DC via primary ISP( whichever is active at the time works as primary) and that ISP should respond from DC.

• how do you see ISPs internal AS[s] - as single AS, or as several?
  

as a Single AS.

• will your ISPs accept BGP AS numbers of another ISPs? Are they agree about AS numbers per site/DC?
  

All three ISPs have their seperate BGP AS no. and our DC and remote sites AS no are constant.As at site end remains single for all the sites for all the providers.

• how many routers do you have per remote location?
  

Its around 500+ routers per provider.

• what if remote site A has ISP1 as primary and ISP2 as secondary, but site B has ISP2 as primary and ISP1 as secondary - then what links should be used for traffic A->B and for traffic B->A ? Should they traverse Datacenter even they have common ISPs?
  

Yes.This is a situation where one site is using ISP1 as a primary but other site is using ISP1 as a secondary link. Only the up link should send \ receive the data. if in case, both the primary and secondary links are UP, only primary should send\receive the data.secondary remains passive.

• what if remote site A has ISP1 as primary and ISP2 as secondary, but site B has ISP3 as primary and ISP2 as secondary - then what links should be used for traffic A->B and for traffic B->A ? Should they traverse Datacenter even they have common ISP2?
  

Look, we have aprox 1800 remote site's locations deivided among three ISP's as per their feasibility to the end locations. We have given WAN and LAN IP pools to each ISPs as per their no of sites they are providing connectivity.So in case where a site is having one ISP as a primary and others site the same IPS is as a secondary. doesnt matter.Individual site will follow the same rule i.e only UP and one link will be used for sending\receiving the data.

• how does DC LAN know which WAN route to forward traffic to? Do you use HSRP/VRRP or some IGP?
  

We have TWO MPLS routers using HSRP. Primary is used to forward the DC LAN to the know WAN route.

• do your ISPs have any routing policy based on community values? Do they support MED?
  

They will create VRFs for their respective sites LAN \ WAN pools towards DC . Yes They support MED.

• 
  
• is it allowed to inject (by DC) 0.0.0.0/0 into ISPs' clouds?
  

No. only 10.0.0.0\8 and 10.96.0.0\12 is allowed to inject by DC into ISPs clouds.

• what summaries could we use to summarize: a) DC only, b) whole network?
  

At DC : 10.96.0.0\12 and 10.112.0.0 ( DR & DR) and 10.0.0.0\8 for the whole network.

Regards,

Anil

Here are some design notes - please let me know you thought about each of them:

- spokes should advertise only localy originated routes (as-path-filter $^);

- spokes should advertise local prefixes via primary link "as is", and with 3 prepends via secondary;

- spokes assign local preference 200 for inbound prefixes over primary link, keep default (100) over secondary;

- spokes filter out all inbound updates except of originated from HUB (by  originated AS or community);

- over primary links Hub advertises all routes it learnt from ISPs + 10.0.0.0/8 + 10.96.0.0/12 + 10.112.0.0, setting MED to 50;

(if you do not advertise spokes' specifics, then there is no way for provider to route traffic over DC, but not via site's secondary link)

- over secondary links Hub advertises 10.0.0.0/8 (only), setting MED to 100;

- Hub accepts all the routes (10.0.0.0/8 le 30), but set local preference to 200 over primary link;

- Hub routers should have iBGP.

- Hub should have unique (not equal to spoke) AS number.

Future optimizations:

- each site should have unique ID (let's say Hub has ID = 96);

- site should accept inbound route if it has community with local ID = SiteID:100 (should set local preference 100) and = SiteID:300 (should adjust local preference to 300);

- if all the spokes are using single AS number, it's expected that ISPs are using "neighbor as-override".

Hi Mikhailovsky,

Yes, these points are required.

-spokes should advertise only localy originated routes (as-path-filter $^);

- spokes should advertise local prefixes via primary link "as is", and with 3 prepends via secondary;

- spokes assign local preference 200 for inbound prefixes over primary link, keep default (100) over secondary;

- spokes filter out all inbound updates except of originated from HUB (by originated AS or community);

- over primary links Hub advertises all routes it learnt from ISPs + 10.0.0.0/8 + 10.96.0.0/12 + 10.112.0.0, setting MED to 50;

(if you do not advertise spokes' specifics, then there is no way for provider to route traffic over DC, but not via site's secondary link)

- over secondary links Hub advertises 10.0.0.0/8 (only), setting MED to 100;

- Hub accepts all the routes (10.0.0.0/8 le 30), but set local preference to 200 over primary link;

- Hub routers should have iBGP.

- Hub should have unique (not equal to spoke) AS number.

Regards,

Anil K.

Hello, here is a draft config (had no as-override feature ion my IOS, so had to use different AS-number per remote site, but that changes nothing):

HUB-primary router:

ip prefix-list TO_BGP seq 5 permit 10.0.0.0/8 le 29

route-map BGP_IN_PRIMARY permit 10

set local-preference 200

route-map BGP_OUT_PRIMARY permit 10

match ip address prefix-list TO_BGP

set metric 50

router bgp 111

no synchronization

bgp log-neighbor-changes

network 10.0.0.0 <- you may use static to null0, so BGP would advertise it

network 10.96.0.0 mask 255.240.0.0 <- you may use static to null0, so BGP would advertise it

neighbor 10.1.1.3 remote-as 1

neighbor 10.1.1.3 route-map BGP_IN_PRIMARY in

neighbor 10.1.1.3 route-map BGP_OUT_PRIMARY out

neighbor 10.2.1.4 remote-as 2

neighbor 10.2.1.4 route-map BGP_IN_PRIMARY in

neighbor 10.2.1.4 route-map BGP_OUT_PRIMARY out

neighbor 10.3.1.5 remote-as 3

neighbor 10.3.1.5 route-map BGP_IN_PRIMARY in

neighbor 10.3.1.5 route-map BGP_OUT_PRIMARY out

neighbor 10.96.0.2 remote-as 111

neighbor 10.96.0.2 next-hop-self

no auto-summary

HUB secondary:

ip prefix-list SUMMARY_ONLY seq 5 permit 10.0.0.0/8

route-map BGP_OUT_SECONDARY permit 10

match ip address prefix-list SUMMARY_ONLY

set metric 100

router bgp 111

no synchronization

bgp log-neighbor-changes

network 10.0.0.0

network 10.96.0.0 mask 255.240.0.0

neighbor 10.1.2.3 remote-as 1

neighbor 10.1.2.3 route-map BGP_OUT_SECONDARY out

neighbor 10.2.2.4 remote-as 2

neighbor 10.2.2.4 route-map BGP_OUT_SECONDARY out

neighbor 10.3.2.5 remote-as 3

neighbor 10.3.2.5 route-map BGP_OUT_SECONDARY out

neighbor 10.96.0.1 remote-as 111

neighbor 10.96.0.1 next-hop-self

no auto-summary

Remote-site:

ip as-path access-list 1 permit ^$

ip as-path access-list 111 permit _111$

route-map BGP_IN_SECONDARY permit 10

match as-path 111

route-map BGP_IN_PRIMARY permit 10

match as-path 111

set local-preference 200

route-map BGP_OUT_SECONDARY permit 10

match as-path 1

set as-path prepend 6 6 6

route-map BGP_OUT permit 10

match as-path 1

router bgp 6

no synchronization

bgp log-neighbor-changes

network 10.6.0.0 mask 255.255.255.224

neighbor 10.6.0.33 remote-as 1

neighbor 10.6.0.33 route-map BGP_IN_PRIMARY in

neighbor 10.6.0.33 route-map BGP_OUT out

neighbor 10.6.0.161 remote-as 2

neighbor 10.6.0.161 route-map BGP_IN_SECONDARY in

neighbor 10.6.0.161 route-map BGP_OUT_SECONDARY out

no auto-summary