FabricPath or OTV between two DataCenter using Direct fiber cable

[Steev112]

Hello,

I have two DataCenter  both of them has the same equipment N7k, N5k and N2k, and we want both dataCenter be Active/Active, i am really confusing to use OTV or FabricPath Feature, if any one can help me for my scenario and explain to me what is the best solution and  advantage and disadvantage between OTV and PabrcPath.

Thanks  a lot in advanced

[Bilal Nawaz]

Steven,

Fabric path is a replacement (cisco proprietary) for spanning-tree - but can work with STP if required.

OTV is a layer 2 extension - Data centre interconnect via Layer 3. This can help with things like vmotion of VM's etc...

They are completely separate things here. Though technically you could run FP over your data centre interconnects, I'm not sure if this is a good or bad thing.

What do you want to achieve, what are your goals? Then we can help you better.

[Steev112]

Hi Bilal,

Thanks for your reply, i have read if we have direct fiber cable between the two Datacenters we can use either FabricPath or OTV, and this is our scenario we have fiber cable between them. Correct me if i am wrong please?

And I can’t configure FabricPath between nexus 7k and access switch 3750 only I can use VPC, please correct me?

What we have as below:

1. Each DataCenter has ( two N7k, two N5k, N2k, access switches 3750, and two ASA 5500) will create two VDCs one for server layer and another one for access switches layer, the two ASA will be connected to N7k (VDC server layer) and we will configure the four ASA5500 as clustering

Now i have two options to achieve this scenario either use FabricPath or OTV so i am confusing i can't decide which feature i have to use for best practice and advantage and disadvantage,

Please if you can help to select which one based on advantage and disadvantage,

Also i have read two links but not clear for me if you can help:

http://www.cciedash.com/2013/07/why-otv-and-not-fabricpath-for-dci.html

https://supportforums.cisco.com/discussion/11823981/ask-expert-deploying-cisco-fabricpath-data-center-networkfabricpath

Thanks a lot

[Bilal Nawaz]

Hello Steven,

Ok now I have a good idea of what you are faced with I can suggest a vague design that I normally follow when building a data centre. I must say that I haven't come across any Cisco documents stating to use fabric path for DCI. See here: http://www.cisco.com/c/en/us/solutions/enterprise/data-center-designs-data-center-interconnect/index.html

This design/architecture heavily relies on which line cards you will be using in the N7K, also the requirements for your hosts and servers.

I suggest having some M1-32XP-12L and F3-48XP-25 line cards.

Primarily the F series line card will serve our servers and hosts, all layer 2 and SVI's for servers and VM's

M series line cards can be used for the VDC interconnects and also the WAN (DCI) interlinks.

I recommend that we create 3 VDC's on the N7K's. They will be used as functions listed below.

• LAN
• WAN
• OTV

LAN will be used for servers, downstream towards N5K's

WAN will be our WAN edge, leading to other potential sites and the internet

OTV will carry out OTV functions, the layer 2 stretch between our DC's.

Personally I do not think we need to use the 3750's, perhaps we could use them with the ASA's and clustering them.

LAN VDC we will enable fabric path downstream towards the N5K's. N5K's will be leaf nodes if we have STP below them and N7K's will be the spine.

WAN VDC we will enable multicast to help us with OTV deployment, we can use OSPF with ASA (if you want) or just static routes and the LAN/OTV vdc's. WAN will ultimately have the default route out to the net, and advertise this to other vdc's.

Let me know your thoughts first, then I can suggest more detailed design.

hth

Bilal

[Steev112]

Hi Bilal,

Thanks for your explaination, in our scenario we have F2e card and M2 cards, but as mentioned before we will devide n7k as follow:

1. First VDC for users ( the VDC will be connected to C3750) so we will need to use vPC between access switches and this VDC. so i need to configure vPC becuase i can't configure fabric between this VDC and c3750 correct me please?

2. Second VDC for Servers (the VDC will be connected to N5k and ASA5500).here i have two options either configure vPC between this VDC and N5k or i can use fabric path , please correct me ?,and  we will configure vPC between this VDC and ASA 500 will be L3, i can't  use Fabric Path between this VDC and ASA5500 please correct me?

3. Between the two VDCs  will be L3.

4. between the two data Center still i am confusing to use Fabric Path or OTV beucase i want to know what it is the advantage and disadvantage to use Fabric or OTV between two data Center becuase i have many options as follow:

1. if i will use vPC internally (between VDC#1 and C3750 and Between VDC#2 and N5k as well as ASA), what is the advantage and disadvatnge if i use fabric path or OTV between the two Data center.

2. if i use vPC between VDC#1 and C3750, and between VDC#2 and ASA5500 and between use fabric path between VDC#2 and N5k,  in this case i have mix some configuration will be vPC and some configuration will be fabric path, what is the advantage and disadvantage  if i use fabric path or OTV between the two Data center.

i believe in below link mentioned if the internally configuration vPC and between the two Data center fabric path or OTV, and if the internally use fabric path   between the two Data center fabric path or OTV.

https://supportforums.cisco.com/discussion/11823981/ask-expert-deploying-cisco-fabricpath-data-center-networkfabricpath

i hope Bilal is clear to help me for this scenario what is the best use fabric path or otv between the two data center and why?

attached file is network diagram for one site and other site has the same

[Bilal Nawaz]

Hi Steven,

One thing I must warn you about with F2e card. They do not do layer 3 routing, like you cannot create a VLAN and SVI that works without an M card assigned to the same VDC, so they will require the M2 card to be in the VDC in order to carry out the routing on behalf of the F2e card. This feature is called proxy routing I think. Please search for it to be familiar.

The reasons why OTV is advised to be used rather than Fabric Path for DCI were stated in the link you shared with me.

• If there is any flooding in one datacenter it is carried to the other datacenter also which is completely not desirable, needless to say if there are multiple datacenter interconected you have one hell of a problem. OTV has the upper hand as it localizes all this to a single datacenter.
• Both datacenter need to run fabricpath & all vlans needs to be configured, because it becomes one single topology. when we use OTV we only extend the required vlans and not all vlans
• Gateways cannot be local on the datacenter and there will be a single forwaring HSRP gateway for both datacenter making it inefficient in every way & the DCI link will forwarding traffic full fledged.
• The arp optimization that OTV delivers cannot be leveraged when using the fabric path

These are the biggest reasons why you should be using OTV as your DCI rather than Fabric Path.

You will need to re-consider and change your design if you want to implement OTV.

Fabric Path we can use internally, OTV would be for DCI.

I could show you a rough topology of what I think you should do, if you would like that? or is your design dictated to you by someone else?

hth

Bilal (CCIE #45032)

[Steev112]

Hi Bilal,

i will explain what is the issue exactly maybe to be clear and help me more

why we are confusing for this design if  i can use fabric path between two datacenters, no need for M2 and the OTV license only i can buy fabricpath license and the customer built this idea on this article :

http://www.packetmischief.ca/2013/06/17/dci-using-fabricpath-for-interconnecting-data-centers/

so they want to know what is the advatnage and disadvantge if he use fabricpath since he has only two datacenter and no plan to have more in the future,

what i understand from fabricpath is:

1. only can use it between two datacenters of you have more we can't, please correct me?

2. HSRP localization can not be implemented as OTV. However You can have two differnet Gateways at the Data Center 1 and 2 using two different HSRP groups. If server is moved dynamically from, (i didn't understand this point can you please explain with example?

3. unknown unicast flooding  (can you give me an example?)

4. ARP optimization between Data Center  (can you give an example regarding ARP optimization?)

5. Typically two flows (Odd VLANs by OTV-VDC-1 and even vlans by OTV-VDC-2) carry the entire layer 2 traffic flow between the two Data Centes. Hence the load balancing the links is not efficient. ( (can you explain compare with FabricPath if you have example?)

6. VLAN scalability for OTV is lower than FabricPath as of this content writing. (can you explain what this mean i didn't understand it)

7. Resiliency of FabricPath network is better than OTV in some failure scenarios.(can me an example ?)

Sorry Bilal for all these points but really i want to understand these and

i need your recommendation either to go with OTV or fabricPath,  hopefully you can help me to be clear for me.

really thanks for your support.

[Bilal Nawaz · Accepted solution]

Hi Steven,

No problem, I will go through your points as thoroughly as I can. I do advise you to read about these protocols more, perhaps if you have access to INE or similar, see their video's on this. I'd also like to state again that I have not seen any documentation from Cisco stating that FabricPath to be used as a DCI.

In terms of Fabric Path you ask the following...

1. only can use it between two datacenters of you have more we can't, please correct me?

No, you can use Fabric path with more than two data centres but same with OTV, you can use with more than two data centres.

2. HSRP localization can not be implemented as OTV. However You can have two differnet Gateways at the Data Center 1 and 2 using two different HSRP groups. If server is moved dynamically from, (i didn't understand this point can you please explain with example?

OK so this is a BIG topic. HSRP localisation CAN be implemented with OTV, but CANNOT be implemented with Fabric Path. First hop redundancy protocols can be localised and is supported by Cisco with OTV, essentially this allows the same default gateway to reside in both of your data centres providing ACTIVE/ACTIVE setup. So no matter where your VM's are, they do not have to change their default gateway, even if your servers move to the other data centre.

If we did not have this, we would have only one active HSRP member split across DC's and things would get extremely messy with regards to traffic flows. A VM in DC2 in VLAN A needs to talk to host in VLAN B. But the default gateway is all the way in DC1. So frame gets sent over the DCI to DC1, then the default gateway, routes the packets to VLAN B. This VLAN B actually resides in DC2, so now it has to go all the way back to DC2. You get my point....? :)

With localisation this only happens local to the DC. So all servers / VMs in the DC can talk locally to its "own" default gateway.

3. unknown unicast flooding  (can you give me an example?)

Unknown Unicast traffic is unicast packets/frames with unknown destination mac address. By default switches flood this type of traffic to all ports in the VLAN. With fabric path this would take place over your DCI, however with OTV, this is all taken care of locally, so massive savings on bandwidth here and it is far more efficient.

4. ARP optimization between Data Center  (can you give an example regarding ARP optimization?)

This is another function of OTV that makes it far superior over fabric path. Essentially we are reducing the amount of traffic going over the transport infrastructure (i.e. the dci)

When ARP takes place, from host in DC1 to a responding host in DC2, we utilise the links and there is packet travel time, which could be minimal for sure, but it is not the most optimum. OTV AED - or edge device snoops the ARP reply and subsequently knows that this mapping exists from here onwards. After the first ARP takes place, the AED almost proxy arps locally at DC1 so the ARP request does not have to travel to DC2.

5. Typically two flows (Odd VLANs by OTV-VDC-1 and even vlans by OTV-VDC-2) carry the entire layer 2 traffic flow between the two Data Centes. Hence the load balancing the links is not efficient. ( (can you explain compare with FabricPath if you have example?)

In my humble opinion this is wrong and right. OTV load balances if you have more than one AED at the site. Odd vlans go via one AED, even numbered VLANs go via the other. Depending on traffic on the VLANs this could become unbalanced. Fabric path uses all of its links to "route" mac addresses to the respective SID - Switch ID's that it needs to get to. So maybe a more even split here.

6. VLAN scalability for OTV is lower than FabricPath as of this content writing. (can you explain what this mean i didn't understand it)

I completely disagree with this comment. I too do not understand it.

7. Resiliency of FabricPath network is better than OTV in some failure scenarios.(can me an example ?)

I also disagree with this. Fabric Path resiliency could be same as OTV or maybe better. However my personal experience is that fine tuning OTV with things like BFD the failover is much much quicker!

Fabric path is good because the control plane ISIS, and how it operates is admirable, but same could be said for OTV.

Lets say one of the DCI links was to die, the forwarding of Fabric path would continue via the other links, so maybe for low latency, high frequency trading environments this would be beneficial. OTV would have to change the AED and re-learn mac addresses advertised by other AED's, but like i said, the time could be extremely minimal with tuning. This isn't a big deal unless you require sub second convergence times!

I hope i answered your questions well, I recommend use OTV for your DCI, use fabric path for your internal switching local within your DC. This has been implemented many times and the links I sent you the Cisco Validated Designs also state this.

Remember - fabric path was built to be a step stone towards TRILL, and replacement for spanning-tree, OTV was built specifically for the dci. They are both built and design for specific use cases. It does not make any sense to get these confused or mixed up, unless there is a real pressing case.

Joel's conclusion is fair, use the right tools for the job. If the use case is good for FP then OK, if not then OTV.

Rcmnd reading - http://www.packetmischief.ca/2013/04/23/dci-series-overlay-transport-virtualization/

These are just my thoughts.

Bilal (CCIE #45032)

[Steev112]

Hi Bilal,

I am really appreciate your support now is very clear, but it come in mind this thing:

1. can i use vPC between two datacenters instead of fabric path ? this is just for my knowladge :). since fabric path need license and we can't make active/active scenario unless i configure MHSR i can use vPC and i can use MHSRP?

2. regrading ARP optimization on OTV, the the ARO not go through DCI how if one host on DC#1 want to commnuicate to the host on DC#2, this is not clear for more, or maybe i understood wrong

2. Regarding Resiliency of FabricPath network is better than OTV in some failure scenarios. please check below links :

http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Data_Center/DCI/whitepaper/DCI3_OTV_Intro/DCI_1.html#wp1186096 (OTV) 256 VLAN's

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/guide_c07-690079.html#wp9000484 (FP) 2000-4000 VLANS

i don't know if it is right or wrong.please check

Thanks alot :)

[Steev112]

Hi Bilal.

I hope you are doing well,

i am still waiting your reply,

1. just i want to add one thing, if we use fabric path and i want active/active i read we can use anycast hsrp to acheive this, please correct me and if you have example for this configuration between two datacenters.

[Bilal Nawaz · Accepted solution]

Hi Steven,

Yes you can use vPC between two data centres.

ARP optimisation with OTV, the AED snoops ARP replies. It then keeps a record of that. So any further arp requests locally at the site, the AED responds on behalf of that other host.

Your other point does not make any sense. Or maybe you are trying to state that OTV supports 256 VLANs and FP supports 2000+. Well my only question to this would be why would you want to stretch 256 VLANs, that is quite a lot to stretch between DCs!

To be honest, I have not configured anycast HSRP, its new to me. I have just read about it, and it seems to be a solution for active/active setup which is great! Obviously some caveats still with fabric path compared with OTV, vice versa. Still doesn't make it superior in my view, because of the hair pinning reliance on MDT root, and very inefficient in behaviour regarding broadcast, unicast, arp type traffic, though this is a good step forward.

switch# configure terminal
switch(config)# interface vlan 1
switch(config-if)# hsrp version 1
switch(config-if)# hsrp 1 ipv4
switch(config-if-hsrp)# ip 1.1.1.1
switch(config)# hsrp anycast 1 ipv4
switch(config-anycast-bundle)# force gateway-down
switch(config-anycast-bundle)# switch-id 1300
switch(config-anycast-bundle)# vlan 1
switch(config-anycast-bundle)# priority 90
switch(config-anycast-bundle)# track 2
switch(config-anycast-bundle)# timer 15 25
switch(config-anycast-bundle)# no shutdown

 Reference: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/6_x/nx-os/fabricpath/configuration/guide/b-Cisco-Nexus-7000-Series-NX-OS-FP-Configuration-Guide-6x/b-Cisco-Nexus-7000-Series-NX-OS-FP-Configuration-Guide-6x_chapter_0100.html

I'd also like to share these links for your reference and reading:

https://supportforums.cisco.com/discussion/12057226/fabricpath-hsrp-localization-dci

http://adamraffe.com/2013/08/23/anycast-hsrp-4-way-hsrp-with-fabricpath/

hth

Bilal (CCIE #45032)

[Steev112]

Hi Bilal,

Thanks a lot for your support now is clear :)

Thanks

[grant.maynard]

Just a quick note on scaleability of OTV - it has improved a lot in recent NXOS

Limitations for OTV (from NXOS 6.2):

• 10 Overlays
• 8 sites
• 2 Edge Devices per site
• 1,500 VLANs extended via OTV
• 32,000 MAC addresses across all the extended VLANs
• 12,000 local MAC addresses per site
• 4,000 local multicast routes, 256 multicast data groups

[From Nexus 7000 Series NX-OS Verified Scalability Guide, http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NX-OS_Verified_Scalability_Guide.html]

[Robert Rowland III]

Grant,

   I would be somewhat disbelieving of the VLAN limit.  I suffered multiple 7K issues a few months ago that led to repeated service interuptions .

Contributing root cause .. ... ready for it ... 

We had 350 vlans on a pair of 7Ks and they were in the same campus.

1500 ... probably mean 1500/8 per location not ... 1500 total

[Luigi Valenza]

Hi Steven,

you can use both protocols together.

Fabric Path within the DC from the Aggregation layer down to the access switches of the DC.

And the usage of the OTV will be for the DCI.