This discussion is locked

Ask the Expert: Nexus Virtual Port Channel (vPC)

Unanswered Question
Aug 26th, 2011

With Hatim Badr and Iqbal Syed

Read the bioRead the bio

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn about design, configuration and troubleshooting of VPC with Cisco experts Hatim Badr and Iqbal Syed. Iqbal is a product manager and technical marketing engineer for the Cisco Nexus 7000 Series of switches. He is responsible for product road-mapping and marketing the Nexus 7000 line of products with a focus on virtual port channel design and training. Syed has been with Cisco for more than 8 years, which includes experience in Cisco Advanced Services and the Cisco Technical Assistance Center. His experience ranges from reactive technical support to proactive engineering, design, and optimization. He holds CCIE (Routing & Switching), CCDP, Cisco Data Center, and TOGAF (v9) certifications. Hatim is a network consulting engineer for Cisco Advanced Services in Toronto, where he supports Cisco customers across Canada as a specialist in data center architecture, design, and optimization projects. He has more than 10 years of experience in the networking industry. He holds CCIE certification #14847 in Routing and Switching and also holds TOGAF 9, VCPv4, and PMP certifications.

Remember to use the rating system to let Hatim and Iqbal know if you have received an adequate response.

Hatim and Iqbal might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the discussion forum shortly after the event. This event lasts through September 9, 2011. Visit this forum often to view responses to your questions and the questions of other community members.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 4.9 (12 ratings)
Mohamed Sobair Fri, 08/26/2011 - 17:19

Hello Hatim,

Its Good to see you here hosting Nexus VPc, and I am proud that you as a previous teacher from my previous university as I knew you have taught some classes to students at Computer Man College.

Now, my knowledge about VPc it provides Layer-2 Multipathing and the benefit of using a Spanning-tree block port as a Forwarding port. and provides rapid convergence than STP does. I personally dont have experience with Nexus yet, The Last series of Switches I worked on are 6500 Series with VSS Technology.

With that being said, I have some questions about VPc:

1- Is a VPc is normal Layer-2 Etherchannel? and do we have VPc suuports for layer-3 forwarding just like we have layer-3 Etherchannel?

2- If we have a pair of VPc devices connected a Core/distribution layer, and we have multiple nexus switches connected at the access layer, does the Access layer see both VPc Core Peers as One logical device? does this provides the same functionality as with Cisco VSS on the 6500 Switches? what is exactly the difference?

3- Does enabling VPc provides ONLY rapid Convergence to a failure detection than Spanning-tree provides? and when enabling VPc on nexus, Do we actually eleminate the need of Spanning-tree protocol?

4- From the Access layer, if we have Nexus for example 5000 series, Can we have One VPc identifier connects to both Nexus 7000 Distrbuttion/COre VPc pair?

5- Does using VPc eleminate of using first Hop redundancy protocols like HSRP? and How frames are forwarded through the Active/Standby HSRP peers? do we have still One as Active and one as Standby , or this concept has changed with VPc?

I appreciate and Greatful for your time and answers.

Regards,

Mohamed

isyed Sat, 08/27/2011 - 14:21

Hi Mohamed,

Please see your answers inline below:

1- Is a VPc is normal Layer-2 Etherchannel? and do we have VPc suuports for layer-3 forwarding just like we have layer-3 Etherchannel?

There is a very good white paper explaining VPC , Its benefits and how L3 works over VPC below:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-516396.html

2- If we have a pair of VPc devices connected a Core/distribution layer, and we have multiple nexus switches connected at the access layer, does the Access layer see both VPc Core Peers as One logical device? does this provides the same functionality as with Cisco VSS on the 6500 Switches? what is exactly the difference?

Yes the access layer devices will see the VPC peers as one logical device .Regarding the differences between VPC and vss , just like Marwa mentioned , the major difference is the fact that VSS provides one control plane while VPC has two.For other differences , the document pointed by Reza would be a great reference.

3- Does enabling VPc provides ONLY rapid Convergence to a failure detection than Spanning-tree provides? and when enabling VPc on nexus, Do we actually eleminate the need of Spanning-tree protocol?

The main difference between a vPC configuration and a non-vPC configuration is in the forwarding behavior of the vPC peer link and the BPDU forwarding behavior of vPC member ports only.Non-vPC ports on a vPC-configured switch behave in the same way as on a regular switch ( use STP), except that the vPC peer link is always forwarding, which may require a slightly different topology.

Please refer to the document below which explains how STP works in a VPC environment along with examples:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572834-00_STDG_NX-OS_vPC_DG.pdf

4- From the Access layer, if we have Nexus for example 5000 series, Can we have One VPc identifier connects to both Nexus 7000 Distrbuttion/COre VPc pair?

I assume by VPC identifer you mean VPC domain , if yes then jst like Marwan mentioned wou will need two VPC domains

5- Does using VPc eleminate of using first Hop redundancy protocols like HSRP? and How frames are forwarded through the Active/Standby HSRP peers? do we have still One as Active and one as Standby , or this concept has changed with VPc?

With VPC configured , HSRP is still needed however the improvement was made to the forwarding engine specifically to allow local Layer 3 forwarding at both the active HSRP peer and the standby HSRP peer. This enhancement provides, in effect, an active-active HSRP configuration with no changes to current HSRP configuration recommendations or best practices and no changes to HSRP. The HSRP control protocol still acts like an active-standby pair, so that only the active device responds to Address Resolution Protocol (ARP) requests, but a packet destined for the shared HSRP MAC address is accepted as local on either the active or standby HSRP device.

Hope that Helps,

Regards,

Iqbal Syed

habadr Sat, 08/27/2011 - 14:55

Hi Mohammed, Marwan and Reza

First thank you to all of for enriching the discussion with questions and answers. I believe we will have lots of fun in the next two weeks.

Iqbal already commented on all them but let me just add my comments as well.

Q1- Is a VPc is normal Layer-2 Etherchannel? and do we have VPc suuports for layer-3 forwarding just like we have layer-3 Etherchannel?

A1- the answer is Yes and No. Mawrwan showed 1 of the scenarios however I’ll refer to Brad Hedlund paper in his blog

http://bradhedlund.com/2010/12/16/routing-over-nexus-7000-vpc-peer-link-yes-and-no/

As you can see there are several scenarios for L3 with vPC. Some of them are supported while other are not, This is due to the most important rule for vPC duplicate frame prevention logic that drops traffic traversing the peer link (destined for a vPC member port) when there are no failed vPC ports or links.

Q2- If we have a pair of VPc devices connected a Core/distribution layer, and we have multiple nexus switches connected at the access layer, does the Access layer see both VPc Core Peers as One logical device? does this provides the same functionality as with Cisco VSS on the 6500 Switches? what is exactly the difference?

A2-  Answered by Iqbal

Q3- Does enabling VPc provides ONLY rapid Convergence to a failure detection than Spanning-tree provides? and when enabling VPc on nexus, Do we actually eleminate the need of Spanning-tree protocol?

A3 – We eliminate the Spanning-Tree blocking ports but spanning-Tree still running in the background. While still operating with two separate control planes, vPC helps ensure that the neighboring devices connected in vPC mode see the vPC peers as a single spanning-tree and LACP entity. For this to happen, the system has to perform IEEE 802.3ad control-plane operations in a slightly modified way (which is not noticeable to the neighbor switch). Please refer to the followign design docuemnt for more details

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/design_guide_c07-625857.pdf

Q4- From the Access layer, if we have Nexus for example 5000 series, Can we have One VPc identifier connects to both Nexus 7000 Distrbuttion/COre VPc pair?

A4- As Marwan and Iqbal mentioned if the Nexus 5000 will have another vPC domain and you want to create double sided vPC then you should use different domain ID.

However if you mean you just want to connect an access switch , whether Nexus 5000 switch, a Cisco catalyst switch or even 3rd party switch, then none of these switches or any other devices need to know anything about the Nexus vPC technology, all you need is to configure regular ether-channel, the intelligence is in Nexus vPC technology.

Q5- Does using VPc eleminate of using first Hop redundancy protocols like HSRP? and How frames are forwarded through the Active/Standby HSRP peers? do we have still One as Active and one as Standby , or this concept has changed with VPc?

A5 – As Marwan and Iqbal mentioned, you need HSRP since we have separate control plane for each vPC peer switch.

The most significant difference between the HSRP implementation of a non-vPC configuration and a vPC configuration is that the HSRP MAC addresses of a vPC configuration are programmed with the G (gateway) flag on both systems, compared with a non-vPC configuration, in which only the active HSRP interface can program the MAC address with the G flag.

Given this fact, routable traffic can be forwarded by both the vPC primary device (with HSRP) and the vPC secondary device (with HSRP), with no need to send this traffic to the HSRP primary device. Without this flag, traffic sent to the MAC address would not be routed.

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/design_guide_c07-625857.pdf

Thanks

Hatim Badr

Mohamed Sobair Sat, 08/27/2011 - 15:00

Thanks Marwan and Reza for the link and explanation.

and Thanks Hatim/Iqbal for the answers, I have a couple of questions though,

1- You mentioned STP works normally for the non-vPC ports as a loop avoidance mechanism, Now this is clear, what is not clear, how is actually vPC member ports avoid loops? Does vPC by itself considered a loop avoidance mechanism? and how this is possible if all link of the vPC member ports are all forwarding?

2- You say, (but a packet destined for the shared HSRP MAC address is accepted as local on either the active or standby HSRP device) , As far as I know, the shared Mac address of the HSRP is used for the local communication between the HSRP Active and Standby devices, but it doesnt respond by itself to the ARP request by the client, I am still not sure I fully understand how both becomes Active/forwarding? and for this Scenario, is it required to have HSRP configured on a vPC member ports? if you elaborate on this, it would be better.

3- Final question, is vPC is actually configured on a layer-2 switch port? if so, that means if we have Layer-3 routed Access design with Distribution/Core also works on alayer-3, Do we still need vPC here?

Regards,

Mohamed

Marwan ALshawi Sat, 08/27/2011 - 17:55

Thanks to every one and it looks like really will be interesting discussion

back to Mohamed questions

1- vPC has Loop prevention mechanisms where traffic/frames coming over the peer link they will not be forwarded over a vPC member port ( Duplicatre frame Prevention )

2- the Active HSRP in Nexus reply with the virtual MAC of the VIP as described by Hatim, this will be the MAC - to HSRP - VIP which will be mapped to both the active and standby from arp prospective and this way they are both in forwarding which is different from other Cisco Devices while in Nexus HSRP virtual MAC is populated into the L3 hardware forwarding tables, making local forwarding capability on the HSRP standby

3-  vPC is a L2 feature if you have routed access layer then just use equal cost multiPathing ECMP using a routing protocol no need for vPC, HSRP ..etc

Very Good questions and very interesting "Ask the expert" Discussion

HTH

Marwan ALshawi Sat, 08/27/2011 - 02:24

Its is nice to see more Ask the expert about Nexus and its technologies in CSC

for Mohamed's qoestions, answers as belllow and Hatim can correct me if any is not accurate

1- yes it is normal L2 interfaces but there are some other types of L2 interfaces required for this vPC to work such as the peer linka nd keep alive link and the ports connected to a VPC host called vPC member port 

and for L3 you can not have vPC as it is a L2 technology only however if you have two L3 interface in the upstream and downstream passing through a L2 Path with vPC then you can Pass the L3 routing for example

2-  core is L3 only,however from access to distribution there will be vPC and the vPC pairs will appear to the access switch as one logical device and from distribution to core is L3 routing with ECMP

the main difference between vPC and VSS is that the later one uses one control plane in active standby while vPC has two  differnt seperate control planes each belong to one of the vPC peers

from forwarding point of view both technologies provide you with all forwarding paths

3- vPC eliminate the need of spaning Tree ( but better to enable it as fallback method ) and there is no convergence just all forwarding paths

4- vPC identifier if you mean here "vPC domain" then yes from on side to other ( access to distribution vPC ) one domain

and ifyou need double sided then you will need tow vPC domains

5 No, HSRP for example still required but the deference here from forwarding point of view as both the active and standby will be in forwarding and the active HSRP will be responding to to ARP form forwarding prospective

i think very good questions and i wish my participation will be helpful in this discussion/session

Thanks and Regards,

Marwan Alshawi

Mohamed Sobair Sat, 08/27/2011 - 20:21

Thanks Hatim for the answers, I beleive we have posted on actually the same time, I found my answers on the Blog link you have posted by Brad and the describtion of your HSRP.

Marwan,

Thanks for taking the time to answer, However, I have just a few comment. You seem to have excellent knowledge about Nexus and vPC, besides your answers and describtion are awsome/excellent. but , I wanted just to say , it would be better if we leave the answers to the original Hoster for this session. If you may for any reason find thier answer misleading or not complete or if you want to add technical point or have any related other question, you can then post your comment or question if any, it would be better approach than replying to question Immediately addresses for Hatim and Iqbal before getting thier replies.

I thank you for understanding, and you are truly expert.

Regards,

Mohamed

habadr Sun, 08/28/2011 - 07:00

That is great Mohamed and thanks Marwan for the answers,

So Q1&2 are clear now but for 3rd question about routed access I just want to clarify that you do not need vPC in this case as you do not need spanning tree as well.

vPC technology is for extending L2 domain with eliminating spanning tree blocked ports hence uses all available uplink bandwidth. Utilize half the number of links for the same bandwidth, or allow more efficient use of current available ports.

Data Center applications and technologies such as virtualization require larger L2 domains. Enabling a more efficient use of L2 scaled domains is the goal of vPC

Thanks

Hatim Badr

sean_evershed Sat, 08/27/2011 - 23:46

Hi,

My question concerns the scenario of a pair of N7K's that are deployed in geographically separate data centres.

In order to support vPC between these two devices are there any recommended SLA's for the WAN link between the two data centres? For example bandwidth, latency and packet loss?

Thanks

Sean

isyed Sun, 08/28/2011 - 10:54

Hi Sean,

VPC itself doesnt  impose any distance limitations between the two geographically seperated  DCs , the limitation would depend upon the optics in use and the kind  of interconnection (WAN Link ) you have between the two DCs.

Please see the detailed document below for some of the testing performed in this area :

http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns949/ns304/ns975/data_center_interconnect_design_guide.pdf

Similarly  there are no recommendations on SLA's in regards to latency , packet  loss etc , it will depend upon the applications in use and their  sensitivity to latency and packet loss.The bandwidth requirement would  be driven by the amount of traffc between the two DCs.

As  such careful consideration should be given to these parameters and  thorough testing should be performed when designing a DCI soloution.

HTH,

Regards,

Iqbal

Marwan ALshawi Sun, 08/28/2011 - 18:40

Dear Hatim and Iqbal

could you please confirm my reposes in the bellow discussion are accurate/up to date, about multihomed server NICs working in Active/active mode with Nexus vPC as i am not 100% sure if there is a new release that started support new vPC capability

https://supportforums.cisco.com/thread/2101678

Thanks and Regards,

Marwan

habadr Mon, 08/29/2011 - 07:14

Hi Marwan,

You are right. The only un-supported vPC scenario is to have vPC between N5K--> N2K and then from N2K --> Server. You can do either of them.

Thanks

Hatim Badr

Giuseppe Larosa Mon, 08/29/2011 - 07:30

Hello Hatim and others,

I'm sorry to have to report my bad experiences with Nexus here but I think it is important to know if the problem is fixed.

Short history:

while building a new datacenter last year with two Nexus 7018 used as L2 only with VPC we faced a serious issue.

We had 20 C4948-10GE connected downstream to the Nexus pair

One of my colleagues trying to add a vlan to the VPC made an error:

instead of using

switchport trunk allowed vlan add x,y,z

he typed by error

switchport trunk allowed vlan x,y,z

on second nexus only

the VPC consistency check was triggered and all of this network block was isolated from the rest of the network.

We had to power off the second Nexus to restore connectivity

It took one hour to recover.

In my humble opinion this is not acceptable on devices like this with a full price in excess of 100,000 $ each

Nexuss CLI should be changed in order to have the commit logic like IOS XR

Forgive me if this post looks like aggressive, I'm usually more friendly here in CSC.

I hope no one will be hurt.

the NX-OS version was something like 4.0

Best Regards

Giuseppe Larosa

CCIE SP#14802

isyed Mon, 08/29/2011 - 10:23

Hi Giuseppe,

Thanks for reporting your issue to us , we are constantly working to improve user experience with NX-OS .

There is a new feature in NX-OS release 5.2 which can help in the scenarios like this , This feature is enabled by default ( cant be disabled) in 5.2 and is called 'Per-Vlan Consistency Check'.. ( Not applicable to Spanning tree MST Mode )

--------------------

http://www.cisco.com/en/US/partner/docs/switches/datacenter/sw/5_x/nx-os/interfaces/configuration/guide/if_vPC.html#wp1834065

-Currently if spanning-tree vlans enabled on vPC peers do not match, it is a global type-1 inconsistency and as such vPC peer-link and all vPCs on both peers are brought down

- With Per-vlan consistency check, spanning-tree vlans that do not match on both peers, will be brought down on all vPCs and peer-link. Other vlans will stay up
--------------------
Regards,
Iqbal Syed
Giuseppe Larosa Mon, 08/29/2011 - 11:09

Hello Igbal,

I agree that this is a smarter way to perform VPC consistency check, however it would have been of limited use in my case:

in our case peer 1 had a vlan list made of

{existing Vlans on VPC} U {x,y,z}

peer2 had only ( for the missing add keyword)

{x,y,z}

as a result of this only the new vlans {x,y,z} would be in service and downstream C4948-10GE would be still unaccessible in-band if an access-class is applied and none of the few in  service vlans is the management vlan

both NX would be not accessible on the management vlan too, so a console access would be needed to recover from this.

Notice that in environments with strict control on the VLANs allowed on trunks for STP scalability the risk of a similar error is not so rare.

Thanks for your attention I appreciate that the behaviour has been improved

By the way I think this thread is really high quality with useful  and interesting information for us that are spread around the world.

Best Regards

Giuseppe

isyed Mon, 08/29/2011 - 12:07

Hi Giuseppe,

Hmm ...I see  - yeah in your particular case - it makes sense as none of the in service vlans x,y,z are mgmt vlans .

Anyway , thanks for bringing this scenario to our attention.

Regards,

Iqbal

Marwan ALshawi Mon, 08/29/2011 - 14:12

  Mismatched configurations can cause errors or misconfigurations that can result in service disruptions. The configuration synchronization (config-sync) feature in Cisco NX-OS Release 5.0(2)N1(1), allows you to configure one switch profile and have the configuration be automatically synchronized to the peer switch

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/system_management/502_n1_1/Cisco_n5k_system_mgmt_cg_rel_502_n1_1_chapter3.html

Not sure if this is a planned feature for future releases in the N7K as i think this will over come issues like the one Giuseppe mentioned

Thanks

Marwan

isyed Mon, 08/29/2011 - 14:46

Hi Marwan,

Agreed , Config-sync will help too!

At this time - this feature is in the roadmap for N7K .

Regards,

Iqbal

Marwan ALshawi Mon, 08/29/2011 - 16:07

By the way I think this thread is really high quality with useful  and interesting information for us that are spread around the world.

Best Regards

Giuseppe

100% Agree with you Giuseppe

Jon Marshall Mon, 08/29/2011 - 06:01

Hi Hatim/Iqbal

Firstly, i dont have much knowledge on Nexus switches so some of my questions may not make a lot of sense

From a previous answer in this thread -

1) From the Access layer, if we have Nexus for example 5000 series, Can we have One VPc identifier connects to both Nexus 7000 Distrbuttion/COre VPc pair?

I assume by VPC identifer you mean VPC domain , if yes then jst like Marwan mentioned wou will need two VPC domains

Can you elaborate on this. Does this mean if you have 2 x 5000 and 2 x 7000 and you wanted to run a vPC from each 5000 to the pair of 7000 switches then you need to use 2 vPC domains.

I'm not sure i understand what is meant by domain in this context.

2)  In a recent thread Marwan posted that if you have 2 x 2000 FEX using  vPC to a pair of 5000 switches then you cannot run a vPC from your server to the 2000 FEXs. ie.

FEX1 has a vPC to both 5000s

FEX2 also has a vPC to both 5000s

Is this a specific server limitation or does it apply to all Nexus products ie. if you have -

5000_1 vPC to 7000_1 and 7000_2

5000_2 vPC to 7000_1 and 7000_2

then you cannot run vPCs from the 2000 FEXs to the 5000s ?

3) What is the best practice design for vPCs ie.

is it recommended to always run vPC's upwards towards the distro/core layers. What i mean is does it ever make sense to have a vPC pointing the other way in that each core switch has a vPC that terminates across a pair of access layer switches ?

Can you actually have a 2 way vPC ie. at either end the vPC is terminated across a pair of switches ?

Hope some of the above makes sense !

Jon

habadr Mon, 08/29/2011 - 10:00

Hi Jon,

##I have to delete my previous response and chage it with this since diagrams were not properly added ##

You are always welcome, this discussion is for all people interested vPC.

My answers are inline

Q1) Can you elaborate on this. Does this mean if you have 2 x 5000 and 2 x 7000 and you wanted to run a vPC from each 5000 to the pair of 7000 switches then you need to use 2 vPC domains.

I'm not sure i understand what is meant by domain in this context.

A1) There are two scenarios here

     1-     Connecting each N5K to N7K vPC domain as shown below does not need any vPC configuration in N5K and all you need in N5K is regular port channel configuration similar to any other switch

2-     Creating vPC between two N5K and connect them to N7K vPC domain 1. As illustrated below


In this scenario you need different vPC domain ID for N5K as we used in the above diagram (vPC domain ID for N7K is 1 and for N5K is 2)

Q2)  In a recent thread Marwan posted that if you have 2 x 2000 FEX using  vPC to a pair of 5000 switches then you cannot run a vPC from your server to the 2000 FEXs. ie.

FEX1 has a vPC to both 5000s

FEX2 also has a vPC to both 5000s

Is this a specific server limitation or does it apply to all Nexus products ie. if you have -

5000_1 vPC to 7000_1 and 7000_2

5000_2 vPC to 7000_1 and 7000_2

then you cannot run vPCs from the 2000 FEXs to the 5000s ?

A2) You can run vPC either

     1-     between N5K and N2K

     2-     Extend to Server  with N5K or N7K which is called "host vPC"

But you cannot run vPC between N5K --> N2K and then from N2K --> server (host vPC).


Q3) What is the best practice design for vPCs ie.

is it recommended to always run vPC's upwards towards the distro/core layers. What i mean is does it ever make sense to have a vPC pointing the other way in that each core switch has a vPC that terminates across a pair of access layer switches ?

Can you actually have a 2 way vPC ie. at either end the vPC is terminated across a pair of switches ?

A3) Yes you can do that. It is called Double Sided vPC as shown in the diagram above. If your design has vPC in access N5K and in distribution N7K then it is recommended to run double sided vPC.

Thanks

Hatim Badr

Jon Marshall Mon, 08/29/2011 - 10:11

Hatim

Many thanks for the reply, a lot clearer now.

Jon

ex-engineer Tue, 08/30/2011 - 15:24

I have an interesting one that hasnt been discussed thus far....

What does a vPC failure look like? Take into consideration a software failure or hardware port and link failures. What is the WORST thing that can happen if vPC fails?

I'm asking this because we just had an interesting discussion at work. There was a comparison drawn between vPC and stacking in terms of failures, not that they are deployed for the same reason. As an FYI, the two technologies were being considered to achieve one particular goal for the particular situation we were confronted with at work. The goal was to have a design that would allow for full cross-sectional BW between the access layer (ToR) and the EoR/Agg layer in a non-blocking architecture. So having the aggregation layer in a vPC domain or stacking two switches would both achieve that.

One of the distinctions made against stacking is that the technology is notoriously buggy (on ALL vendor platforms) and can represent a single point of failure with a huge failure domain, depending on where it is deployed. For example, if a single stack is deployed at the EoR/aggregation layer and it requires a reboot to fix a problem or some other bug causes the whole stack to go down, the results can be devastating. I have seen that on Cisco StackWise and Juniper VC.

Hence, the question: what is the worst conceivable result from a vPC failure? And which is the safer bet when being deployed in a data center agg/core layer? By the way, PLEASE, let us NOT have a Cisco marketing discussion about how great vPC and stacking are and how they never fail. Let's assume they will fail for the purposes of this discussion.

For what it's worth, my answer is that if vPC fails, some traffic can be blackholed or a link can "fall" out of the vPC domain, creating a redundnat path/bridging loop, which will be addressed with STP. So, it won't be as potentially devastating as a stack failure.

habadr Wed, 08/31/2011 - 07:42

Hi ex-engineer,

Thank you for your question about vPC failure scenarios and yes vPC can fail as documented in the vPC design document at

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/design_guide_c07-625857.pdf

Please look at "vPC Failure Scenarios" section where it provides several failure scenarios and highlight the impact and how vPC behaves in each scenario. Also as you mentioned Spanning tree is running in the background as backup to prevent any loop in the network.

Also there are enhancements/features added with new releases to help reduce failure impact. Here are the major enhancements for vPC

Peer switch

Beginning with Cisco NX-OS Release 5.0.2a peer-switch feature is introduced to help both switches appear as single STP bridge and sending BPDU with same bridge ID ensure that the downstream device does not detect a spanning-tree misconfiguration.

Auto Recovery

Beginning with Cisco NX-OS Release 5.2(1), you can configure the Cisco Nexus 7000 Series device to restore vPC services when its peer fails to come. If both switches fails and only 1 switch come back online vPC will start working with 1 switch after configured delay (default 240 seconds). In previous versions this functionality is achieved using reload restore feature .

Delay restore

Beginning with Cisco NX-OS Release 4.2.(1)Delays vPCs bringup after a vPC device reload (SVI bring-up timing is unchanged) to avoid blackholing of routed traffic from access to core until layer 3 connectivity is reestablished

vPC config-sync

Config-Sync provides a mechanism to synchronize configuration between a pair of switches in a network to reduce mis-configuration problems as discussed earlier in previous thread. Config-synch is available in Nexus 5000 beginning NX-OS 5.x and in roadmap for Nexus 7000.

Thanks

Hatim Badr

adrian.steyn@gm... Tue, 08/30/2011 - 20:22

I would like to pose a question regarding a specific vPC failure scenario:

When you have a keep-alive and Peer link failure the vPC domain is completely broken and Spanning-Tree protocol is required to prevent loops and place ports in blocking mode.

What is the mechanism by which STP gets to block some paths in the dual-active vPC scenario, after failure?

If everything was working perfectly fine and someone (theoretically) cuts the peer-link and keep-alive link between the peers.... well, HOW does STP kick in (?) or would you potentially have a scenario where you have both switches forwarding which could cause wierd paths, duplicate packets etc?

You would assume a change in the MAC id, but you would require an LACP renegotiation (changing of system-id) - but there is none (correct?)

5k-A thinks he is alone, he tries to be non-disruptive, he still uses old system MAC derived from vPC Domain

5k-B does exactly the same. They both forward up and down...

7k A and B (upstream) for them them nothing changed they still talk to a switch with 5k MAC address (vPC derived). The 7Ks still think they are talking to the same device...

STP could only kick in if 5k (A or B) would change system MAC ("Im no longer in vPC, I'll use my real MAC, lets do LACP renegotiation").

would love to hear your thoughts

many thanks

Adriaan Steyn

ex-engineer Wed, 08/31/2011 - 06:54

Hey, where did all the experts go? Calling Hatim, Iqbal, Jerry, Mohammad and Marwan....?

habadr Wed, 08/31/2011 - 09:06

Hi Adriaan,

Thanks for highlighting this case. In fact this is documented in defect CSCtl70133 and you will have packet loss if peer-keepalive followed by peerlink fail  in vPC domain that does not contain the STP root which is N5K in your scenario.

It is also documented in the test scenarios in site to site vpc-vpc test results. (table 2-4 page 2-32).

http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns949/ns304/ns975/data_center_interconnect_design_guide.pdf

Test Case

Test Details

Failure Ucast

Failure Mcast

Restore Ucast

Restore Mcast

Result

DC1-DCI-7K vPC   Peer Keepalive Link and Peer Link Failure

Physically disconnected   the cables connected to DC1-DCI-7K1's peer keepalive interface, Management 0,   and entire vPC Peer Link, Ethernet 1/9 and 2/9 for failure. Links reconnected   for restoration.

sustained loss

sustained loss

N/A

N/A

Fail

By failing all vPC peer link members plus the vPC peer keepalive link simultaneously, a vPC dual active condition is forced. If this scenario happens on a vPC domain that does not contain the STP root, the STP dispute mechanism inadvertently blocks links that cause intermittent traffic drop. While this is considered an unlikely scenario (triple failure within <3 seconds).

Please note that if peer-link fails before peer-keepalive then you should not have this issue since secondary vPC peer switch will shut down all vPC ports.

Thanks

Hatim Badr

Changing the format to show the table properly

dtran@behr.com Fri, 09/02/2011 - 13:58

Hi all,

I am looking into deploying a pair of N7010's running vPC and a few N2K's connecting to N7010's via port-channel  (Please see drawing below). Most of my servers today are single homed to either N2K or directly to the N7K. My question is is there any issue/concern if I configure the vPC peer link to allow all VLANs ? in what scenario would you configure vPC vlan and none vPC vlan ?

Thanks !!!

D.

isyed Fri, 09/02/2011 - 16:43

**Re-posting with minor edit**

Hi There,

To answer your question , by definition any vlan that is forwarded on the vpc peer link beocmes a vpc vlan ...If the devices are single homed ( i.e connected to only one peer either directly or via N2K) , then the question should be do you really want those vlans to be forwarded on the peer link and by doing so extend your L2 domain across the two peers ....because you can easily  configure the vlan only on the peer where the device is connected to and use an SVI for further connectivity.That way there will be no need to extend these vlans across on the peer link essentially making those vlans  'non-vpc' vlans.

As for the Non-VPC vlan - any vlan which is not forwarded on the peer link is a non-vpc vlan and follows the regular STP rules , A lot of times customers use it for single homed devices or for the devices which arent capable of running etherchannel.

Please also note that single attached devices that are not connected via a vPC ( including single homed )  but still carry vPC VLANs are also known as orphan ports.In case of a peer-link shut or restoration, an orphan port's connectivity may be bound to the vPC failure or restoration process

Hope it is clear now.

Regards,

Iqbal

isyed Fri, 09/02/2011 - 17:57

Also to add our best practice recommendation would be to dual connect all devices to both the peers because in case of failure scenarios , the single homed devices would be isolated and traffic from them would be blackholed.

dtran@behr.com Fri, 09/02/2011 - 18:31

Hi Isyed !!! I appreciate your help !!

In my scenario I will have servers on the same VLAN connecting across both N7Ks, so I am extending L2 domain across both N7K's. Do you see any issues / concerns allowing all VLANs across the vPC peer-link ? I would like to get your inputs on this !!!!

I agree that all servers should be dual homed and that's what I am planning to do moving forward. That's why I decided to use vPC between the N7K's.

Thanks !!

Danny

isyed Tue, 09/06/2011 - 17:09

Hi Danny,

As mentioned earlier - It is strongly recommended to dual attach every device to vPC domain to avoid the isolation of the orphan ports in case of peer link failure.

I would suggest you to consider the following 3 design options before opting the option 4 which is the orphan port design you are currently considering. I have also listed down the pros and cons of each of the design options to help with your evaluation.

If after evaluating the first three options, option 4 still seems the only feasible option for you, then I would encourage you to reconsider the vpc implementation at this point since by single homing the devices and using vpc , you are not really gaining anything anyway. It would be best for you to configure vpc when you are ready to go dual homed (dual connect ) with your devices.

Hope that helps.

Regards,

Iqbal

********************

Here are the recommendations for connecting devices to vPC domain (in order of preference)

•1.      ALWAYS try to dual attach devices using vPC (not applicable for routed links).

PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-active scenarios. Ensures full redundant active/active paths through vPC.

CONS: None

•2.      If (1) is not an option – connect the device via a vPC attached access switch (could use VDC to create a “virtual access switch”).

PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-active scenarios. Availability limited by the access switch failure.

CONS: Need for an additional access switch or need to use one of the available VDCs. Additional administrative burden to configure/manage the physical/Virtual Device

•3.      If (2) is not an option – connect device directly to (primary) vPC peer in a non-vPC VLAN  and provide for a separate interconnecting port-channel between the two vPC peers.

PROS: Traffic diverted on a secondary path in case of peer-link failover

CONS: Need to configure and manage additional ports (i.e. port-channel) between the Nexus 7000 devices.

•4.      If (3) is not an option – connect device directly to (primary) vPC peer in a vPC VLAN

PROS: Easy deployment

CONS: Generally Bad. Bound to vPC roles, Full Isolation on peer-link failure when attached vPC toggles to a secondary vPC role.

freedom106 Wed, 09/07/2011 - 08:15

Hi Hatim and Iqbal

   I have few questions about vpc

   1)Do vpc peers use the mac add 00:23:04:ee:be:xx as their des mac to communicate with eachother,like the ospf use the multicast add 224.0.0.5 and 224.0.0.6?and at the scenario of double-side vpc,device must use the unique vpc domain id,if not ,they will get confuse with who is the primary vpc.

  2)I confuse with the vpc role and the operational role,what is their responsibilities?and I use the show vpc role command at nexus 5000 found that there is olny vpc role.

  3)The default vpc priority is 1024 ?

habadr Wed, 09/07/2011 - 19:45

Hi Yue

Thanks for your questions please find my answers inline

Q1)Do vpc peers use the mac add 00:23:04:ee:be:xx as their des mac to communicate with eachother,like the ospf use the multicast add 224.0.0.5 and 224.0.0.6?and at the scenario of double-side vpc,device must use the unique vpc domain id,if not ,they will get confuse with who is the primary vpc.

A1) As you know this mac address is vpc system MAC address and derived from domain ID. The vPC peer devices use the vPC domain ID to automatically assign a unique vPC  system MAC address . You MUST use utilize unique Domain id’s for all vPC pairs defined in a contiguous layer 2 domain.

vPC Peers use this MAC address as source MAC address in the following two cases

     1-      LACP neighbor needs to see the same System ID from both vPC peer. The LACP system ID is the combination of the LACP system priority value and the MAC address of the router. The vPC ‘system-mac’ is used by both vPC peers in LACP system ID to appear as single device to its neighbors.

     2-      When using peer-switch feature. Beginning with Cisco NX-OS Release 5.0.2a peer-switch feature is introduced to help both switches appear as single STP bridge and sending BPDU with same bridge ID, which is the vpc system MAC address, to ensure that the downstream device does not detect a spanning-tree misconfiguration.

Q 2)I confuse with the vpc role and the operational role,what is their responsibilities?and I use the show vpc role command at nexus 5000 found that there is olny vpc role.

A2) The vPC Primary is manually defined by the role priority. The switch with lower priority will be elected as the vPC primary switch.  and in normal operation the primacy vpc switch is the “Operational primary” and secondary  as “Operational Secondary” switch but it will show only as primary or secondary in the show vpc output as shown below

switch1# sh vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 2

Peer status                     : peer adjacency formed ok

vPC keep-alive status           : peer is alive

Configuration consistency status: success

Per-vlan consistency status     : success

Type-2 consistency status       : success

vPC role                        : primary

Number of vPCs configured       : 3

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : -

Graceful Consistency Check      : Disabled (due to peer configuration)

switch2# sh vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 2

Peer status                     : peer adjacency formed ok

vPC keep-alive status           : peer is alive

Configuration consistency status: success

Type-2 consistency status       : success

vPC role                        : secondary

Number of vPCs configured       : 3

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : -

However in case of primary switch failure the secondary switch will take over as “Operational primary” although it is configured as secondary and that is what you will see

switch2# sh vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 2

Peer status                     : peer adjacency formed ok

vPC keep-alive status           : peer is alive

Configuration consistency status: success

Type-2 consistency status       : success

vPC role                        : secondary, operational primary

Number of vPCs configured       : 3

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : -

And when the primary switch comes online it will appear as “operational secondary” although it is configured as primary

Switch1# sh vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 2

Peer status                     : peer link is down

vPC keep-alive status           : peer is alive

Configuration consistency status: success

Per-vlan consistency status     : success

Type-2 consistency status       : success

vPC role                        : primary, operational secondary

Number of vPCs configured       : 3

Peer Gateway                    : Disabled

Dual-active excluded VLANs      : -

Graceful Consistency Check      : Disabled (due to peer configuration)

So The role is nonpreemptive, so a device may be operationally primary but secondary from a configuration perspective.

Q3)The default vpc priority is 1024 ?

A3) I think you are referring to vPC role priority. The default is 32667

Thanks

Hatim Badr

Actions

Login or Register to take actions

This Discussion

Posted August 26, 2011 at 3:15 PM
Stats:
Replies:36 Avg. Rating:4.9
Views:54545 Votes:0
Shares:0

Related Content

Discussions Leaderboard

Rank Username Points
1 15,007
2 8,150
3 7,725
4 7,083
5 6,742
Rank Username Points
165
82
69
65
55