Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tech-Talk Virtual Port Channel ( vPC ) Operation and Design Best Practices

Bronze

I am Vinay Kumar (CCIE R&S # 35210) and I work for the High Touch Technical Support (HTTS) team, a team that provides reactive technical support to majority of Cisco’s premium customers. Vivek Ruhil (CCIE SP # 40530) , Pradeep Malik ( CCIE R&S, SP # 25302 ) who works for Cisco Advance Services Team that provides Plan Design and Implementation Services to Cisco’s customer and we have created a video and this blog on Virtual Port channel (vPC)

 

In today's data centers, there is constant worry about high-availability and resiliency to ensure that customers can get to their data, while meeting a service level agreement (SLA) that is very close to 100%. To achieve this, we must look at every possible way that the network (or the gear that makes up the network) could fail without impacting data transmissions. This is where virtual port channel technology (VPC) enters. Virtual port channel (VPC) takes redundancy to a new level in the data center.

 

The whole concept of VPC is to join two switches together and make them function as if they were a single switch and act to connected devices as one logical switch. VPC is currently available on the Cisco Nexus 5000 and 7000 series switches, as well as some high-end Catalyst models. We have covered the following topics in both the blog and the tech-talk video.

  1. vPC Overview and Operation
  2. Configuration and Show commands
  3. vPc Failure Recovery
  4. General Recommendations to ensure Stable operation of a Virtual Port-channel topology

VPC tech talk video.PNG

Overview:-

 

vPC is a Port-channel concept extending Link aggregation to Two Separate Physical switches. To understand this concept better lets first discuss about the traditional L2 networks using STP.

 

Image 1

 

This is a traditional Layer 2 network with one Primary root and other switches a secondary root running STP. All the switches have their link connected to Secondary root switches in blocked state. Loop avoidance is based on STP and VLAN based load balancing will be used.

 

With vPC

 

A virtual PortChannel (vPC) allows links that are physically connected to two different Cisco Nexus Series devices to appear as a single PortChannel to a third device. The third device can be a Cisco Nexus 2000 Series Fabric Extender or a switch, server, or any other networking device. A vPC can provide Layer 2 multipathing, which allows you to create redundancy by increasing bandwidth, enabling multiple parallel paths between nodes and load-balancing traffic where alternative paths exist.

 

Image 2

 

With vPC we achieve a Loop free topology without keeping any uplink in blocked state. A vPC provides the following benefits:

 

• Allows a single device to use a PortChannel across two upstream devices

• Eliminates Spanning Tree Protocol blocked ports

• Provides a loop-free topology

• Uses all available uplink bandwidth

• Provides fast convergence if either the link or a device fails

• Provides link-level resiliency

• Helps ensure high availability

 

Virtual Port-channel ( vPC  ) terminology:-

 

Image 3

 

vPC: vPC refers to the combined Port-Channel between the vPC peer devices and the downstream device.

 

• vPC peer switch: The vPC peer switch is one of a pair of switches that are connected to the special PortChannel known as the vPC peer link. One device will be selected as the primary device, and the other will be the secondary device.

 

• vPC peer link: The vPC peer link is the link used to synchronize states between the vPC peer devices. The vPC peer link carries control traffic between two vPC switches and also multicast, broadcast data traffic. In some link failure scenarios, it also carries unicast traffic. The vPC peer link is a standard 802.1Q layer 2 trunk which potentially carries all VLANs. The vPC Peer Linkalso carries the critical CFS (Cisco Fabric Services) traffic.

 

• vPC domain: This domain includes both vPC peer devices, the vPC peer keep alive link, and all the PortChannels in the vPC connected to the downstream devices. It is also associated with the configuration mode that you must use to assign vPC global parameters.

 

• vPC peer keepalive link: The peer keep alive link monitors the vitality of a vPC peer switch. The peer keep alive link sends periodic keep alive messages between vPC peer devices. The vPC peer keep alive link can be a management interface or switched virtual interface (SVI). No data

or synchronization traffic moves over the vPC peer keep alive link; the only traffic on this link is a message that indicates that the originating switch is operating and running vPC.

 

• vPC member port: vPC member ports are interfaces that belong to the vPCs.

 

vPC VLAN : One of the VLAN carried over the Peer Link and used to communicate via vPC with a peer device.

 

• Non- vPC VLAN: One of the STP VLAN not carried over the peer-link.

 

• CFS—Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC

 

vPC Configuration Consistency Check:-

 

In traditional Port-channels before two link can form a port-channel they must agree on certain parameters and similarly virtual PortChannels are subject to consistency checks and compatibility checks.

 

During a compatibility check, one vPC peer conveys configuration information to the other vPC peer to verify that vPC member ports can actually form a PortChannel. For example, if two ports that are going to join the channel carry a different set of VLANs, this is a misconfiguration. Depending on the severity of the misconfiguration, vPC may either warn the user (Type-2 misconfiguration) or suspend the PortChannel (Type-1 misconfiguration). In the specific case of a VLAN mismatch, only the VLAN that differs between the vPC member ports will be suspended on all the vPC PortChannels.

Inconsistencies can be global or interface specific:

 

● Global inconsistencies: Type-1 global inconsistencies affect all vPC member ports (but do not affect non-vPC ports)

 

● Interface-specific inconsistencies: Type-1 interface-specific inconsistencies affect only the interface itself The vPC Type 1 configuration parameters are as follows:

 

• Port-channel mode: on, off, or active

• Link speed per channel

• Duplex mode per channel

• Trunk mode per channel

– Native VLAN

– VLANs allowed on trunk

– Tagging of native VLAN traffic

• Spanning Tree Protocol (STP) mode

• STP region configuration for Multiple Spanning Tree

• Enable/disable state the same per VLAN

• STP global settings

– Bridge Assurance setting

– Port type setting—We recommend that you set all vPC peer link ports as network ports.

– Loop Guard settings

• STP interface settings:

– Port type setting

– Loop Guard

– Root Guard

• Maximum transmission unit (MTU)

• Allowed VLAN bit set

 

vPC Configuration Synchronization:-

 

 

A vPC allows two links that are physically connected to two Cisco Nexus switches to appear as a single PortChannel. Some configurations must be identical on both switches for vPCs to forward traffic. Such configurations include port mode, channel mode, speed, and duplex.

 

The config-sync command simplifies the management of vPCs by synchronizing vPC configurations between primary and secondary vPC peers.

 

vPC config-sync is currently available on the Cisco Nexus 5000 Series starting from Cisco NX-OS 5.0(2)N1(1). The config-sync feature uses the concept of the configuration profile. The switch profile is the construct that allows configurations to be applied both locally and on the config-sync peer. The config-sync peer definition is independent of the vPC peer definition and is specified in the switch profile configuration mode as follows:

 

Nexus5000(config-sp)# sync-peers destination {destination IPs}+ [source <source IP> | vrf <vrf>]

 

Note: Even if the config-sync peer is the same as the vPC peer device, the config-sync infrastructure has been designed so that it can be decoupled from vPC. Thus, you need to define the config-sync peer even in presence of a vPC configuration.

 

After the config-sync peer has been defined, the configuration that uses vPC config-sync appears as follows:

 


 

Switch# config sync

Switch(config-sync)# switch-profile profiledefinition

Switch(config-sp)# interface Port-channel100

Switch(config-sp-if)# interface Ethernet1/1

Switch(config-sp-if)# channel-group 100

Switch(config-sp-if)# exit

Switch(config-sp)# commit

 

Configurations are applied only after the user enters a commit command. The configuration is synchronized with the remote peer through the mgmt0 interface using routable Cisco Fabric Service protocol over IP. If the remote peer cannot be reached, the configuration is applied only locally.

 

All commit operations follow the two-phase commit approach: If the config-sync peer is reachable, either the configuration is fully committed on both peers or it is rolled back on both. If the config-sync peer is not reachable, then the configuration is applied only locally. When the peer becomes reachable, the configurations are merged

 

vPC Configuration:-

 

vPC configuration includes these steps

 

1. Enable the vPC feature.

 

N7K-LAB (config )#feature vpc

 

The vPC feature must be enabled before it can be configured.

 

2. Create a vPC domain and assign a domain ID.

 

N7K-LAB (config )#vpc domain 1

 

3. Configure the vPC peer keep alive link.

 

N7K-LAB (config-vpc-domain)# peer-keepalive destination 172.28.230.85

 

Configure the IPv4 address for the remote end of the vPC peer keepalive link. The system does not create the vPC peer link until you configure a vPC peer keepalive .

 

4. Configure system priority. ( Optional )

 

N7K-LAB (config-vpc-dom ain)#system -priority2000

 

The range of values is 1 to 65535. The default value is 32667. You should manually configure the vPC system priority when you are running Link Aggregation Control Protocol (LACP) to help ensure that the vPC peer devices are the primary devices on LACP. When you manually configure the system priority, make sure that you configure the same priority value on both vPC peer devices. If these values do not match, vPC will Not be activated.

 

5. Configure vPC role priority. ( Optional )

 

N7K-LAB (config-vpc-dom ain)#role priority1000

 

The range of values is 1 to 65636, and the default value is 32667. The switch with lower priority will be elected as the vPC primary switch. If the peer link fails, vPC peer will detect whether the peer switch is alive through the vPC peer keepalive link. If the vPC primary switch is alive, the vPC secondary switch will suspend its vPC member ports to prevent potential looping while the vPC primary switch keeps all its vPC member ports active.

 

6.     Create the vPC peer link.

 

N7K-LAB (config)# interface port-channel 20

N7K-LAB (config-if)# vpc peer-link

 

Select the PortChannel that you want to use as the vPC peer link for this device, and enter the interface configuration mode. Configure the selected PortChannel as the vPC peer link.

 

7.   Move the PortChannel to vPC.

 

N7K-LAB (config)# interface ethernet2/2

N7K-LAB (config-if)# channel-group20

N7K-LAB (config)# interface port-channel 20

N7K-LAB (config-if)# vpc 30

 

Add the interface to the PortChannel and then move the PortChannel to the vPC to connect to the downstream device. The vPC number ranges from 1 to 4096. The vPC number does not need to match the PortChannel number, but it must match the number of the vPC peer switch for that vPC bundle.

 

A PortChannel is needed even if there is only one member interface for the PortChannel. When there is only one member for the PortChannel, the hardware PortChannel resource Will not be created.

 

vPC Operation Verification:-

 

Following show commands helps to verify if vPC is working fine or not.

 

1.      Show vpc brief

 

This command shows the vPC domain ID, the peer-link status, the keepalive message status, whether the configuration consistency is successful, and whether peer-link formed or failed.

 

 


 

N7K-LAB(config)# show vpc brief
 

 

Legend:

 

                (*) - local vpc is down, forwarding via vPC peer-link

 

 

 

vPC domain id                   : 10

 

Peer status                     : peer adjacency formed ok

 

vPC keep-alive status           : peer is alive

 

Configuration consistency status: success

 

vPC role                        : primary

 

Number of vPC configured        : 1

 

 

 

vPC Peer-link status

 

---------------------------------------------------------------------

 

id   Port   Status Active vlans

 

--   ----   ------ --------------------------------------------------

 

1    Po10   up     1-100

 

 

 

vPC status

 

----------------------------------------------------------------------

 

id   Port   Status Consistency Reason              Active vlans

 

--   ----   ------ ----------- -------------------------- ------------

 

20   Po20   up     success     success             1,11,35,99

 

The above commands output shows that our vPC is working perfectly fine. vPC status is up, Peer is alive and there is no configuration inconsistency.

 

Now suppose we have some vPC Configuration consistency failure in which case we have output something like this.


 

switch(config)# show vpc brief

 

Legend:

 

                (*) - local vpc is down, forwarding via vPC peer-link

 

 

vPC domain id                   : 10

 

Peer status                     : peer adjacency formed ok

 

vPC keep-alive status           : peer is alive

 

Configuration consistency status: failed

 

Configuration consistency reason: vPC type-1 configuration incompatible - STP interface port type inconsistent

 

 

vPC role                        : secondary

 

Number of vPC configured        : 1

 

 

vPC Peer-link status

 

---------------------------------------------------------------------

 

id   Port   Status Active vlans

 

--   ----   ------ --------------------------------------------------

 

1    Po10   up     1-100

 

 

vPC status

 

----------------------------------------------------------------------

 

id   Port    Status                 Consistency Reason                     Active vlans

 

--   ----   ------ ----------- -------------------------- ------------

 

20   Po20   up     failed      vPC type-1 configuration   -

 

                                         incompatible - STP interface port type

 

                                         inconsistent

 

 

Now we see that There is some configuration inconsistency because of STP interface type so we should correct for proper operation of vPC.

 

2.    Show vPC role

 

 

This command output shows the role of vPC, vPC mac address, vPC System priority , Mac address of the local device and role priority of local device.

 


 

N7K-LAB(config)# show vpc role

 

vPC Role status

 

----------------------------------------------------

 

vPC role                                  : secondary

 

Dual Active Detection Status    : 0

 

vPC system-mac                     : 00:13:04:ee:B1:F8

 

vPC system-priority                  : 32667

 

vPC local system-mac               : 00:0d:4c:a3:5c:bc

 

vPC local role-priority                  : 5000

 

 

3.     Show vPC Peer-keepalive

 

This command shows the IP address of the vPC Peer with which keep alive messages are exchanged and details about the last keep alive messages exchanged.

 


 

N7K-LAB(config)# show vpc peer-keepalive

 

vPC keep-alive status           : peer is alive

 

--Peer is alive for                  : (310) seconds, (29) msec

 

--Send status                      : Success

 

--Last send at                      : 2013.08.14 11:42:00 176 ms

 

--Sent on interface               : mgmt0

 

--Receive status                  : Success

 

--Last receive at                   : 2013.08.14 11:42:01 5 ms

 

--Received on interface         : mgmt0

 

--Last update from peer         : (0) seconds, (818) msec

 

vPC Keep-alive parameters

 

--Destination                       : 10.0.0.2

 

--Keepalive interval               : 1000 msec

 

--Keepalive timeout               : 5 seconds

 

--Keepalive hold timeout        : 3 seconds

 

--Keepalive vrf                     : management

 

--Keepalive udp port             : 3200

 

--Keepalive tos                    : 192

 

 

4.     Show vPC statistic peer-keeplive

 

 

This command output shows the statistic of Peer keep alive messages.


 

N7K-LAB# show vpc statistics peer-keepalive

 

 

vPC keep-alive status           : peer is alive                

 

VPC keep-alive statistics

 

----------------------------------------------------

 

peer-keepalive tx count:          1036

 

peer-keepalive rx count:          1028

 

average interval for peer rx:     995

 

Count of peer state changes:      1  

 

 

vPC Failure Recovery:-

 

 

We do have certain scenarios in which vPC operations may experience failure. This section will explain the failure condation and how we can recover from those

 

 

 

1. vPC Member port failure

 

If any of the link in the port-channel fails, traffic from the affected flows is redistributed on the remaining links.

 

 

2. vPC Peer Link Failure

 

In a vPC topology, one vPC peer switch is elected as the vPC primary switch and the other switch is elected as the vPC secondary switch, based on the configured role priority for the switch. If vPC peer link goes down, the vPC secondary switch shuts down all of its vPC member ports if it can still receive keepalive messages from the vPC primary switch (which indicates that the vPC primary switch is still alive). The vPC primary switch keeps all of its interfaces up. As a result, the hosts or switches redistribute all the flows to the vPC member ports that are connected to the vPC primary switch.

 

3. vPC Peer Keep alive link failure

 

 

The vPC keepalive link carries the heartbeat message between two vPC peer switches. The failure of the vPC keepalive link alone does not impact the vPC operation or data forwarding. Although it has no impact on data forwarding, we recommend that you fix the keepalive as soon as possible to avoid a double failure scenario that could impact the data traffic.

 

4. vPC Peer switch Failure

 

 

If one of the vPC Peer switch fails , other switch will take over. Now suppose there is a situation where both the switches loose power and after the power is restored only one switch comes up , in that case vPC primary selection can’t takes places as a result vPC switch keeps the vPC ports in suspended mode.

 

To fix these problems, use the reload restore feature and the auto recovery feature as follows:

 

In NX-OS Release 5.0(2)N1(1), enter the reload restore command:

 

N7K-LAB(config-vpc-domain)# reload restore <timeout in second>

 

In NX-OS Release 5.0(2)N2(1), enter the auto-recovery reload-delay command:

 

N7K-LAB(config-vpc-domain)# auto-recovery reload-delay ?

 

  <240-3600>  Time-out for restoring vPC links (in seconds)

 

These commands allow the vPC peer switch to bypass the vPC consistency check and bring up vPC ports after the delay timer expires.

   

5. vPC Peer Link Failure Followed by a Peer Keepalive Link Failure

 

 

If a peer link failure occurs, the vPC secondary switch checks if the primary switch is alive. The secondary switch suspends its vPC member ports after it confirms that the primary switch is up.

 

If the primary switch fails and secondary switch stop receiving the keep alive messages after the three consecutive keep alive messages timeout , secondary  switch takes the role of the primary and brings up its vPC members ports

 

 

6. vPC Keepalive Link Failure Followed by a Peer Link Failure

 

If the vPC keepalive link fails first and then a peer link fails, the vPC secondary switch assumes the primary switch role and keeps its vPC member ports up.

 

If the peer link and keepalive link fails, there could be a chance that both vPC switches are healthy and the failure occurs because of a connectivity issue between the switches. In this situation, both vPC switches claim the primary switch role and keep the vPC member ports up. This situation is known as a split-brain scenario. Because the peer link is no longer available, the two vPC switches cannot synchronize the unicast MAC address and the IGMP group and therefore they cannot maintain the complete unicast and multicast forwarding table.

 

vPC Design Recommendations:-

 

To ensure stable vPC operations we have following recommendations.

 

vPC Peer Link

 

  • The vPC Peer Link should be formed of a minimum of 2 x 10GE links on      separate linecards
  • The 10GE ports should be configured to operate in dedicated bandwidth      mode
  • Configure vPC Peer Link ports as STP 'Network' ports to enable      Spanning Tree Bridge Assurance on these links (assuming BA is enabled      globally).
  • Enable UDLD on vPC Peer Links
  • Running a dedicated VLAN / SVI across the peer link to cater for      uplink failure (e.g. to the core) may also be desirable.

 

vPC Peer Keep-alive Link

 

The vPC Peer-keepalive Link (PK-link) is used to provide protection against dual active scenarios in the event of the primary Peer Link being lost. If loss of the Peer Link takes place, the PK-link is used to determine the status of the opposite peer (in other words, to determine whether the loss of connectivity is due to link failure or node failure).

 

The PK-Link uses a simple heartbeat between vPC peers these messages are sent out every 2 seconds, and the link uses a 3 second hold timeout upon loss of the primary Peer Link.

 

In order of preference, the following types of interface should be used for the vPC PK-link:

 

  1. Dedicated Link (1Gbps is sufficient) using dedicated VRF
  2. mgmt0 interface (shared link with management traffic)
  3. Routed over L3 infrastructure (least preferred)

 

In the event that the chassis is not equipped with any 1GE linecards (i.e. the chassis supports 10GE only), then option 1 becomes less desirable - it is considered inefficient and expensive to dedicate a 10GE connection solely for the PK-link. In this case, option 2 (mgmt0 interface) should be considered.

 

One topology which should not be considered is where the vPC PK-link is routed across the vPC Peer Link. This defeats the object of having a dedicated PK-link - the PK-link is designed to act as a backup for the primary Peer Link, therefore it does not make sense to route the PK-link across the Peer Link.

 

vPC member link recommendations

 

The configuration of the vPC member links should be identical across the two vPC peer switches in particular, the vPC ID must match across the two peers, while the Port Channel number should match across the two peers for ease of management

 

Configuration parameters which must match across vPC peers

 

There are a number of configuration parameters which must be identical across the two vPC Peer Switches. Failure to comply with this requirement will result in the vPC moving into suspend mode. The configuration parameters which must be identical are as follows:

 

  • Port-channel mode: on, off, or active
  • Link speed per channel
  • Duplex mode per channel
  • Trunk mode per channel:
  • Native VLAN
  • VLANs allowed on trunk
  • Tagging of native VLAN traffic
  • Spanning Tree Protocol (STP) mode
  • STP region configuration for Multiple Spanning Tree
  • Enable/disable state per VLAN
  • STP global settings:
  • Bridge Assurance setting
  • Port type setting-We recommend that you set all vPC interfaces as network ports.
  • Loop Guard settings
  • STP interface settings:
  • Port type setting
  • Loop Guard
  • Root Guard
  • Maximum Transmission Unit (MTU)

 

Watch the Tech-Talk and you may download a copy of Presentation Slides here.

9 Comments
New Member

When is video coming ?

Cisco Employee

Hi Vishal

Link to the Video:

https://supportforums.cisco.com/videos/7328

Silver

If you are having issues with viewing the Video - Please check your browser's security settings.

video-help.png

Good topic  and good way explained

New Member

Hi,

I am having one query on the vpc system-mac address which is been creating automatically from the domain -id .

In my case, eg we are using the domain id as 701 & below is the result(system-mac) of the same.

vPC Role status
----------------------------------------------------
vPC role                        : secondary, operational primary
Dual Active Detection Status    : 0
vPC system-mac                  : 00:23:04:ee:c0:bd             
vPC system-priority             : 32667
vPC local system-mac            : xx.xx.xx.xx.xx.xx        
vPC local role-priority         : 100

As you see the HEX value for 701 is 2bd and it seems the system-mac is changed only with the last octet(bd) of domain-id.

Incase now I am going to have another one vPc pair with the domain id as 957 & HEX value for 957 as 3BD .so again the vpc system-mac will take the last octet of the domain id & it would be same as the above one - 00:23:04:ee:c0:bd .

Now this will create some problem as 2 different vpc pairs have the same system-mac address .

My query is ,

1) Is my understanding wrt the vpc system-mac is correct? If I am wrong, kindly explain the process of automatic system-mac creation from the vpc domain id.

2) if so the recommended action is to create the system-mac manually or by automatic?

 

Thanks,

Vel

 

Cisco Employee

Hi Vel

The domain ID is encoded in the last octet and the trailing 2 bits of the previous octet of the MAC address.

So your understanding is correct. 

My recommendation is to choose a domain-id whose HEX value doesn't come out to be same. And as such domain-id is just a value you can choose it to be anything.

 

Regards

RV

Hi,

I have a vPC topology up and running using 2 Nexus 5596. I need to exchange those 5596 for two 5600 Nexus Switches.

In order to migrate the solution with minimum downtime, is it possible for me to disconnect one of the 5596 and connect a preconfigured 5600? As long as all the parameters are consistent with one another would the vPC go up again?

Or is it mandatory for me to have 2 identical Nexus switches ?

 

Thank you in advance! 

 

Cisco Employee

Hi 

 

As long as you have the same configuration on the switches in a vPC pair, there is no problem. It is not a requirement to have the same chassis type in a vPC pair.

 

Before migration do ensure that you have Graceful Consistency Check Enabled.

 

Thanks

Vivek Ruhil

Ok Vivek will do!

 

Thank you so much for your reply!

 

 

6617
Views
15
Helpful
9
Comments