LACP Channel to a Server across 2 Nexus 2K FEX

[awatson20]

Currently, we have a single Nexus 5K with a pair of Nexus 2K FEX connected to the single 5K.  We are adding a 2nd 5K and will have a vPC domain, and the FEX will be dual homed to each 5K in a vPC but currently there is just 1.  We are wanting to configure an LACP channel to a server/firewall that has 4 connections.  We wanted to connect 2 connections to FEX1, and the other 2 to FEX2 all in the same port-channel.  When we do this, we received this message.

command failed: port not compatible [vPC cannot be defined across ST and AA FEX ]

Will this type of configuration have to wait until the other Nexus 5K is installed and we have a VPC domain, and the FEX dual homed to the 5K's in a vPC?

[richbarb]

Hi,

Yes, you must have your fex dual connected to configure it. In a failure scenario it works with only one path, but to bring up at the first moment you must have the fex dual attached.

Remember to configure a vpc in a dual-homed scenario you don't need configure the vpc command under the port-channel to host.

Richard

[awatson20]

Thanks Richard. Just to clarify, once the FEX are dual connected in a vPC, then we can have the single host dual connected to each FEX (4 connections) in the same channel group?

Also, in order to dual home to the host on the 2K's, do the FEX need to be dual connected to a single 5K and not to both?

Sent from Cisco Technical Support iPhone App

[richbarb]

Yes, you can have 4 connections from two FEXes dual attached to the same host and you can use 802.3ad as well.

As I said, your design must be complete to do so, I mean, you need both FEX connected at both N5k at the same time to start working. The host can be also dual-attached.

This is the documentation reference: (There is a very good configuration example in the end).

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5500/sw/layer2/6x/b_5500_Layer2_Config_602N12_chapter_0101.html#task_32A6F57E59864B169B5396F52A4FE023

Please rate useful answers.

Regards,

Richard

[Steve Fuller · Accepted solution]

Hi,

Another document that is definitely worth a look is Mark Stinnette's Quick Start Guide :: Virtual Port Channel (vPC). What you'll eventually have is shown on slide 21 of the PowerPoint, and I have to say, shown in a very easy to understand way. As an aside, Mark's also done a great job with a FabricPath quick start guide if you ever look at that feature.

Just to add, and sorry Richard, question the point that you won't be able to configure this until you have both N5K, I wonder if the vPC auto-recovery command would allow you to get things working. The command comes into play when both vPC peers are reloaded at the same time e.g., due to power outage, but only one of the devices becomes operational. The device that comes up assumes the vPC primary role and bring up all local vPCs. You can find out more details in the Enabling vPC Auto-Recovery section of the Nexus 5000 interfaces configuration guide.

And a couple of points on dual homed FEX:

• Just to confirm, as you don't mention it, that you do have Nexus 5500 series switches? Dual homed FEX requires Enhanced PC and this is not supported on the Nexus 5000 series switches.
• You might also like to take a look at the forum post N2K dual homing to N5K final sentence. On the face of it dual homed FEX seem a wonderful idea, but I've had some bad experiences, and there are some other considerations e.g., the number of FEX supported across an N5K pair. I'm not trying to put you off, just make you aware.

Regards

[mcoller]

Hi there,

I don't think this question is answered.  I have the same issue.  Forget for a minute about the second N5k mentioned in the initial question and any potential vPC between them.  His issue (and mine) is seen on a single N5k with two N2k directly attached.  I am not intending to use vPC or to dual-home the N2ks to multiple N5ks.

I assumed you should be able to have a server with four NICs to have two in N2k-1 and two in N2k-2, and have a single port channel that feeds all four switchports, but this is not working.  The two N2ks are being fed from the same N5k fabric, so this should be no different from putting 2 ports in one 6500 line card and 2 ports in a different card and having a single etherchannel with all 4 ports.  Or is it?

I can successfully put a port from one N2k into the port channel (and can add a second port from the same FEX), but when I try to add a port from the other N2k I receive the same error message: "command failed: port not compatible [vPC cannot be defined across ST and AA FEX ]".  This is exactly the same when using LACP or just using "mode on".

Config commands:

N5k(config)# int po 501

N5k(config-if)# int e101/1/12 -13
N5k(config-if-range)# channel-gr 501 mode on
N5k(config-if-range)# int e102/1/33
N5k(config-if)# channel-gr 501 mode on
command failed: port not compatible [vPC cannot be defined across ST and AA FEX ]

N5k# sh port-ch summ | in 501
501   Po501(SD)   Eth      NONE      Eth101/1/12(D)  Eth101/1/13(D)

N5k# sh fex
  FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
------------------------------------------------------------------------
101    N5k-FEX101          Online   N2K-C2248TP-E-1GE   xxxxxx
102    N5k-FEX102          Online   N2K-C2248TP-E-1GE   xxxxxx

So, why is this a vPC:  is it because it is spanning two different N2ks even though it is on a single N5k?

Is this not possible?  Why?

What does it mean about ST and AA FEXes?  Is it Straight-Through and Active-Active?  What is the difference and who do you configure it?

Am I going to have to put all four server connections onto the same FEX to get a single 4-port port-channel?

Thanks in advance for your help.  Sorry about resurrecting an old thread but it seemed better than opening a new one (or is that wrong?)

 Mike

[awatson20]

Thank you very much for the information.

Yes, these are 5548's.  In our main data center, we dual home the FEX to a pair of Nexus 5548's.  We do not dual home the hosts to the FEX.  The scenario I describe above is in our backup data center where we currently have 1 5548 and 2 FEX, but we are going to be adding another 5K soon.  We do have a need here to dual home a host to the 2 FEX, but after reading through the forum I am wondering if its better to single home the FEX to a single 5K.  Thanks again for the excellent information.

[Steve Fuller]

My personal preference is to use dual homed FEX when needed e.g., I'm connecting a large number of single attached hosts. In the case where hosts are dual (or quad) homed, I'd let the teaming software on them take care of a failure of a single FEX or N5K.

Glad to be of help. Appreciate if you would mark the post as answered to help others with similar questions in the future.

Regards

[awatson20]

Just one more question Steve. 

When you say, In the case where hosts are dual (or quad) homed, I'd let the teaming software on them take care of a failure of a single FEX or N5K.

Is this with FEX Dual homed or Single Homed that you are referring to?

[Steve Fuller]

I was referring to when the FEX single homed. My preference is for connectivty as follows:

So where hosts are connected to two different FEX i.e., we have resilient connectivity and OS NIC teaming, then I can tolerate an outage of a single FEX or a single N5K and the NIC teaming software will restore connectivity. I don't really need the additional complexity of dual homed FEX to maintain connectivity. This is as indicated on the illustration on the left of the diagram.

Where my servers are single attached I want to make my FEX as resilient as possible because if I lose a FEX there is no resilience for the hosts connected to the failed FEX. As the FEX could fail because of an N5K failure then dual homing to two N5K gives me some additional resilience.

Regards