Configure FCoE Nexus 5548UP - UCS 6120

[Juan Pablo Hidalgo Villacres]

Hi, I am a networking guy and i need to understand which configuration do i need to connect 2 UCS 6120 to my Nexus 5000 (2 with VPC),

I know that i must activate npiv and fcoe feature, and read about creating vsans and vfc, my first question is does anybody have a procedure? (for example: 1.- You must create a vsan, 2.- You need to associate vsan with vlans, 3.- etc)..

Secondly, i have some doubts,

1.- If i have fc interfaces do i need to create an vfc interface? (or this is only if i am going to use the 10G ethernet interfaces?.)

2.- Why i need to bind the vsan to the vlans?.

Please help me with this, the SAN world is new to me!!

Hope you could help me,

Regards,

Juan Pablo Hidalgo

[Charles Morrall]

Well, I got it all working after much tinkering. One lingering question, probably answered by reading the specifications but still: I had to configure the port from the FI (6120) as an "FCoE uplink port" in order to see the fabric logins in the Nexus switch. My initial assumption was that a "Network" port would be sufficient, Unified Fabric and all. The VLANs still carry, so I'm all good in this configuration, but to me it appears a bit limiting to only allow exactly one VSAN on an FCoE uplink. What if I want to have a multi-tennant solution with many more VSANs (although the FI supports 32 VSANs unless I'm mistaken).

[Rafael Guedes de Souza]

Hi,

We are planning to deploy a FCoE Multihop scenario where we have two FIC 6248UP connecting to two Nexus 5548UP configured in vPC. The question here is if we can carry ethernet and FCoE traffic through the same cable, as the hosts do using CNA's. When I heard about FCoE Multihop northbound support in Fabric Interconnect I always thought that FCoE traffic would pass on the same cable as the ethernet traffic. Is it not possible??

The only possible way I am seeing to do FCoE Multihop is to link FIC-A to N5K-A and FIC-B to N5K-B for FCoE traffic. And keep ethernet traffic using vPC in a classic fashion. Is it?

[Jeremy Waldrop]

You can use the current vPC uplink for both Ethernet and FCoE traffic if you configure the port as Unified within UCSM. If you configure the existing uplink port as an FCoE Uplink it will change the role to Unified Uplink.

After you change the port to Unified you will need to map the appropriate VSAN to it. To switch over from traditional FC to FCoE disable the FC uplinks.

[Rafael Guedes de Souza]

Hi Jeremy,

I just read this article http://blogs.cisco.com/datacenter/ucs-multihop-fcoe-in-under-an-hour/ and it seems that use vPC to carry FCoE + Ethernet traffic is not a valid scenario. See bellow:

3. With the Nexus 5k systems, you can PortChannel FCoE and Ethernet links together, or you can use vPCs. You cannot do both. If you want need to use vPCs on your Ethernet links, you will need to use separate links for FCoE (but you can port-channel those together, if you wish). Using UCS with a Nexus 7k you will need to use dedicated FCoE links between the Fabric Interconnects and the 7k.

Do you have any documentation about this?? I googled a lot, but it seems that Cisco did not release many related content yet.

Thank you!

[Jeremy Waldrop]

Hey Rafael, I was doing this in a lab just to see if it would work. I was able to get it working but due to the horrible performance of accessing a LUN from an ESXi host over multihop FCoE I had to revert back to traditional FC connections.

This performance thing is a major bug in my opinion. I tested it again last week on gen 2 hardware and a brand new VNX and had the same results. It was so bad I couldn't even get a Windows VM install to finish. In this setup I had 2 dedicated Ethernet uplinks configured in an FCoE Port Channel. Once I moved to traditional FC there were no performance issues.

[NaelShahid_2]

Hi Jeremy, So you reverted to native FC between the FI and Nexus or was it dedicated ‘SAN’ FCoE uplink with separate LAN uplinks (vPC)

[Jeremy Waldrop]

Reverted to native FC.

When I had dedicated FCoE links there was a serious performance issue. I only intialy setup the FCoE on the vPC links just to see if it could be done but I never used it.

I have tested this on both gen 1 6100s/2104/M81KR and gen 2 6200/2208/1240/1280 hardware and it is the same result. I can boot the ESXi hosts from the SAN boot LUN but when I power on a VM it takes 20+ minutes and I can't ever login to it.

I am using ESXi 5.1 with the latest VIC enic/fnic drivers.

[NaelShahid_2]

Sounds bad! I better order some FC SFPs....

One last question, from the Nexus to storage was this FCoE or native?

Thanks for the heads up.

[Jeremy Waldrop]

Native FC from an EMC VNX. In this same lab I have 2 rack mount ESXi servers with QLogic CNAs with FCoE going through the same Nexus 5548s and same VNX SAN with no performance issues.

[NaelShahid_2]

So this is a multi-hop issue then. I have done end to end fcoe deployments before direct from  host > nexus > storage without issues.

[loizosko]

it appears that when you can't bind the vfc interface to a portchannel with more than 1 link.

i have 2 nexus 5K uplinked via 2 vpc etherchannels to the ucs 6296

each vpc has 4 links (2 from each nexus)

i will see if i can make the uplinks in 4 different vpc (with 2 total connections each)

anybody has any feedback on this?

[vsathiam]

loizosko,

To configure ethernet network infrastructure, you can do the following - { just like yours, 4 links total, 2 links from each nexus }

Nexus 5k-1

interface ethernet 1/1

description Connected to Fabric-Interconnect-A Eth 1/13

channel-group 10 mode active

no shut

interface ethernet 1/2

description Connected to Fabric-Interconnect-B Eth 1/13

channel-group 11 mode active

no shut

interface port-channel10

switchport mode trunk

spanning-tree port type edge trunk

vpc 10

interface port-channel11

switchport mode trunk

spanning-tree port type edge trunk

vpc 11

Nexus 5k-2

interface ethernet 1/1

description Connected to Fabric-Interconnect-A Eth 1/14

channel-group 10 mode active

no shut

interface ethernet 1/2

description Connected to Fabric-Interconnect-B Eth 1/14

channel-group 11 mode active

no shut

interface port-channel10

switchport mode trunk

spanning-tree port type edge trunk

vpc 10

interface port-channel11

switchport mode trunk

spanning-tree port type edge trunk

vpc 11

Go to UCSM, and configure ports 1/13,1/14 (in Fabric-A) as Uplink Ports, Under Lan > Lan Cloud > Fabric A, you can create Port-channels. Repeat the above for Fabric B

Ethernet Uplink Port-channels should come up.

To extend SAN from Data center core to Fabric Interconnects, do the following - (assuming you are trying to configure using FCoE - between 5k and FI and FI is in end host mode)

To maintain SAN seperation, as an example, connect 2 links from 5k-1 to FI-A and 2 links from 5k-2 to FI-B. Create SAN Port-channel

On Nexus 5k-1

feature npiv

feature fcoe

feature lacp

vlan 100

name FCOE_VLAN_100

vsan database

vsan 100

vsan 100 name SAN100

vlan 100

fcoe vsan 100

interface Ethernet 1/21

description Connected to Fabric_Interconnect_A Eth 1/21

channel-group 30 mode active

no shut

interface Ethernet 1/22

description Connected to Fabric_Interconnect_A Eth 1/22

channel-group 30 mode active

no shut

interface port-channel 30

switchport mode trunk

switchport trunk allowed vlan 100

spanning-tree port type edge trunk

interface vfc 30

bind interface port-channel30

switchport trunk allowed vsan 100

switchport mode F

no shut

On Nexus 5k-2

feature npiv

feature fcoe

feature lacp

vlan 101

name FCOE_VLAN_101

vsan database

vsan 101

vsan 101 name SAN101

vlan 101

fcoe vsan 101

interface Ethernet 1/21

description Connected to Fabric_Interconnect_B Eth 1/21

channel-group 30 mode active

no shut

interface Ethernet 1/22

description Connected to Fabric_Interconnect_B Eth 1/22

channel-group 30 mode active

no shut

interface port-channel 30

switchport mode trunk

switchport trunk allowed vlan 101

spanning-tree port type edge trunk

interface vfc 30

bind interface port-channel30

switchport trunk allowed vsan 101

switchport mode F

no shut

Go to UCSM, define VSANS (find in SAN Cloud),  under Fabric A and Fabric B, create FCoE Port-channels (select appropriate interfaces), associate the Port-channels to correct VSANs.

Refer

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.1/b_UCSM_GUI_Configuration_Guide_2_1_chapter_0110.html

[loizosko]

i wanted to have 4 10GB links out of each FI

your solution only has 2 out of each FI

now i have 2 interfaces on each port channel  (4 total per vpc)

your solution calls for only 2 uplinks from each FI

review the 2 diagrams below

[vsathiam]

Ioizosko,

Nexus 5k-1

feature lacp

interface ethernet 1/1

description Connected to Fabric-Interconnect-A Eth 1/13

channel-group 10 mode active

no shut

interface ethernet 1/2

description Connected to Fabric-Interconnect-A Eth 1/14

channel-group 10 mode active

no shut

interface ethernet 1/3

description Connected to Fabric-Interconnect-B Eth 1/13

channel-group 11 mode active

no shut

interface ethernet 1/4

description Connected to Fabric-Interconnect-B Eth 1/14

channel-group 11 mode active

no shut

interface port-channel10

switchport mode trunk

spanning-tree port type edge trunk

vpc 10

interface port-channel11

switchport mode trunk

spanning-tree port type edge trunk

vpc 11

Nexus 5k-2

feature lacp

interface ethernet 1/1

description Connected to Fabric-Interconnect-A Eth 1/15

channel-group 10 mode active

no shut

interface ethernet 1/2

description Connected to Fabric-Interconnect-A Eth 1/16

channel-group 10 mode active

no shut

interface ethernet 1/3

description Connected to Fabric-Interconnect-B Eth 1/15

channel-group 11 mode active

no shut

interface ethernet 1/4

description Connected to Fabric-Interconnect-B Eth 1/16

channel-group 11 mode active

no shut

interface port-channel10

switchport mode trunk

spanning-tree port type edge trunk

vpc 10

interface port-channel11

switchport mode trunk

spanning-tree port type edge trunk

vpc 11

In UCSM, configure appropriate Ethernet uplink ports (and create Port-channel configurations) for both Fabrics.

[loizosko]

this is what i currently have

but with this setup cannot pass FCOE

when i try associate the vfc interface to the port channel i get this errorr

N5K-SW1(config-if)# bind interface port-channel 2

ERROR: fcoe_mgr: VFC cannot be bound to Port Channel as it has more than one member (err_id 0x4207002C)

N5K-SW1(config-if)#

all the articles indicate that can't do in this case.

if i do the scenario i indicated (possible solution??) might work but did not test it.

as alternative i could have a separate FC port channel but i would like to see if i can get it via fcoe