cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
33142
Views
9
Helpful
32
Replies

Configure FCoE Nexus 5548UP - UCS 6120

Hi, I am a networking guy and i need to understand which configuration do i need to connect 2 UCS 6120 to my Nexus 5000 (2 with VPC),

I know that i must activate npiv and fcoe feature, and read about creating vsans and vfc, my first question is does anybody have a procedure? (for example: 1.- You must create a vsan, 2.- You need to associate vsan with vlans, 3.- etc)..

Secondly, i have some doubts,

1.- If i have fc interfaces do i need to create an vfc interface? (or this is only if i am going to use the 10G ethernet interfaces?.)

2.- Why i need to bind the vsan to the vlans?.

Please help me with this, the SAN world is new to me!!

Hope you could help me,

Regards,

Juan Pablo Hidalgo

32 Replies 32

Vincent Jamart
Cisco Employee
Cisco Employee

Hi Juan Pablo,

Please follow this guide for details:

http://www.cisco.com/en/US/products/ps10281/products_configuration_example09186a0080afd130.shtml

UCS Fabric Interconnect can work in two modes: native (NPV) or switch.

By default it's NPV, so it means you need to create TF-N link with Nexus 5k (with NPIV feature active on N5k) for FC links. There is no direct FCoE support between N5k and FI at this point (

https://supportforums.cisco.com/message/3497244#3497244)

-On each N5k (fabric A, VSAN10 and fabric B, VSAN20):

feature fcoe

feature npiv

interface fcx/y (attached to UCS FI)

  switchport mode F

  switchport trunk VSAN 10 (20 on B)

  no shutdown

-For FC interfaces, there is no need to create a vfc interface, it's mandatory indeed for 10GigE ports that attach to a CNA: you bind each GigE to a vfc.

-VLAN-VSAN binding is mandatory: you need to have at least one VLAN in fcoe mode that binds to a VSAN, in order for CNA to register pWWN to the name server, then receive FCID.  You must also trunk default VLAN on the GigE port attached to CNA in order to allow the DCBX exchange with switch.

vsan database

vsan 10

vsan 10 interface fcx/y

vlan 10 (20 on B)

fcoe vsan 10 (20 on B)

# sh vlan fcoe

Original VLAN ID        Translated VSAN ID      Association State

----------------        ------------------      -----------------

      10                        10               Operational

interface Ethernetx/y

  switchport mode trunk

  spanning-tree port type edge trunk

  no shutdown

interface vfcy

  bind interface Ethernetx/y

  switchport trunk allowed vsan 1 (default VLAN)

  switchport trunk allowed vsan add 10 (20 on B)

  no shutdown

vsan database

vsan 10

vsan 10 interface vfcy

You also need to configure zoning on N5k.

I strongly recommend you to acquire this book if you're new to storage networking, as it covers FCoE and FC integration on Cisco product, procedure-like:

http://www.amazon.com/Cisco-Storage-Networking-Cookbook-Families/dp/146646318X

Best Regards,

Vincent

Hi Vicent,

Thanks for your reply and information!!,

I only have one more question, it's about the vlan and vsan, i don't understand what is the purpose of binding the vlan and vsan, i understand that the UCS and N5K needs the vsan, but why the binding, it is because i use the fcoe?,

is this vlan used for traffic between switches, or it is only used for the vsan and that's it.

again thanks you very much for your help,

Best Regards,

Juan Pablo

Hi Juan Pablo,

Binding VLAN-VSAN is needed only for FCoE: switch needs to know on which VLAN mapping it's expected to receive FC data, as FC is encapsulated in the ethernet frame while EtherType will be FCoE.

Also it will first receive FIP (FCoE initialization protocol=control plane) frames on that link, in order to discover/negociate FCoE-capable devices (CNA). This will be used for FCoE VLAN discovery, FCF and fabric login.

FCoE traffic (data plane) will then be received by switch.

Then binded VFC interface will collect the FCoE frames and send them to the FC fabric, decapsulated from ethernet.

So basically this VLAN-VSAN is used within switch (or between switches for multi-hop FCoE), but won't carry classic ethernet traffic. For this one you need to use other VLAN(s) trunked on the GigE port of that CNA.

Also, be aware that you need to disallow these FCoE VLANs over the vPC peer-link of your N5k pair since you need to enforce strict FC fabric A/B segregation for redundancy (load-balancing over fabrics is done by MPIO driver at host level and storage array side).

http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns945/ns1060/at_a_glance_c45-578384.pdf

Best Regards,

Vincent

hi Vicent,

Thanks for your reply and explanation.

Best Regards,

Juan Pablo

Jumping in in this thread as a latecomer, but I've not been able to find a straight answer.

"There is no direct FCoE support between N5k and FI at this point", is this still the case?

I'm a newbie in the Cisco (and pretty much Ethernet) world, I come from a traditional FC background. I'm trying to get my head around this, but if I can't connect FCoE natively between a Nexus 5548 and 6120XP FI, I have to rethink my design.

Hi Charles,

This is no more a limitation, we introduced support for north-bound FCoE (multihop) in latest UCS-M release 2.1(1a).

Please find more info in that article: http://blogs.cisco.com/datacenter/ucs-multihop-fcoe-in-under-an-hour/

I hope this answers your question.

Best Regards,

Vincent

Hi,

Thanks for the quick reply. I've read the post you linked, but I still feel I'm missing something quite fundamental.

I have a Cisco UCS system with the latest images, and it looks like its running 2.1(1a), so I should be covered.

There are two Fabric Interconnects, 6120XP connected to a single Nexus 5548UP. This is just a lab setup, nothing in production.

I've created a VLAN 1001, mapped this to VSAN 10. Created two vfc's, one on the port where Fabric Interconnet A connects, and one vfc on a port on the Nexus 5548 where I've got a Netapp unified storage port connected.

Both ports are trunk ports.

Created a VSAN 10 in UCS, linked it to VLAN 1001. I have an ESXi with a vHBA connected to VSAN 10.

Allowed VLAN 1001 on the ports where I have the link to the 6120 and the port where the NetApp connects.

show flogi database shows nothing

fcoe is enabled.

The 6120XP is in switch mode, but that should be ok now that we have FCoE multihop, or?

I've googled and googled, but since the multihop support is so recent, I'm still searching for some useful documentation on how to configure all this.

Hi Charles,

From our deployment experience, it should work fine if FI are in end-host mode (NPV), but in FC switching mode there may be a problem (hardware limitation on 1st generation FI 61xx). Ethernet switching mode is not in the scope for this limitation.

Please check if you have similar fault reported in your 6120XP switch-mode:

FI-A# scope fc-up
FI-A /fc-uplink # scope fab a
FI-A /fc-uplink/fabric # scope fcoeinterface x y
FI-A /fc-uplink/fabric/fcoeinterface # show fault
Severity  Code     Last Transition Time     ID       Description
--------- -------- ------------------------ -------- -----------
Major     F1084    2013-01-10T17:49:38.352    154788 FCoE uplink port 1/9 cannot
be supported on springfields when FI is in eth endhost mode and FC switching mode
Major     F1083    2013-01-10T15:27:45.126    105418 FCoE uplink is down on Vsan
1

If so, please modify design for FC NPV, or test with a 2nd gen FI 62xx.

Thanks.

Best Regards,

Vincent

I get this on the 6120:

fi-ucs-got-demo-A /fc-uplink/fabric/fcoeinterface # show fault

Severity  Code     Last Transition Time     ID       Description

--------- -------- ------------------------ -------- -----------

Cleared   F1082    2013-01-18T14:32:26.882    886546 FCoE uplink port 1/3 is down

Major     F1083    2013-01-18T14:32:11.505    886550 FCoE uplink is down on Vsan 1

Major     F1083    2013-01-18T14:32:11.505    886549 FCoE uplink is down on Vsan 10

fi-ucs-got-demo-A /fc-uplink/fabric/fcoeinterface #

And this on the Nexus 5548 where I have the link to the 6120 (Eth 1/3)

ucs# show int vfc 13

vfc13 is trunking (Not all VSANs UP on the trunk)

    Bound interface is Ethernet1/3

    Hardware is Ethernet

    Port WWN is 20:0c:54:7f:ee:e2:af:3f

    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled

    Port mode is TF

    Port vsan is 10

    Trunk vsans (admin allowed and active) (1,10,20)

    Trunk vsans (up)                       ()

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (1,10,20)

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      0 frames input, 0 bytes

        0 discards, 0 errors

      0 frames output, 0 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

    Interface last changed at Fri Jan 18 12:00:06 2013

Hello Charles,

Here is sample configuration to enable FCoE between UCS and N5K

#####  On UCSM 2.1  ######

1) FC end mode

2) Configure a port as FCoE uplink  and assign the interface to VSAN 10

## Verifying the output

6248-01-A(nxos)# show run int vfc 1255

interface vfc1255

  bind interface Ethernet1/24

  switchport mode NP

  no shutdown

##### Configuration on N5K  #####

1) Enable NPIV,fcoe and lldp features

2) Create vlan, configure ethernet and VFC interfaces

----------- sample configuration ----------------

conf t

vlan 1001

fcoe vsan 10

interface Ethernet1/19

switchport mode trunk

spanning-tree port type edge

spanning-tree bpdufilter enable

no shutdown

int vfc 14

bind interface Ethernet1/19

switchport trunk allowed vsan 1

switchport trunk allowed vsan add 10

no shutdown

vsan database

vsan 10 interface vfc 119

------------------------------------

####  Comands to verify the configuration  #####

show feature

show vlan

show fcoe vlan

show int vfc 119

show fcoe database

show flogi table

Hi padramas,

I did not understand why you used vfc14, and after vfc119? Could you explain please?

int vfc 14

bind interface Ethernet1/19

switchport trunk allowed vsan 1

switchport trunk allowed vsan add 10

no shutdown

vsan database

vsan 10 interface vfc 119

Another question is why did you allowed VSAN 1??

Lastly, if you would allow VLANs in the ethernet port, would allow VLAN 1001 or also VLAN 1?? I'm question this because I believe FIP relies in the native VLAN.. Is it right?

Thank you!

Thanks for the detailed reply. I'm probably way off here, but what if I wanted to run the FI in "switch mode" for FC? This is based on the assumption that now with FCoE multihop support, I can connect the FI to the Nexus 5548 as a switch and still have end to end FCoE connectivity-

Hi Charles,

The Nexus 5000 have hardware limitation for this, but issue don't exist for Nexus 5500 to run in FC host mode and FCoE multihop.

Best Regards,

Vincent

Charles,

We do not need FI to be in FC switch mode. FC end host mode ( NPV) allows FI ports ( NP ) to connect NPIV ( N5K switch ) .

If N5K software version allows multihop, you can then extend FCoE from N5K to another switch via VE port.

HTH

Padma

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: