Fibre Channel Over Ethernet (FCoE) - Questions and Answers from live Webcast

Document

Mar 28, 2011 9:03 AM
Mar 28th, 2011

Introduction

Ozden Karakok.jpg

Ozden Karakok is a Technical Leader at Global Technical Center in Europe. Ozden has been with Cisco for twelve years. She specializes in Storage Area and Data Center Networks. She worked as an escalation engineer in the Cisco Storage Technical Assistance Center before joining to Customer Advanced Engineering Team. She is currently supporting Data Center and Unified Computing Solutions through product testing and early field trials. She holds CCIE certification #6331 in Routing and Switching, SNA/IP and Storage Networking.  She is also VMware Certified Professional.

This document contains the answers provided for the questions asked during the live "Ask the Expert" Webcast session on the Topic - Fibre Channel over Ethernet (FCoE)



The following experts were helping Ozden to answer few of the questions asked during the session:  Ph.D. J Michel Metz and Nina Rezai-Mia. Both are data center technical experts at Cisco.



The Complete Recording of this live Webcast can be accessed  here.

The presentation provided during the live event is attached to this document. Scroll down to the bottom to be able to download it.

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;}

1. Fibre Channel Over Ethernet (FCoE)

1. Does NX-0S 5.0 run NPIV?

Yes, it is supported in NX-OS 5.0. Nexus 5000 platform supports N port  identifier virtualization (NPIV) and N-Port Virtualization(NPV) since NX-OS 4.0  version. Nexus 5K supports two modes. By default, Cisco Nexus 5000 and 5500  Series switches operate in fabric mode, in this mode switch provides standard FC  switching capability and features. Second mode is NPV, in NPV mode Nexus 5K  switches relays all the traffic from server-side ports oto the core switch.(NPIV  enabled switch). The core switch provides F port functionality and all FC  switching capabilities.

2. Is FCoE between data centers possible?

Yes, FCoE between Data Centers is possible. VE Ports allow you to build  multi-hop FCoE fabrics by interconnecting multiple FCF, VE port functions as a  Fibre Channel E-port running on top of a lossless Ethernet fabric. The maximum  distance between two Cisco Nexus 5000 Series switches is 3000m for FCoE  (lossless Ethernet) traffic with NX-OS 5.0(2)N1.1. It can be enabled using the  pause no-drop buffer-size buffer-size pause-thresholdxoff-size resume-threshold xon-size command in QOS configuration. Here are the maximum  distances:

Nexus 55xx to Nexus 55xx - 3km

Nexus 50x0 to Nexus 50x0 - 3km

Nexus 50x0 to Nexus 55xx - 3km

Nexus 55xx to Nexus 2232 - 300m

Nexus 50x0 to Nexus 2232 - 300m

More on http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/qos/502_n2_1m/Cisco_Nexus_5000_Series_NX-OS_Quality_of_Service_Configuration_Guide_Rel_502_N1_1_chapter3.html#task_81ABFBE86A57475DA65966D5C9BC24A1

3. Can you explain the vlan trunking channels a little bit more?

We support Virtual LAN on top of the Fibre channel Over ethernetwork. So  trunking means we carry different Virtual LAN altogether in one ISL. And the  same on FCOE we also support Virtual SANs and can carry multiple virtual SANs on  one isl. In turns of multi-hop FCOE we will see that we could carry multiple  virtual LAN and virtual SANs together.

4. Can FCoE be used with UCS?

Yes. In latest UCS 1.4 release, you can directly connect FCoE targets to the  Fabric Interconnect ports. Also on UCS system we have FIP cabable (Gen2) CNAs  and Pre-FIP (Gen1) CNAs running on the blades that utilizes FCoE between blades  and Fabric Interconnect. If you would like to extend Fabric Interconnect  connectivity with VE ports and connecting them to Nexus 5K switches in terms of  running multi-hop FCoE, we don't support this on the UCS software yet.

5. What is the next data rate after 10Gig for FC? is it 16Gig? When will it  be supported?

After the 10Gig model the discussion are in 12Gig and 16Gig. If you are  asking Cisco will be supporting 16gig FC, yes we will be supporting it and we  will have products that will be supporting 16gig FC.

6. Is there a limit on the number of hops that can be made in a multihop  environment/design?

The limit is 7 actually. You can have upto 7 FCoE hops that are tested and  supported by Cisco. If there is any requirement to suppoprt more than 7 hops,  that can be tested by Cisco. Please contact your Cisco Account Manager if you  require more than 7 hops.

7. Can we configure multiple COS values for PFC ?

Yes, if you are asking you can use different COS values for PFC. PFC uses the  IEEE 802.1p CoS values in the IEEE  802.1Q VLAN tag to differentiate up to eight  CoSs that can be subject to  flow control independently. So on Nexus 5K,by  default we have 8 lanes , 8 qos-groups. Qos 6 and Qos 7 are reserved for control  plane trafic and the rest 6 qos-groups can be assigned to drop-class or  no-drop-class. By default FCoE is assigned COS 3.

8. What if we want to trunk multiple data vlans to a service thats running  ESXi for example do we still need to configure trunking on switch ports?

Yes if you want to carry multiple data VLANs on ESXi , we definitely need to  configure trunking. Also another reason if you want to use FCoE, Fiber  Channel over Ethernet Initialization Protocol (FIP) uses the native VLAN and  therefore all FCoE links should be trunked to carry the FCoE VLAN as well as the  native VLAN.

9. What is the difference between CEE and a DCE?

CEE is the Converged Enhanced Ethernet. The CEE authors group was an informal  group of companies that submitted the initial proposals of PFC , ETS and DCBX to  the IEEE DCB working group.DCE is a Cisco term that describes the architectural  collection of ethernet extensions based on open standards. DCE includes support  for the ethernet extensions found in EEE DCB.

10. How is the FCOE license activated on the Nexus 5000 or 5500 series?

There is an option in the Nexus 5000 called storage service license which is  activated when it is shipped from Cisco. When ordered as a seperate line card  item an activation key and the procedure for activating the code is delivered in  an envelope. Because this is always present and only activation is needed and  any Nexus 5k will automatically initiate the 120 days temporary license for a  storage protcol service from the first use of FC or FCoE command.

11. Is FCOE IP Routable?

No, there is no Ip layer in FCoE. FC-BB-5 contains FCIP that is  "IP-routable". FCoE is FC-routable. FCoE switches may forward FC frames across  different Ethernet Clouds, FCoE switches may forward FC frames over the Internet  using FCIP.

12. Do I need to enable Jumbo frames for FCOE?

Yes. You need to enable jumbo frames because the FC payload is 2,112  bytes  and enabling jumbo frames will keep FCoE traffic from being fragmented. On Nexus  5K, we have a pre-defined Class of Service (CoS) to be FCoE, which is by default  has the value 3. Within this CoS, the MTU size it set to 2158.

13. Can you advise on any resources for FCOE where I can read more about  them?

You can refer the book “I/O  Consolidation in the Data Center” by Silvano Gai, Claudio DeSanti

You can also visit the following resources:

Unified  Fabric

Fibre Channel over Ethernet (FCoE)

IEEE 802.1 Data  Center Bridging – White Papers

http://www.fcoe.com & FC-BB-5 Standard http://www.fcoe.com/09-056v5.pdf

Nexus  5000 Troubleshooting Guide

14. Can I use multiple classes of service for FCoE?

Yes you can use multiple classes of service for FCoE. We have 8 lanes, 8  QOS-groups. 6 and 7 are reserved for control plane traffic that is used by the  system.  That leaves us 6 qos-group to use. One of the 6 qos-groups is  pre-defined as drop class and it is class-default(drop class) and that leaves us  5 qos-groups. In the remaining 5 qos-groups, only 4 can be configured as  "no-drop" and one of these could be FCoE.

15. Did Cisco produce interoperability testing results with various vendors  (SAN, CNA adapters) ?

Yes we have the interoperability matrix which we published on CCO  Cisco.com.

http://www.cisco.com/en/US/partner/docs/switches/datacenter/mds9000/interoperability/matrix/Matrix7.html

We try to certify, different Storage, different CNAs, different O/S and  different NX-OS versions. Also most of the OSM vendors publish their  interoperability matrixes in their own web sites.

16. Can the concept and purpose of  Worldwide name can be briefy  explained?

A World Wide Name (WWN) is a unique identifier  which identifies a particular Fibre Channel target, Each WWN is an 8 byte number  derived from an IEEE OUI and vendor-supplied information.

  • Original format: addresses are assigned to manufacturers by the IEEE   standards committee, and are built into the device at build time,  similar to an  Ethernet MAC Address.  The first 2 bytes are either hex 10:00 or 2x:xx (where  the x's are  vendor-specified) followed by the 3-byte vendor identifier and 3  bytes  for a vendor-specified serial number

17.Which CNA types are supported on Nexus 5000 and Nexus 2232PP?

We have two flavours of Nexus 5K switches.They are Nexus 55xx and Nexus  5010/5020. On Nexus 5010/5020 we support Gen1 and Gen2 CNAs.

On Nexus 55xx and Nexus 2232PP we only support Gen2 CNAs such as  QLE814x/QLE815x, Emulex One Connect...


18. What are the requirements of FCOE?

You will need a lossless ethernet network that will support FCoE, to build  this network minimum requirement is to have a Fibre Channel Forwarder (FCF  switch) , host/server that has CNA installed and a FC or FCoE target that is  providing the access to the block storage for the server.

19. Can you explain a little background about FC and ISCSI, and the  differences between FCoE and ISCSI? I've never had the need for FC, so it's a  different world with "credits" for me.

FC and iSCSI share the same SCSI stack (to some extent), there are some  differences in the protocol stacks. FC is generaly used for tier-1 apps so will  FCoE by extension. iSCSI however is also seeing attraction in that area.

20. I would like to know the difference nowdays between ISCSI and FCoE as I  have never had the need for FC in the old days.

The iSCSI protocol can be implemented in networks that are  subject to   packet loss, and iSCSI can run over 1 Gigabit Ethernet (1GbE). FCoE   requires  10 Gigabit Ethernet (10GbE) and a lossless network with  infrastructure   components that properly implement pause frame requests  and per-priority pause   flow control (PFC) based on separate traffic  classes that map to different   priorities. The idea behind PFC is that  during periods of congestion,   high-priority traffic is allowed to  continue while lower-priority traffic is   paused.
   
    Your 10GbE switches will need DCB support for FCoE,  including a   range of enhancements for classes of service, congestion control,  and   management. FCoE also requires Jumbo Frames because the FC payload is   2,112  bytes and cannot be broken up; iSCSI does not require Jumbo  Frames.

21. Do you know the difference for the ethernet header between ISCSI and  FCoE ? Could this be an advantage for one of the protocols?


iSCSI has a TCP overhead contributing to both the size of frame and also  operational inefficiencies that come with TCP retransmissions. There maybe  benefits to iSCSI in an ethernet loss-less environment as these retransmissions  are eliminated.

image-FAQ.jpg

22. Does FCoE (in general) support by all SAN vendors (for their own  proprietary modes), or do their gears need to run in interop mode?
It's important to remember that FCoE *is*  FC. It uses the exact same protocol services as FC. Once a FCoE switch  decapsulates the FC from the Ethernet Frame, it is FC, and is handled by SAN  vendors as a FC frame. See the UNH Plugfest information at http://fibrechannel.org for  interoperability tests.

23. Does Cisco have any multi-hop FCoE products?


Cisco currently has multi-hop FCoE in the Nexus 5000 series of switches.  Additionally, the Nexus 7000 and MDS 9500 series Director switches have  announced multihop capability; watch for further information very soon.

24. Is CEE still active ?


I hope I understand your question properly. The DCB standards are built  on what CEE defined originally. Cisco switches abide by both standards.

25. What is the typical uplink oversubscription on Fiber Channel switches ?  If the oversubscription cannot be avoided, does it mean FCoE must be deployed  with DCB together to avoid FC packets being dropped?

DCB is an absolute requirement for FCoE. The links *need* to be loss-less  before it can even be considered for FC traffic as storage applications are  generally intolerant of loss. Regarding O/S, every organisation is different and  a value is chosen based on their tolerances. We have seen 4/1 over subscription  to up to 15/1. When designing the storage network the link O/S should be  considered as well as the internal switch hardware O/S however the latter is  generally lower so not a bottleneck but needs to be considered. This O/S is  platform specific and changes in accordance to the HW.

26. What is the aggregate encap/decap rate of FCF inside the switch, like  Nexus 5K?

I assume the question is if the process of de-encap/re-encap FC-to-FCoE  causes a performance hit, and that is not the case since packets are parsed and  modified as they are transmitted to xbar on their way to egress regardless of  protocol type.That all happens at speeds much faster then any input device can  forward frames in,or receive frames out.

27. Does the default QoS setting imply I/O consolidation would apply only to  servers running FCoE or below? how does it relate to the trend of server  consolidation, where I/O rate will get higher and higher as more VMs are  created?

The QoS setting (negotiated) sets up the adjacent neighbors so that they can  have a loss-less ethernet conversation. The packets are tagged with a CoS which  indicates the Virtual Lane they ough to be in. In your case where I/O rates are  increasing due to VM adoption, if the host is tagging these packets as loss-less  with the correct CoS marking, then the frames are transmitted with those  capabilities and the switch treats them as such. However you only ever have the  10G link offered. This QoS setting doesn't affect the increasing I/O rates.

28. Based on slide #51. "Trunking is not required on the host driver - all  FCoE frames are tagged by the CNA." what about if we want to trunk multiple data  VLANs to the servers running ESXi for example? do we still need to configure  trunking on the switch ports?

Yes, we  need to configure "switchport trunk" since Fiber Channel over  Ethernet Initialization Protocol (FIP) uses the native VLAN and therefore all  FCoE links should be trunked to carry the FCoE VLAN as well as the native  VLAN.

29. Can you explain what you meant by multihop extending beyond the  access?


Up until now, you could connect FCoE from the server (using CNA) to an  access switch. With Multihop FCoE as defined by FC-BB-5 working group of T11 we  can extend FCoE beyound the access switch (via an ISL) to other switches and  perhaps into the SAN core. You can now build Ethernet based SAN rather than a FC  only SAN.

30. Is it possible carry FCoE traffic across fabricpath cloud?

At this point in time, no. Currently ethernet traffic and FCoE is handled on  their own respective VLANs, which means that you can apply FabricPath to the  Ethernet VLAN. FCoE uses FSPF for its forwarding mechanisms.

31. Do I understand correctly ISL, so DOT1Q trunks are a no go?

It is 802.1q , please have a look at the frame formats for FCoE and FIP in  the FC-BB-5 standard page 126.

http://www.t11.org/ftp/t11/pub/fc/bb-5/09-056v5.pdf

32. How would you connect a 3rd party blade switch (FCoE enabled) to Cisco  Nexus infrastructure? Do i need to do multi-hop FCoE design?

At this moment, there is no interoperability with the third party Blade  switches.

33. Can you elaborate a little on how zoning is implemented in the N5K  please? I understand that zoning cannot be configured directly on the N5K but  has to be downloaded from a SAN switch, is this undertanding correct?

You can zone on the N5K. It's per VSAN. Zoning is distributed to all the  switches connected in that VSAN. Zoning is a security feature which limits the  ports which are participating in the SAN. If the Nexus 5K switch is running in  NPV mode, the zoning must be distibuted from the core NPIV enabled switch.

34. Referring to slide 54, if I understand correctly, you can still have  more than one link aggregated into single logical link for both FCoE and LAN...  just not through the same logical link?

That's correct VE_ports can be configured between two switches using a single  port-channel or multiple individidual links. To keep the SAN A and SAN B  isolation, we have to carry FCoE and LAN traffic with different  port-channels.

35. Is Nexus 2K supporting FCoE?

There are many flavors of Nexus 2Ks (FEX). Only Nexus 2232PP supports FCoE on  10GE host ports with NX-OS 4.2(1)N1 and higher releases.

36. Is there is any training or course certification leading to FCoE   Knowledge ?


As of now we dont have any certification for FCoE with Cisco or any other  vendor. These certifications will be launched very soon.

2. Related Information


Live Webcast Video

https://supportforums.cisco.com/videos/2023

Unified Fabric
http://www.cisco.com/en/US/netsol/ns945/index.html


Fibre Channel over Ethernet (FCoE)

http://www.cisco.com/go/fcoe


IEEE 802.1 Data Center Bridging – White Papers

http://www.cisco.com/en/US/netsol/ns783/index.html
http://www.fcoe.com

FC-BB-5 Standard

http://www.fcoe.com/09-056v5.pdf


“I/O Consolidation in the Data Center” by Silvano Gai, Claudio DeSanti
http://www.ciscopress.com/bookstore/product.asp?isbn=158705888X


Nexus 5000 Troubleshooting Guide
http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/N5K_Troubleshooting_Guide.html

Average Rating: 4.5 (2 ratings)

Comments

ELENA Mon, 08/20/2012 - 07:19

With buffer to buffer credit increase for Nexus 5500 from 64 to 240, will maximum distance increase from 3km to any higher number?

Which exactly?

Actions

Login or Register to take actions

This Document

Posted March 28, 2011 at 9:03 AM
Stats:
Comments:1 Avg. Rating:4.5
Views:27993 Contributors:1
Shares:0

Related Content