EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

[nickeoannidis]

Hi All,

Having a performance issue with FCoE multihop from UCS.

The layout is as follows:

UCS 5108 Chassis --> 6248UP FI's --> Pair of N5k's (5548) --> EMC VNX 5300 2 port FCoE 10gig card x2

Everything is functional - i can fc ping from n5k fabrics to the EMC pwwn.

I can see the pathing in vmware on the vhbas all is up and active. The VMware mode is set to round robin and the EMC side ALUA 4.

I'm at a loss on how to troubleshoot this further.The SAN is connected to the fabric with LC fibre om3 + 10gb SR SFP's. UCS has a seperate pair of fcoe port channels from the 5k fabric down to the FI's (using twinax leads). Data is seperated on its OWN vpc. I essentially followed the cisco guide on how to do this:

http://www.cisco.com/en/US/products/ps9670/products_configuration_example09186a0080c13e92.shtml

The hosts are not boot from SAN for now, they are booted off USB flash storage on the blade.

The show flogi on the 5ks shows everything logged in and as i said before i can see the LUNs on the blades. When howerver i try to uplod an ISO file say 2.5gb in size to a datastore is taking roughly 2 and half hours.

I have the newest UC 2.1(2)a firmware installed. The N5ks have 6.x NX-OS. The blades have ESXi 5.1 with the newest enet and fnic drivers from cisco installed.

The default qos for n5k is configured.

Any ideas would be greatly appreciated!

-Nick

[Brian Morrissey]

Hi Nick,

How many QoS drop classes do you have on the UCS and 5k side (http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCuh72875)?

What happens if you copy the ISO or VMDK from one datastore to another from the ESXi host cli?  Is it still slow?

[nickeoannidis]

Hi Brian,

Ok, i uploaded a 300mb iso which talk around 8 mins. I then copied it from one datastore to the other with ESXi cli - it took less than 30 seconds to copy the file.

So something is def dropping / impacting the traffic to the SAN from UCS...

Whats the best way to check if qos is doing it?

[Brian Morrissey]

Hi Nick,

Based on the results of copying from datastore to datastore which is purely FCoE, it seems that the storage path is ok  and the problem may likely be somewhere on the ethernet side between the Vsphere Client and ESXi mgmt vmkernel.  What kind of load balancing is configured on the vSwitch (for b-series blades it shouldn't be route based on ip hash)?  Have you tried copying the file using Winscp to the esxi host to rule out an issue with the vsphere client?

[nickeoannidis]

what i just noticed then brian - is copying directly to the blades USB storage (esxi) for a VIB file - i got full bandwidth and thats from a VM sitting on another host via 2k --> 5k --> FI --> Blade

so that tells me it aint the ethernet link to the vmware kernel - it points to the blades vhbas / fcoe uplinks from FI.

[Kshitij Purwar]

Hi All,

Looking for your help for setup FCOE network between VNX,nexus 5k and FI 6248.

i have below query-

1. where exactly i should use FCOE,between FI and VNX or N5k and vnx.

2. Which connectivity option should i use Ethernet 10G link on N5K and FI or FC Ports.

[Walter Dey]

see e.g.

http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/116248-configure-fcoe-00.html

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/116188-configure-fcoe-00.html

Questions:

- Has your VNX FCoE and native FC interfaces, or only FCoE ? (I saw your discussion in https://supportforums.cisco.com/discussion/12983641/fcoe-module-part-number-vnx-5400); I would be conservative and go with classical FC ! 

- instead of multihop FCoE, you could also do classical FC from FI to N5k; connect the VNX either with FCoE or FC to N5k. Do zoning on N5k ! 

- this would avoid multhop FCoE, which works, but is challenging.

[Kshitij Purwar]

Hi Walter,

I have FC and 10g iscsi module on VNX,but i dont have FCOE module on VNX.

EMC send me below details and asking for which module i require for FCOE connectivity-

[Walter Dey]

Hi Kshitij

I would simply go with classical FC, not FCoE. 

UCS FI supports classical FC, N5k as well.

Why do you need iSCSI ? iSCSI and FC are block based storage ? do you have clients with Ethernet interfaces (and iSCSI clients) ?

Walter.

[Kshitij Purwar]

now we both have been frustrated and finally they ask me where exactly i want to deply FCOE on my nexus/MDS/FI.

as i know we can configure FCOE only on 10 G Ethernet ports on MDS/FI/Nexus and we require active twinax cables for fcoe connectivity between MDS/NExus/FI to VNX.

[Walter Dey]

If your OS uses native FC, we actually run FCoE between blade and FI.

Between FI and Northbound we can run either FCoE (to a N5k),

or

disaggregate the converged traffic, running FC over a native FC link to a MDS or a N5k FC port, AND Ethernet / IP to N5k Ethernet ports

Storage would be the connected with classical FC to MDS or N5k native FC port.

FC Cabling between VNX and N5k would fibre, 8G SFP's.

Summary; FCoE is only used in the access layer, first hop. 

This is what most UCS customers requiring classical FC do today.

[Kshitij Purwar]

Thanks walter for your reply.

But still i am confused what ports should i use on FI to run FCOE between VNX and FI.

as per my understanding we can use only 10G ports on FI to connect VNX FCOE Module to run FCOE traffic,please do let me know if i can run FCOE on FC ports.

Please do let me know if my understanding is correct.so based on that i need to order 2 Ports 10G FCOE module for VNX and active twinax cables and other parts which i require for FCOE connectivity.

[Walter Dey]

Clarification: is your VNX directly connected to UCS FI, or N5k ?

And if it is directly connected to FI, why ?

You cannot run FCoE on a native FC port !

FI 62xx have so called universal ports: they can be configured to be Ethernet (IP and/or FCoE) or classical FC.

Yes, cheapest connectivity for 10G Ethernet (and FCoE) is passive or active twinax cables, although limited in length.

[nickeoannidis]

Hi Brian,

Ok the vcenter dvs is set to route based on originating virtual port id - i read to leave it like this i knew that ip hash is not supported.

I just tried with the winscp client - same issue get around 500kb/s see image.

[mikko.suomi]

On the ucs side I have only 2 enabled classes: Default Best Effort and FCOE.

(Attached picture).

What command can be used to get the same info out of nexus5500 switches? I think that support did check this out but to be sure I could double check.