Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi All,

Having a performance issue with FCoE multihop from UCS.

The layout is as follows:

UCS 5108 Chassis --> 6248UP FI's --> Pair of N5k's (5548) --> EMC VNX 5300 2 port FCoE 10gig card x2

Everything is functional - i can fc ping from n5k fabrics to the EMC pwwn.

I can see the pathing in vmware on the vhbas all is up and active. The VMware mode is set to round robin and the EMC side ALUA 4.

I'm at a loss on how to troubleshoot this further.The SAN is connected to the fabric with LC fibre om3 + 10gb SR SFP's. UCS has a seperate pair of fcoe port channels from the 5k fabric down to the FI's (using twinax leads). Data is seperated on its OWN vpc. I essentially followed the cisco guide on how to do this:

http://www.cisco.com/en/US/products/ps9670/products_configuration_example09186a0080c13e92.shtml

The hosts are not boot from SAN for now, they are booted off USB flash storage on the blade.

The show flogi on the 5ks shows everything logged in and as i said before i can see the LUNs on the blades. When howerver i try to uplod an ISO file say 2.5gb in size to a datastore is taking roughly 2 and half hours.

I have the newest UC 2.1(2)a firmware installed. The N5ks have 6.x NX-OS. The blades have ESXi 5.1 with the newest enet and fnic drivers from cisco installed.

The default qos for n5k is configured.

Any ideas would be greatly appreciated!

-Nick

Everyone's tags (3)
1 ACCEPTED SOLUTION

Accepted Solutions
New Member

Re: EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Nick,

Interesting observation. We have a hsrp IP in catalyst 4500 as a router to the management network with ip redirects on for some reason. I disabled the ip redirects as well and currently I'm doing 30K iops with RR path dual fabric's enabled and no fnic errors on esxi's logs.

I'll have to do some additional testing to see weather this is a coincidence or a fix by enabling ip redirects again. But it does look promising.

As my i/o load is done with i/o generator (http://labs.vmware.com/flings/io-analyzer) management network should have no effect on non routed FCOE network...  It newer even occurred to my mind to check Ethernet network side.

Thanks Nick for a good advice.

regards

Mikko

22 REPLIES
Cisco Employee

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Nick,

How many QoS drop classes do you have on the UCS and 5k side (http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCuh72875)?

What happens if you copy the ISO or VMDK from one datastore to another from the ESXi host cli?  Is it still slow?

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Brian,

Ok, i uploaded a 300mb iso which talk around 8 mins. I then copied it from one datastore to the other with ESXi cli - it took less than 30 seconds to copy the file.

So something is def dropping / impacting the traffic to the SAN from UCS...

Whats the best way to check if qos is doing it?

Cisco Employee

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Nick,

Based on the results of copying from datastore to datastore which is purely FCoE, it seems that the storage path is ok  and the problem may likely be somewhere on the ethernet side between the Vsphere Client and ESXi mgmt vmkernel.  What kind of load balancing is configured on the vSwitch (for b-series blades it shouldn't be route based on ip hash)?  Have you tried copying the file using Winscp to the esxi host to rule out an issue with the vsphere client?

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

what i just noticed then brian - is copying directly to the blades USB storage (esxi) for a VIB file - i got full bandwidth and thats from a VM sitting on another host via 2k --> 5k --> FI --> Blade

so that tells me it aint the ethernet link to the vmware kernel - it points to the blades vhbas / fcoe uplinks from FI.

Cisco Employee

Hi All,

Hi All,

Looking for your help for setup FCOE network between VNX,nexus 5k and FI 6248.

i have below query-

1. where exactly i should use FCOE,between FI and VNX or N5k and vnx.

2. Which connectivity option should i use Ethernet 10G link on N5K and FI or FC Ports.

VIP Green

see e.g.

see e.g.

http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/116248-configure-fcoe-00.html

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/116188-configure-fcoe-00.html

Questions:

- Has your VNX FCoE and native FC interfaces, or only FCoE ? (I saw your discussion in https://supportforums.cisco.com/discussion/12983641/fcoe-module-part-number-vnx-5400); I would be conservative and go with classical FC ! 

- instead of multihop FCoE, you could also do classical FC from FI to N5k; connect the VNX either with FCoE or FC to N5k. Do zoning on N5k ! 

- this would avoid multhop FCoE, which works, but is challenging.

Cisco Employee

Hi Walter,

Hi Walter,

I have FC and 10g iscsi module on VNX,but i dont have FCOE module on VNX.

EMC send me below details and asking for which module i require for FCOE connectivity-

VIP Green

Hi Kshitij

Hi Kshitij

I would simply go with classical FC, not FCoE. 

UCS FI supports classical FC, N5k as well.

Why do you need iSCSI ? iSCSI and FC are block based storage ? do you have clients with Ethernet interfaces (and iSCSI clients) ?

Walter.

Cisco Employee

now we both have been

now we both have been frustrated and finally they ask me where exactly i want to deply FCOE on my nexus/MDS/FI.

as i know we can configure FCOE only on 10 G Ethernet ports on MDS/FI/Nexus and we require active twinax cables for fcoe connectivity between MDS/NExus/FI to VNX.

VIP Green

If your OS uses native FC, we

If your OS uses native FC, we actually run FCoE between blade and FI.

Between FI and Northbound we can run either FCoE (to a N5k),

or

disaggregate the converged traffic, running FC over a native FC link to a MDS or a N5k FC port, AND Ethernet / IP to N5k Ethernet ports

Storage would be the connected with classical FC to MDS or N5k native FC port.

FC Cabling between VNX and N5k would fibre, 8G SFP's.

Summary; FCoE is only used in the access layer, first hop. 

This is what most UCS customers requiring classical FC do today.

Cisco Employee

Thanks walter for your reply.

Thanks walter for your reply.

But still i am confused what ports should i use on FI to run FCOE between VNX and FI.

as per my understanding we can use only 10G ports on FI to connect VNX FCOE Module to run FCOE traffic,please do let me know if i can run FCOE on FC ports.

Please do let me know if my understanding is correct.so based on that i need to order 2 Ports 10G FCOE module for VNX and active twinax cables and other parts which i require for FCOE connectivity.

VIP Green

Clarification: is your VNX

Clarification: is your VNX directly connected to UCS FI, or N5k ?

And if it is directly connected to FI, why ?

You cannot run FCoE on a native FC port !

FI 62xx have so called universal ports: they can be configured to be Ethernet (IP and/or FCoE) or classical FC.

Yes, cheapest connectivity for 10G Ethernet (and FCoE) is passive or active twinax cables, although limited in length.

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Brian,

Ok the vcenter dvs is set to route based on originating virtual port id - i read to leave it like this i knew that ip hash is not supported.

I just tried with the winscp client - same issue get around 500kb/s see image.

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

On the ucs side I have only 2 enabled classes: Default Best Effort and FCOE.

(Attached picture).

What command can be used to get the same info out of nexus5500 switches? I think that support did check this out but to be sure I could double check.

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Nick,

I have had same kind of problems when I started testing our new FCOE fabric during summer. I have support tickets open to cisco and vmware but neather has yet managed to find solution.

Our environment is allmost identical to yours:

FCOE Connection from UCS fabric interconnect -> Nexus with 2x 10Gbit dual twinax /Fabric (portchannel)

Dedicated to FCOE, (Lan Ethenet has own connections to Nexus swithces with vpc)

FCOE connection from nexus to VNX5500 with FCOE 10GB fiber connector to VNX FCOE card.

Zoning done in the NEXUS only, UCS in end host mode.

vcenter 5.1

esxi5.1 build 1117900

Cisco UCS B200M3 blades, SAN boot Native FCOE (Whole path to storage) VNX5500 SAN (FCOE connectors)

Cisco UCS 6248 Fabric interconnects version 2.1(1d)

Cisco FC drivers fnic version 1.5.0.22-1oem.500.0.0.472560

SAN/LAN Swithces:

Cisco Nexus 5548 swithces (fc/fcoe/ethernet)

system: version 6.0(2)N1(1)

STORAGE HW:

EMC VNX5500 OE 05.32.000.5.201 (block) with FCOE cards

What we have noticed in test is that if you disable fabrics or path performance goes normal until you enable paths again or something else happens. but we cant isolate the problem to one path as it might work now and not work tomorrow...

Allso changing from MRU from RR in path policy, seems to help little.

If you want to test the SAN performance vmware i/o analyzer Vapp is handy for that. You can get nice graphs of i/o's and latencies with that.

esxi dmesg shows scsi errors/aborts max retries etc in fcnic.

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Mikko,

yep same setup here - i have VM_RR and ALUA 4 set on the VNX. If i disable paths on either side of the fabric makes no difference. I have TAC case im working on with Cisco.

We are lookin at qos - apparently it has to match perfectly end to end or you see this type of behaviour...

Whats your qos set to mikko?

do your vhba's have qos set on them under the service profile?

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

I suppose what we need is a sample end to end qos from - Service Profile, System QOS on UCS and N5k

can anyone help with this?

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi All,

I think i have finally resolved my issue. It was as Brian said something to do with the comms from the guest to the vmk for management on the blade.

Enter ICMP Redirects.

I did a sniff on the VM im using for inband management - watching a 3gb file take ages and ages - hunders of icmp redirect packets were coming to the VM. I proceeded to disable icmp redirects on the CORE upsteam switch which is a Nexus 7009:

conf t

int vlan 76 (in band mgmt / default routing)

no ip redirects

As soon as i did this the upload took less than a minute. Thanks Brian and Mikko for getting me thinking!

New Member

Re: EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Nick,

Interesting observation. We have a hsrp IP in catalyst 4500 as a router to the management network with ip redirects on for some reason. I disabled the ip redirects as well and currently I'm doing 30K iops with RR path dual fabric's enabled and no fnic errors on esxi's logs.

I'll have to do some additional testing to see weather this is a coincidence or a fix by enabling ip redirects again. But it does look promising.

As my i/o load is done with i/o generator (http://labs.vmware.com/flings/io-analyzer) management network should have no effect on non routed FCOE network...  It newer even occurred to my mind to check Ethernet network side.

Thanks Nick for a good advice.

regards

Mikko

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi Mikko,

This issue appear to rear its head again!

im the same as you, see errors in dmesg from the host. Ip redirects i believe was part of this issue from the nexus side however with ALUA 4 and vmware round robin - storage is constantly seeing i/o error.

New Member

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Hi,

For me everything still works fine. I even enabled the IP redirects back in the routers and still everything works fine when I'm testing with Vmware I/O analyzer workload. My environment is used only for a vmware vcloud director testing so the normal io load is next to nothing with few test vapp's that we deploy.

EMC VNX 5300 multihop FCoE to Cisco UCS FI / B series blades

Could be QoS in UCS, I ran into this when I first implemented FCoE in the lab earlier this year. Make sure packet drop is checked on the platinum class

Here is the blog post I did on this back in April - http://jeremywaldrop.wordpress.com/2013/04/19/cisco-ucs-multihop-fcoe-qos-gotcha/

This relates to this bug - http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCuh72875

2786
Views
5
Helpful
22
Replies
CreatePlease to create content