Nexus 5000 configuration for NetApp 1139A Universal Target Adapters

JUSTIN LOUCKS · ‎09-14-2010

I am having problems getting my interface to come up for this configuration. We have installed the UTA (FCoE target cards) in our NetApp heads and cabled them to our Nexus 5Ks. I have the Nexus interfaces configured as follows.

vlan 1010
fcoe vsan 10
name Storage_Mapped_VSAN10

interface e1/15
description NetApp-3140-01-3A
switchport mode trunk
switchport trunk native vlan 1010
switchport trunk allowed vlan 1010
spanning-tree port type edge trunk

interface vfc115
bind interface e1/15
no shut

vsan database
vsan 10 interface vfc115

There is nowhere in the NetApp device to specify the VLAN information for them so I tried doing it as an access port and as a trunk port making the VLAN on the Nexus the native VLAN and only allowed VLAN but it still does not work. It shows up on the NetApp side as Link Not Connected. I don't know if it really matters on the fact that it is running in a NetApp or not so I'll include the QLogic part number I'm dealing with as well.

Fibre Channel Target Adapter 3a (Dual-channel, QLogic CNA 8112 (8152) rev. 2)

Any ideas would be appreciated.

Prashanth Krishnappa · ‎09-15-2010

Configuration should not cause the link to not come up. If the interfaces show up as not connected or down, I would first look into the
physical layer. After this is taken care of, I see you are not allowing your FCoE VLAN on the trunk. You will need to correct this.

kbhagat · ‎10-07-2010

I'm having the same exact issue -- what am i missing ?

did u get a solution?

JUSTIN LOUCKS · ‎10-08-2010

No. I have not gotten anywhere yet. I can see the Ethernet side as up in the NetApp but the FC interface is still down in the NetApp and the vFC interface is down in the Nexus. I got absolutely no where with the NetApp case I opened and gave up. I will probably start looking at it again next week. Let me know if you figure it out between now and then. I think there has to be a privileged command in the NetApp side to configure the FCoE VLAN or something.

You can see the CNA properties under privileged options in the NetApp which shows the Ethernet Link status as up.

netapp-3140-01*> cna show

Slot: 3

Description: Dual-channel, QLogic CNA 8112(8152) rev. 2

FC Adapter Name: 3a

MPI Firmware Rev: 1.38.0

PHY Firmware Rev: 1.6.0

FC Firmware Rev: 5.2.0

FC Port ID: 0x000000

FC Nodename: 50:0a:09:80:89:ca:c7:41 (500a098089cac741)

FC Portname: 50:0a:09:81:99:ca:c7:41 (500a098199cac741)

FC Status: LINK NOT CONNECTED

Link Speed: 10 GBit

Link Status: LINK UP

Slot: 3

Description: Dual-channel, QLogic CNA 8112(8152) rev. 2

FC Adapter Name: 3b

MPI Firmware Rev: 1.38.0

PHY Firmware Rev: 1.6.0

FC Firmware Rev: 5.2.0

FC Port ID: 0x000000

FC Nodename: 50:0a:09:80:89:ca:c7:41 (500a098089cac741)

FC Portname: 50:0a:09:82:99:ca:c7:41 (500a098299cac741)

FC Status: LINK NOT CONNECTED

Link Speed: 10 GBit

Link Status: LINK UP

kbhagat · ‎10-08-2010

I got mine working.

What data ontap are you using?

JUSTIN LOUCKS · ‎10-08-2010

ONTAP 7.3.4

How did you configure it to work?

kbhagat · ‎10-08-2010

I'll share my configs with u when I get home to night. I'm running ontap 8.01rc1. I had the same issue. All I did was clear the configs and reboot the heads and the links came up

Sent from my iPhone

kbhagat · ‎10-08-2010

Here are my Configs that worked

I have 2 N5020 with a 3140HA running ontap 8.0.1RC1

The CNA's are on e1/9 and e1/10 on each n5k switch.. I've only show the onse side but it pretty much the same for the other side only differecen is the vsan #. I've rebooted the heads a numbe of times with no issues. I can see the LUNS from MY UCS Blade servers.

---- MY Nexus 5020 -----

NK5-01# sh run
version 4.1(3)N2(1)

feature npiv
feature fcoe

vlan 1010
fcoe vsan 1000

vsan database
vsan 1000 name "Fabric-A"

interface vfc9
bind interface Ethernet1/9
no shutdown

interface vfc10
bind interface Ethernet1/10
no shutdown

vsan database
vsan 1000 interface vfc9
vsan 1000 interface vfc10
vsan 1000 interface fc2/1
vsan 1000 interface fc2/2

interface fc2/1
no shutdown

interface fc2/2
no shutdown

interface Ethernet1/9
switchport mode trunk
spanning-tree port type edge trunk

interface Ethernet1/10
switchport mode trunk
spanning-tree port type edge trunk

zone name vSphere vsan 1000
    member pwwn 50:0a:09:83:89:ab:8b:e5
    member pwwn 50:0a:09:83:99:ab:8b:e5
    member pwwn 20:00:00:25:b5:00:00:31

zoneset name UCS6100 vsan 1000
member vSphere

zoneset activate name UCS6100 vsan 1000

NK5-01# sh flogi database
--------------------------------------------------------------------------------
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME
--------------------------------------------------------------------------------
fc2/1            1000 0x050003 20:41:00:05:73:a9:8a:c0 23:e8:00:05:73:a9:8a:c1
fc2/1            1000 0x050004 20:00:00:25:b5:00:00:31 20:00:00:25:b5:01:01:8f
fc2/1            1000 0x050006 20:00:00:25:b5:00:00:5f 20:00:00:25:b5:01:01:6f
vfc9             1000 0x050001 50:0a:09:83:99:ab:8b:e5 50:0a:09:80:89:ab:8b:e5
vfc10            1000 0x050002 50:0a:09:83:89:ab:8b:e5 50:0a:09:80:89:ab:8b:e5

Total number of flogi = 5.

============== 3140 Head A with Ontap 8.0.1RC1 7-Mode

3140A*> rdfile /etc/rc

hostname 3140A
ifgrp create single c10g-a e3a e3b
vlan create c10g-a 11 26
ifconfig c10g-a-11 xxx.xxx.xxx.xxx netmask 255.255.255.0 partner c10g-b-11
ifconfig c10g-a-26 xxx.xxx.xxx.xxx netmask 255.255.255.0 partner c10g-b-26
route add default xxx.xxx.xxx.xxx 1
routed on
options dns.enable off
options nis.enable off
savecore

3140A*> sysconfig -v 3

        slot 3: Dual 10G Ethernet Controller CNA SFP+
                (Dual-port, QLogic CNA 8112(8152) rev. 2)
                e3a MAC Address:    00:c0:dd:15:da:b8 (auto-10g_sr-fd-up)
                e3b MAC Address:    00:c0:dd:15:da:ba (auto-10g_sr-fd-up)
                Device Type:        ISP8112
        slot 3: Fibre Channel Target Host Adapter 3a
                (Dual-channel, QLogic CNA 8112 (8152) rev. 2, )
                Firmware rev:     5.3.4
                Host Port Addr:   050001
                Cacheline size:   16
                SRAM parity:      Yes
                FC Nodename:      50:0a:09:80:89:ab:8b:e5 (500a098089ab8be5)
                FC Portname:      50:0a:09:83:99:ab:8b:e5 (500a098399ab8be5)
                Connection:       PTP, Fabric
                SFP Vendor Name: FINISAR CORP.
                SFP Vendor P/N:   FTLX8571D3BCL-QL
                SFP Vendor Rev:   A
                SFP Serial No.:   AH60DND
                SFP Connector:    LC
                SFP Capabilities: 10 Gbit/Sec
        slot 3: Fibre Channel Target Host Adapter 3b
                (Dual-channel, QLogic CNA 8112 (8152) rev. 2, )
                Firmware rev:     5.3.4
                Host Port Addr:   880001
                Cacheline size:   16
                SRAM parity:      Yes
                FC Nodename:      50:0a:09:80:89:ab:8b:e5 (500a098089ab8be5)
                FC Portname:      50:0a:09:84:99:ab:8b:e5 (500a098499ab8be5)
                Connection:       PTP, Fabric
                SFP Vendor Name: FINISAR CORP.
                SFP Vendor P/N:   FTLX8571D3BCL-QL
                SFP Vendor Rev:   A
                SFP Serial No.:   AH60DDE
                SFP Connector:    LC
                SFP Capabilities: 10 Gbit/Sec

3140A*> fcp show adapter

Slot:                    3a
Description:             Fibre Channel Target Adapter 3a (Dual-channel, QLogic CNA 8112 (8152) rev. 2)
Adapter Type:            Local
Status:                  ONLINE
FC Nodename:             50:0a:09:80:89:ab:8b:e5 (500a098089ab8be5)
FC Portname:             50:0a:09:83:99:ab:8b:e5 (500a098399ab8be5)
Standby:                 No

Slot:                    3b
Description:             Fibre Channel Target Adapter 3b (Dual-channel, QLogic CNA 8112 (8152) rev. 2)
Adapter Type:            Local
Status:                  ONLINE
FC Nodename:             50:0a:09:80:89:ab:8b:e5 (500a098089ab8be5)
FC Portname:             50:0a:09:84:99:ab:8b:e5 (500a098499ab8be5)
Standby:                 No

Hope this helps... Perhaps updaint you firmware might help.... I did not specifiy and vlans for the CNA for the FCOE on the netapp.

JUSTIN LOUCKS · ‎10-12-2010

I appreciate you putting this out here. I do have one question from the configs post above. In the Nexus 5k, you appear to use VLAN 1010 as your FCoE VLAN which is then mapped to VSAN 1000. I see where you have all of the configuration for VSAN 1000 done, but in the NetApp, you do not appear to have VLAN 1010 configured, only 11 and 26. If this is due to cleansing your configs before posting, I understand, but if not, please explain where that VLAN definition is on the NetApp side as I think that is what I'm missing.

I'm probably going to go back to NOW support and try again later this week. In ONTAP 8.0.1rc, the adapters support all functionality like a true CNA. You can even run iSCSI and CIFS/NFS traffic across them as they show up as actual Ethernet interfaces with the ifconfig options available. Unfortunately, in ONTAP 7.3.4 they are seen by the system but only for FCoE access which is fine except that the ability to configure the Ethernet side seems to be absent.

If I come up with a solution for mine, aside from another ONTAP upgrade, I will be sure to post it here...

davidhirt · ‎09-27-2011

Did you ever figure this out, how the fcoe vlan gets configured on the filer side?

JUSTIN LOUCKS · ‎09-27-2011

Negative. We essentially came to terms with the fact that the adapter is recognized in 7.3.4 but doesn't have enough functionality to use it. We will be upgrading the system soon and running a recent version of ONTAP 8 which has full support for the adapter and we hope to get it going then for iSCSI and FCoE.

kbhagat · ‎09-27-2011

That is correct. For full CNA support, you will need Ontap 8.0.x 7-mode. Ontap 7.3.4 or7.3.5 has limited CNA support. There is no FCoE vlan configuration on the Filer. All of that is done on the 5K as shown above.

I did a small blog a yar ago -- http://blog.bhagat.com/wordpress/2010/11/cisco-ucs-vmware-netapp-nexus-fcoe-part1-4/

michael.james · ‎05-16-2012

Actually, it looks like the difference here is that the working config by kbhagat has only 2 interfaces per FAS controller. In Justin's config he has 4 ports per FAS controller. We have the same issue. apparently you can not create a VPC accross two Nexus 5K switches with more than 1 port each *IF* and only *IF* you are using FCOE. See page 15 of the document below.

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572835-00_NX-OS_vPC_DG.pdf

Now, here is my question. Can you create a regular port channel accross the e1a and e2a ports (PC101) and a regular port channel accross the e1b and e2b ports (PC102). Then a VPC accross PC101 and PC102??

Would this work???