Solved: FCoE - nexus 4000/5548UP - vfc down (Initializing)

Surya ARBY · ‎12-14-2011

Hi all.

I have : nexus 5548UP (5.0(3)N2(1)) with nexus 4000 in Bladecenter-H (last code 4.1(2)E1(1g)).

The config is quite basic and compliant with all best-practices.

Everything went fine until we saw some strange things, we decided to reboot everything. One of the strange things was that my vfc has gone into TF (Trunking F port) at some point, which is totally unexpected. When everything worked it ran as a standard F port. Now trunk mode is "on" on the vfc, I don't know if it's normal or not.

Now I have a VFC which is stuck in "down (Initializing)" state.

N5K-01# show interface vfc 103

vfc103 is down (Initializing)

Bound MAC is 00:c0:dd:1b:82:c5

Hardware is Virtual Fibre Channel

Port WWN is 20:66:54:7f:ee:08:ec:bf

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port vsan is 2

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

0 frames input, 0 bytes

0 discards, 0 errors

0 frames output, 0 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

The VFC is correctly bound to the right mac-address learnt via fip-snooping

N5K-01# show run int vfc 103

interface vfc103

bind mac-address 00:c0:dd:1b:82:c5

no shutdown

N5K-01#

NX-BCH1-M7# show fip-snooping vlan-discovery

Legend:

-------------------------------------------------------------------------------

Interface VLAN FIP MAC

-------------------------------------------------------------------------------

Eth1/3 1 00:c0:dd:1b:82:c5

NX-BCH1-M7#

I believe it's "down (Initializing)" because of FIP negociation not complete ?

Upstream FCF is seen from the 4000

NX-BCH1-M7# show fip-snooping fcf

Legend:

-------------------------------------------------------------------------------

Interface VLAN Active FPMA/ FCMAP FCF-MAC Pri Switch WWN Fabric Name

Enodes SPMA

-------------------------------------------------------------------------------

por2 600 0 FPMA 0x0efc00 54:7f:ee:08 128 10:00:00:05: 10:00:00:05:

:ec:a7 1e:90:4a:0b 1e:90:4a:0b

por2 600 0 FPMA 0x0efc00 54:7f:ee:08 128 10:00:00:05: 10:00:00:05:

:ec:a6 1e:90:4a:0b 1e:90:4a:0b

No more session or enode detected through the 4000 (it was working before)

NX-BCH1-M7# show fip-snooping enode

NX-BCH1-M7# show fip-snooping sessions

NX-BCH1-M7#

It seems that the driver on the host still sends FIP messages as I can read on the 4000

NX-BCH1-M7# show fip-snooping counters interface ethernet 1/3

Legend:

-------------------------------------------------------------------------------

Interface Name Counter Desciption Value

-------------------------------------------------------------------------------

Ethernet1/3 Unicast packets received 0

Multicast packets received 9

Vlan discovery packets received 9

Vlan discovery response packets received 0

flogi (fabric login) packets received 0

flogi lsacc (acknowledgment) packets received 0

flogi lsrjt (reject) packes received 0

fdisc packets received 0

fdisc lsacc received 0

fdisc lsrjt received 0

logo (logout) packets received 0

logo lsacc received 0

logo lsrjt received 0

Enode Keepalive received 0

Enode Discovery received 0

FCF Discovery Solicit received 0

FCF Keepalive received 0

Clear Virtual Link received 0

Clear Virtual Link generated 0

Unicast packets transmitted 0

Packets dropped due to invalid length 0

Packets dropped due to tlv parse error 0

Packets dropped due to unsupported tlv 0

Packets dropped due to fcmap mismatch 0

NX-BCH1-M7# show lldp traffic interface ethernet 1/3

LLDP traffic statistics:

Total frames out: 44

Total Entries aged: 0

Total frames in: 43

Total frames received in error: 0

Total frames discarded: 0

Total TLVs unrecognized: 0

NX-BCH1-M7#

My last idea is to configure a SPAN session on the 5000 and see with wireshark what is going on. Are there any extra commands that can help me before sniffing ? (show, debug...)

I suspect an issue on the host, where can I find the compatibility matrix for FCoE ?

Prashanth Krishnappa · ‎12-27-2011

Yes.. Starting in 5.0(2) releases for N5k with support for multi-hop FCOE, all vFC interfaces are trunks.

View solution in original post

Surya ARBY · ‎12-15-2011

Also I could notice that when I add a VFC to the VSAN I always face this :

N5K-01(config-vsan-db)# vsan 2 interface vfc 103
N5K-01(config-vsan-db)# 2011 Dec 15 10:47:22 N5K-01 %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 2%$ Interface vfc103 is down (Error disabled)

Surya ARBY · ‎12-15-2011

So, I erased the configuration into the Nexus 5548, upgraded to NXOS 5.1.3 and it works.

Something to note, when I add vfc into the VSAN before fc uplinks, no error message.

For all vfc added AFTER the addition of the FC NP uplinks, the error-disabled message is raised. And then all vfc come up and are not stuck in "initializing" state.

Surya ARBY · ‎12-15-2011

I can see FLOGIs (I use NPV mode) but my vfc interfaces are still shown as "trunking" mode, is it expected ?

N5K-01# show npv flogi-table

--------------------------------------------------------------------------------

SERVER EXTERNAL

INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE

--------------------------------------------------------------------------------

vfc103 2 0x010301 21:00:00:c0:dd:1b:82:c5 20:00:00:c0:dd:1b:82:c5 fc1/32

vfc104 2 0x010f02 21:00:00:c0:dd:1b:82:f1 20:00:00:c0:dd:1b:82:f1 fc1/31

vfc105 2 0x010f01 21:00:00:c0:dd:1b:80:41 20:00:00:c0:dd:1b:80:41 fc1/31

Total number of flogi = 3.

Port mode is still TF but I have never configured trunkingand administrative port config is F ?!?!

N5K-01# show interface vfc 103-105

vfc103 is trunking (Not all VSANs UP on the trunk)

Bound MAC is 00:c0:dd:1b:82:c5

Hardware is Ethernet

Port WWN is 20:66:54:7f:ee:08:ec:bf

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port mode is TF

Port vsan is 2

Trunk vsans (admin allowed and active) (1-2)

Trunk vsans (up) (2)

Trunk vsans (isolated) ()

Trunk vsans (initializing) (1)

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

0 frames input, 0 bytes

0 discards, 0 errors

0 frames output, 0 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

Interface last changed at Thu Dec 15 15:01:35 2011

vfc104 is trunking (Not all VSANs UP on the trunk)

Bound MAC is 00:c0:dd:1b:82:f1

Hardware is Ethernet

Port WWN is 20:67:54:7f:ee:08:ec:bf

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port mode is TF

Port vsan is 2

Trunk vsans (admin allowed and active) (1-2)

Trunk vsans (up) (2)

Trunk vsans (isolated) ()

Trunk vsans (initializing) (1)

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

0 frames input, 0 bytes

0 discards, 0 errors

0 frames output, 0 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

Interface last changed at Thu Dec 15 15:01:35 2011

vfc105 is trunking (Not all VSANs UP on the trunk)

Bound MAC is 00:c0:dd:1b:80:41

Hardware is Ethernet

Port WWN is 20:68:54:7f:ee:08:ec:bf

Admin port mode is F, trunk mode is on

snmp link state traps are enabled

Port mode is TF

Port vsan is 2

Trunk vsans (admin allowed and active) (1-2)

Trunk vsans (up) (2)

Trunk vsans (isolated) ()

Trunk vsans (initializing) (1)

1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

0 frames input, 0 bytes

0 discards, 0 errors

0 frames output, 0 bytes

0 discards, 0 errors

last clearing of "show interface" counters never

Interface last changed at Thu Dec 15 15:01:35 2011

N5K-01# show interface brief

-------------------------------------------------------------------------------

Interface Vsan Admin Admin Status SFP Oper Oper Port

Mode Trunk Mode Speed Channel

Mode (Gbps)

-------------------------------------------------------------------------------

fc1/31 2 NP off up swl NP 4 --

fc1/32 2 NP off up swl NP 4 --

[...] output omitted

-------------------------------------------------------------------------------

Interface Vsan Admin Admin Status SFP Oper Oper Port

Mode Trunk Mode Speed Channel

Mode (Gbps)

-------------------------------------------------------------------------------

vfc103 2 F on trunking -- TF auto --

vfc104 2 F on trunking -- TF auto --

vfc105 2 F on trunking -- TF auto --

interface vfc103 bind mac-address 00:c0:dd:1b:82:c5

no shutdown

interface vfc104

bind mac-address 00:c0:dd:1b:82:f1

no shutdown

interface vfc105

bind mac-address 00:c0:dd:1b:80:41

no shutdown

vsan database

vsan 2 interface vfc103

vsan 2 interface vfc104

vsan 2 interface vfc105

vsan 2 interface fc1/31

vsan 2 interface fc1/32

feature npv

feature npiv

These VFC are bound to only 1 VSAN which is VSAN 2 :

N5K-01# show vsan membership

vsan 1 interfaces:

vsan 2 interfaces:

fc1/31 fc1/32 vfc103 vfc104

vfc105

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

Prashanth Krishnappa · ‎12-27-2011

Yes.. Starting in 5.0(2) releases for N5k with support for multi-hop FCOE, all vFC interfaces are trunks.

Surya ARBY · ‎12-27-2011

Thank you for the info.

Surya ARBY · ‎12-28-2011

I'm facing another issue.

Everything seems to be fine : vfc is up, I see the FCIDs in the output of the "show npv flogi-table" command, but when I make a disk scan on the server, the OS never sees the volume.

Is it possible to see if the N5500 is actually sending traffic to the upstream switch ? How ? Any debug / show about live traffic ?

Prashanth Krishnappa · ‎12-28-2011

Do you have the FCoE QoS configured on the NPV Nexus5500? Here is an example of the config needed

system qos
  service-policy type qos input fcoe-default-in-policy
  service-policy type queuing input fcoe-default-in-policy
  service-policy type queuing output fcoe-default-out-policy
  service-policy type network-qos fcoe-default-nq-policy

Surya ARBY · ‎12-29-2011

yes.

Everything seems to be fine on my side. As devices are logged into the fabric on the upstream brocade, I assume the FCoE part is ok end-to-end. Link is up on the CNA, LLDP is ok etc...

Prashanth Krishnappa · ‎12-29-2011

Then the other two things I can think off which could be a problem are

1)Is the initiator and target wwns zoned together on the Brocade SAN switch

2)LUN masking on the target.

Surya ARBY · ‎12-29-2011

Yep. I asked the storage admin about these parts yesterday and he said it was ok.

I'm a n00b regarding troubleshooting SAN, I don't know if there are some tools like wireshark or debug / show to check live traffic (something like SPAN ?)

We use SANsurfer on the host side, the storage array is an HDS.

Surya ARBY · ‎01-03-2012

Just another question around Nexus 4000 :

when I issue the " show fip-snooping vlan-discovery " first I see the amc-addresses discovered, and once I created the vfc bound to it, the mac-address disappears from the output of this command.

I suppose the Nexus 4000 doesn't show the mac-address once it has fully "snooped" the FIP session, is it expected ?

Prashanth Krishnappa · ‎01-03-2012

Correct.. Once host is past "vlan discovery" phase, you will not see the MAC address in

"show fip-snooping vlan-discovery"

Surya ARBY · ‎01-04-2012

a strange behavior :

As I use NPV the nexus is logged into the brocade first. Then the servers in the blade-H are also logged into the fabric.

On the brocade, the fcping to the wwn of the FC uplink of the nexus is ok, but an fcping to the wwn of the servers isn't.

Prashanth Krishnappa · ‎01-05-2012

This ping should work if you have the device logged in and FCoE QoS configured on the 5548. You could use a sniffer or packet counts to troubleshoot this but I suggest that you open a TAC case and work with an engineer.