Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Hi all.

I have : nexus 5548UP (5.0(3)N2(1)) with nexus 4000  in Bladecenter-H (last code 4.1(2)E1(1g)).

The config is quite basic and compliant with all best-practices.

Everything went fine until we saw some strange things, we decided to reboot everything. One of the strange things was that my vfc has gone into TF (Trunking F port) at some point, which is totally unexpected. When everything worked it ran as a standard F port. Now trunk mode is "on" on the vfc, I don't know if it's normal or not.

Now I have a VFC which is stuck in "down (Initializing)" state.

N5K-01# show interface vfc 103

vfc103 is down  (Initializing)

    Bound MAC is 00:c0:dd:1b:82:c5

    Hardware is Virtual Fibre Channel

    Port WWN is 20:66:54:7f:ee:08:ec:bf

    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled

    Port vsan is 2

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      0 frames input, 0 bytes

        0 discards, 0 errors

      0 frames output, 0 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

The VFC is correctly bound to the right mac-address learnt via fip-snooping

N5K-01# show run int vfc 103

interface  vfc103

  bind mac-address 00:c0:dd:1b:82:c5

  no shutdown

N5K-01#

NX-BCH1-M7# show fip-snooping vlan-discovery

  Legend:

-------------------------------------------------------------------------------

      Interface      VLAN          FIP MAC

-------------------------------------------------------------------------------

      Eth1/3         1         00:c0:dd:1b:82:c5

NX-BCH1-M7#

I believe it's "down (Initializing)" because of FIP negociation not complete ?

Upstream FCF is seen from the 4000

NX-BCH1-M7# show fip-snooping fcf

   Legend:

-------------------------------------------------------------------------------

Interface VLAN Active FPMA/ FCMAP      FCF-MAC   Pri   Switch WWN  Fabric Name

               Enodes SPMA

-------------------------------------------------------------------------------

por2      600  0      FPMA 0x0efc00 54:7f:ee:08  128  10:00:00:05: 10:00:00:05:

                                    :ec:a7            1e:90:4a:0b  1e:90:4a:0b

por2      600  0      FPMA  0x0efc00 54:7f:ee:08  128  10:00:00:05: 10:00:00:05:

                                    :ec:a6            1e:90:4a:0b  1e:90:4a:0b

No more session or enode detected through the 4000 (it was working before)

NX-BCH1-M7# show fip-snooping enode

NX-BCH1-M7# show fip-snooping sessions

NX-BCH1-M7#

It seems that the driver on the host still sends FIP messages as I can read on the 4000

NX-BCH1-M7# show fip-snooping counters interface ethernet 1/3

  Legend:

-------------------------------------------------------------------------------

Interface Name    Counter  Desciption                                Value

-------------------------------------------------------------------------------

Ethernet1/3       Unicast packets received                          0

                  Multicast packets received                        9

                  Vlan discovery packets received                   9

                  Vlan discovery response packets received          0

                  flogi (fabric login) packets received             0

                  flogi lsacc (acknowledgment) packets received     0

                   flogi lsrjt (reject) packes received              0

                  fdisc packets received                            0

                  fdisc lsacc received                              0

                  fdisc lsrjt  received                              0

                  logo (logout) packets received                    0

                  logo lsacc received                               0

                  logo lsrjt  received                               0

                  Enode Keepalive received                          0

                  Enode Discovery received                          0

                  FCF Discovery Solicit  received                    0

                  FCF Keepalive received                            0

                  Clear Virtual Link received                       0

                  Clear Virtual Link generated                       0

                  Unicast packets transmitted                       0

                  Packets dropped due to invalid length             0

                  Packets dropped due to tlv parse error            0

                  Packets dropped due to unsupported tlv             0

                  Packets dropped due to fcmap mismatch             0

NX-BCH1-M7# show lldp traffic interface ethernet 1/3

LLDP traffic statistics:

    Total frames out: 44

    Total Entries aged: 0

    Total frames in: 43

    Total frames received in error: 0

    Total frames discarded: 0

    Total TLVs unrecognized: 0

NX-BCH1-M7#

My last idea is to configure a SPAN session on the 5000 and see with wireshark what is going on. Are there any extra commands that can help me before sniffing ? (show, debug...)

I suspect an issue on the host, where can I find the compatibility matrix for FCoE ?

1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Yes.. Starting in 5.0(2) releases for N5k with support for multi-hop FCOE, all vFC interfaces are trunks.

16 REPLIES
Silver

Re: FCoE - nexus 4000/5548UP - vfc down (Initializing)

Also I could notice that when I add a VFC to the VSAN I always face this :

N5K-01(config-vsan-db)# vsan 2 interface vfc 103

N5K-01(config-vsan-db)# 2011 Dec 15 10:47:22 N5K-01 %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 2%$ Interface vfc103 is down (Error disabled)

Silver

Re: FCoE - nexus 4000/5548UP - vfc down (Initializing)

So, I erased the configuration into the Nexus 5548, upgraded to NXOS 5.1.3 and it works.

Something to note, when I add vfc into the VSAN before fc uplinks, no error message.

For all vfc added AFTER the addition of the FC NP uplinks, the error-disabled message is raised. And then all vfc come up and are not stuck in "initializing" state.

Silver

Re: FCoE - nexus 4000/5548UP - vfc down (Initializing)

I can see FLOGIs (I use NPV mode) but my vfc interfaces are still shown as "trunking" mode, is it expected ?

N5K-01# show npv flogi-table

--------------------------------------------------------------------------------

SERVER                                                                  EXTERNAL

INTERFACE VSAN FCID             PORT NAME               NODE NAME       INTERFACE

--------------------------------------------------------------------------------

vfc103    2    0x010301 21:00:00:c0:dd:1b:82:c5 20:00:00:c0:dd:1b:82:c5 fc1/32

vfc104    2    0x010f02 21:00:00:c0:dd:1b:82:f1 20:00:00:c0:dd:1b:82:f1 fc1/31

vfc105    2    0x010f01 21:00:00:c0:dd:1b:80:41 20:00:00:c0:dd:1b:80:41 fc1/31

Total number of flogi = 3.

Port mode is still TF but I have never configured trunkingand administrative port config is F ?!?!

N5K-01# show interface vfc 103-105

vfc103 is trunking (Not all VSANs UP on the trunk)

    Bound MAC is 00:c0:dd:1b:82:c5

    Hardware is Ethernet

    Port WWN is 20:66:54:7f:ee:08:ec:bf

    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled

    Port mode is TF

    Port vsan is 2

    Trunk vsans (admin allowed and active) (1-2)

    Trunk vsans (up)                       (2)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (1)

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      0 frames input, 0 bytes

        0 discards, 0 errors

      0 frames output, 0 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

    Interface last changed at Thu Dec 15 15:01:35 2011

vfc104 is trunking (Not all VSANs UP on the trunk)

    Bound MAC is 00:c0:dd:1b:82:f1

    Hardware is Ethernet

    Port WWN is 20:67:54:7f:ee:08:ec:bf

    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled

    Port mode is TF

    Port vsan is 2

    Trunk vsans (admin allowed and active) (1-2)

    Trunk vsans (up)                       (2)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (1)

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      0 frames input, 0 bytes

        0 discards, 0 errors

      0 frames output, 0 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

    Interface last changed at Thu Dec 15 15:01:35 2011

vfc105 is trunking (Not all VSANs UP on the trunk)

    Bound MAC is 00:c0:dd:1b:80:41

    Hardware is Ethernet

    Port WWN is 20:68:54:7f:ee:08:ec:bf

    Admin port mode is F, trunk mode is on

    snmp link state traps are enabled

    Port mode is TF

    Port vsan is 2

    Trunk vsans (admin allowed and active) (1-2)

    Trunk vsans (up)                       (2)

    Trunk vsans (isolated)                 ()

    Trunk vsans (initializing)             (1)

    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec

      0 frames input, 0 bytes

        0 discards, 0 errors

      0 frames output, 0 bytes

        0 discards, 0 errors

    last clearing of "show interface" counters never

    Interface last changed at Thu Dec 15 15:01:35 2011

N5K-01# show interface brief

-------------------------------------------------------------------------------

Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port

                  Mode   Trunk                          Mode  Speed  Channel

                         Mode                                 (Gbps)

-------------------------------------------------------------------------------

fc1/31     2      NP     off     up               swl    NP      4    --

fc1/32     2      NP     off     up               swl    NP      4    --

[...] output omitted

-------------------------------------------------------------------------------

Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port

                  Mode   Trunk                          Mode  Speed  Channel

                         Mode                                 (Gbps)

-------------------------------------------------------------------------------

vfc103     2      F      on      trunking         --     TF      auto --

vfc104     2      F      on      trunking         --     TF      auto --

vfc105     2      F      on      trunking         --     TF      auto --

interface vfc103   bind mac-address 00:c0:dd:1b:82:c5

  no shutdown

interface vfc104

  bind mac-address 00:c0:dd:1b:82:f1

  no shutdown

interface vfc105

  bind mac-address 00:c0:dd:1b:80:41

  no shutdown

vsan database

  vsan 2 interface vfc103

  vsan 2 interface vfc104

  vsan 2 interface vfc105

  vsan 2 interface fc1/31

  vsan 2 interface fc1/32

feature npv

feature npiv

These VFC are bound to only 1 VSAN which is VSAN 2 :

N5K-01# show vsan membership

vsan 1 interfaces:

vsan 2 interfaces:

    fc1/31            fc1/32            vfc103            vfc104

    vfc105

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

Cisco Employee

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Yes.. Starting in 5.0(2) releases for N5k with support for multi-hop FCOE, all vFC interfaces are trunks.

Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Thank you for the info.

Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

I'm facing another issue.

Everything seems to be fine : vfc is up, I see the FCIDs in the output of the "show npv flogi-table" command, but when I make a disk scan on the server, the OS never sees the volume.

Is it possible to see if the N5500 is actually sending traffic to the upstream switch ? How ? Any debug / show about live traffic ?

Cisco Employee

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Do you have the FCoE QoS configured on the NPV Nexus5500? Here is an example of the config needed

system qos
  service-policy type qos input fcoe-default-in-policy
  service-policy type queuing input fcoe-default-in-policy
  service-policy type queuing output fcoe-default-out-policy
  service-policy type network-qos fcoe-default-nq-policy
Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

yes.

Everything seems to be fine on my side. As devices are logged into the fabric on the upstream brocade, I assume the FCoE part is ok end-to-end. Link is up on the CNA, LLDP is ok etc...

Cisco Employee

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Then the other two things I can think off which could be a problem are

1)Is the initiator and target wwns zoned together on the Brocade SAN switch

2)LUN masking on the target.

Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Yep. I asked the storage admin about these parts yesterday and he said it was ok.

I'm a n00b regarding troubleshooting SAN, I don't know if there are some tools like wireshark or debug / show to check live traffic (something like SPAN ?)

We use SANsurfer on the host side, the storage array is an HDS.

Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Just another question around Nexus 4000 :

when I issue the " show fip-snooping vlan-discovery " first I see the amc-addresses discovered, and once I created the vfc bound to it, the mac-address disappears from the output of this command.

I suppose the Nexus 4000 doesn't show the mac-address once it has fully "snooped" the FIP session, is it expected ?

Cisco Employee

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Correct.. Once host is past "vlan discovery" phase, you will not see the MAC address in

"show fip-snooping vlan-discovery"

Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

a strange behavior :

As I use NPV the nexus is logged into the brocade first. Then the servers in the blade-H are also logged into the fabric.

On the brocade, the fcping to the wwn of the FC uplink of the nexus is ok, but an fcping to the wwn of the servers isn't.

Cisco Employee

FCoE - nexus 4000/5548UP - vfc down (Initializing)

This ping should work if you have the device logged in and FCoE QoS configured on the 5548. You could use a sniffer or packet counts to troubleshoot this but I suggest that you open a TAC case and work with an engineer.

Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

yep, in fact the smartnet has been ordered but it still not fully in the end of the process.

Silver

FCoE - nexus 4000/5548UP - vfc down (Initializing)

Finally got the answer : there is a bug with VSAN 2. If we change it to VSAN 3 no problem.

6115
Views
0
Helpful
16
Replies
CreatePlease login to create content