03-13-2012 03:20 AM
I have a fully integrated network in 2 separate data centers built solely on Nexus 5500UP switches that I just got finished designing and implementing. It connects FCoE to all hosts, FCoE from server access 5596UP to a pair of 5596 that act as "SAN" swtiches where I have native FC SFP that connect to XIV storage. In that environement all works great, multi-hop FCoE, NPIV, etc. All zoning and SAN administration is done fully on the N5K 5500 platform. (I even have MDS 9222i sole used for FCIP tunneling on the A and B fabric and that works like a champ too, DCNM sees both data centers as one large A and B fabric with the FCIP tunnels linking them)
The issue is when I introduced a pair of blade centers with N4K switches and a pair of 5548UP switches as their upstream FCF. The documentation isn't rocket science and yet I can't get a FLOGI login. The N7K sees the server MAC in the FIP snooping, it also sees the 5548 as it's FCF. But the 5548 vfc is stuck in "initializing". QoS is there, NXOS is upgraded, etc...I'm stumped and annoyed at this point. DCNM sees the 5548 as part of the fabric and it is a subordinated switch in the fcdomain and once I brought the switch up, all zoning, aliasing, etc was received on the switch...so I know the 5548 can see the fabric. Below are some configs (I cut out what isn't relevant)
5548UP:
Software
BIOS: version 3.5.0
loader: version N/A
kickstart: version 5.1(3)N1(1)
system: version 5.1(3)N1(1)
!
feature npiv
feature telnet
no feature http-server
feature tacacs+
cfs eth distribute
feature interface-vlan
feature lacp
feature vpc
feature lldp
!
class-map type qos class-fcoe
class-map type queuing class-fcoe
match qos-group 1
class-map type queuing class-all-flood
match qos-group 2
class-map type queuing class-ip-multicast
match qos-group 2
class-map type network-qos class-fcoe
match qos-group 1
class-map type network-qos class-all-flood
match qos-group 2
class-map type network-qos class-ip-multicast
match qos-group 2
system qos
service-policy type qos input fcoe-default-in-policy
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type network-qos fcoe-default-nq-policy
!
interface port-channel10
description 20Gbps Trunk to CLT-BC1-4001-PD-01 port e1/15 and 16
switchport mode trunk
switchport trunk allowed vlan 310
spanning-tree port type network
!
interface vfc141
bind mac-address 00:c0:dd:20:50:41
switchport description CLT-BC1 Blade #1
no shutdown
!
vsan database
vsan 310 interface vfc55
vsan 310 interface vfc141
!
interface Ethernet1/7
description Trunk to CLT-BC1-4001-PD-01 port e1/15
switchport mode trunk
switchport trunk allowed vlan 310
channel-group 10 mode active
interface Ethernet1/8
description Trunk to CLT-BC1-4001-PD-01 port e1/16
switchport mode trunk
switchport trunk allowed vlan 310
channel-group 10 mode active
!
CLT-SAC-5548-PD-05# show int vfc 141
vfc141 is trunking
Bound MAC is 00:c0:dd:20:50:41
Port description is CLT-BC1 Blade #1
Hardware is Ethernet
Port WWN is 20:8c:54:7f:ee:5a:d2:bf
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 310
Trunk vsans (admin allowed and active) (310)
Trunk vsans (up) ()
Trunk vsans (isolated) ()
Trunk vsans (initializing) (310)
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 frames input, 0 bytes
0 discards, 0 errors
0 frames output, 0 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Mon Mar 12 21:26:47 2012
!
CLT-SAC-5548-PD-05# show flogi database
No flogi sessions found.
CLT-SAC-5548-PD-05#show fcdomain
VSAN 310
The local switch is a Subordinated Switch.
Local switch run time information:
State: Stable
Local switch WWN: 21:36:54:7f:ee:5a:d2:81
Running fabric name: 21:36:54:7f:ee:1c:d0:81
Running priority: 128
Current domain ID: 0xec(236)
Local switch configuration information:
State: Enabled
FCID persistence: Enabled
Auto-reconfiguration: Disabled
Contiguous-allocation: Disabled
Configured fabric name: 20:01:00:05:30:00:28:df
Optimize Mode: Disabled
Configured priority: 128
Configured domain ID: 0x00(0) (preferred)
Principal switch run time information:
Running priority: 2
Interface Role RCF-reject
------------------ ------------- ------------
vfc55 Upstream Disabled
----------------------------------------------------------------------------------------------------------------------------------------
Nexus 4000i switch
Software
BIOS: version 1.0.37
loader: version N/A
kickstart: version 4.1(2)E1(1h)
system: version 4.1(2)E1(1h)
!
feature telnet
feature tacacs+
feature interface-vlan
feature lacp
feature fip-snooping
!
policy-map type queuing policy-fcoe-bandwidth
class type queuing 1p7q4t-out-q-default
bandwidth percent 50
class type queuing 1p7q4t-out-pq1
bandwidth percent 7
class type queuing 1p7q4t-out-q2
bandwidth percent 7
class type queuing 1p7q4t-out-q3
bandwidth percent 7
class type queuing 1p7q4t-out-q4
bandwidth percent 7
class type queuing 1p7q4t-out-q5
bandwidth percent 7
class type queuing 1p7q4t-out-q6
bandwidth percent 7
class type queuing 1p7q4t-out-q7
bandwidth percent 7
class-map type network-qos class-fcoe
match cos 3
class-map type network-qos class-non-fcoe
match cos 0-2,4-7
policy-map type network-qos policy-fcoe
class type network-qos class-fcoe
pause no-drop
mtu 2500
class type network-qos class-non-fcoe
system qos
service-policy type network-qos policy-fcoe
service-policy type queuing output policy-fcoe-bandwidth
!
vlan 310
fip-snooping enable
!
interface port-channel10
description 20Gbps Trunk to CLT-SAC-5548-PD-05 port e1/7 and 8
switchport mode trunk
switchport trunk allowed vlan 310
fip-snooping port-mode fcf
spanning-tree port type network
speed 10000
!
interface Ethernet1/1 <----the server is in slot 1
switchport mode trunk
spanning-tree port type edge trunk
speed auto
!
interface Ethernet1/15
description Trunk to CLT-SAC-5548-PD-05 port e1/7
switchport mode trunk
switchport trunk allowed vlan 310
speed 10000
channel-group 10 mode active
interface Ethernet1/16
description Trunk to CLT-SAC-5548-PD-05 port e1/8
switchport mode trunk
switchport trunk allowed vlan 310
speed 10000
channel-group 10 mode active
!
CLT-BC1-4001-PD-01# show fip-snooping fcf
Legend:
-------------------------------------------------------------------------------
Interface VLAN Active FPMA/ FCMAP FCF-MAC Pri Switch WWN Fabric Name
Enodes SPMA
-------------------------------------------------------------------------------
por10 310 0 FPMA 0x0efc00 54:7f:ee:5a 128 21:36:54:7f: 21:36:54:7f:
:d2:8a ee:5a:d2:81 ee:1c:d0:81
!
CLT-BC1-4001-PD-01# show fip-snooping vlan-discovery
Legend:
-------------------------------------------------------------------------------
Interface VLAN FIP MAC
-------------------------------------------------------------------------------
Eth1/1 1 00:c0:dd:20:50:41
!
Any suggestions because I'm stumped!! I'm also attaching a PDF copy of my Visio that shows port by port mapping and configuration for this blade chassis and port-channel 10.
Thanks for the help.
Chris
Solved! Go to Solution.
03-13-2012 06:14 PM
Yes the VLAN discovery happens on the native VLAN which is VLAN 1 by default.
03-13-2012 03:22 AM
In the inital post where I put "
The N7K sees the server MAC in the FIP snooping, it also sees the 5548 as it's FCF. But the 5548 vfc is stuck in "initializing".
I meant to put N4K...not N7K...sorry about that.
03-13-2012 05:56 PM
Can you also add VLAN 1 to your allowed VLAN list on the port-channel between the Nexus 5548 and Nexus 4000?
03-13-2012 06:11 PM
I meant to update this earlier. Yes, adding VLAN 1 to the trunk between the 4001 and the 5548 fixed the issue. Apparently dcbx information is carried on VLAN 1. I wish the documentation included this information. Thanks for the reply!
03-13-2012 06:14 PM
Yes the VLAN discovery happens on the native VLAN which is VLAN 1 by default.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: