Having installed the N1Kv many times in the past in a layer 2 mode and never had any issues I thought it was time to build out an environment with the N1Kv in layer 3 mode. My environment is as followed - Cisco UCS - 2 x 6248 FIs, 2 x B200 M3(esxi 5.1 hosts), a pair of Nexus 5Ks(L3) upstream connected with a VPC to each FI. I have installed the VSM manually and deployed the VEMs to the hosts using VUM which is working fine, when I run show mod I can see the VEMs, also the SVS connection is active to vCentre. My VSM MGMT0 interface and esxi hosts(VMKs) are on the same vlan (25) and they can all ping each other. The VSM is connected to the standard vswitch0 on ESXi01 and no VMs reside on ESXi02. After creating the UPLINKS vethernet port-profile and the MANAGEMENT ethernet port profile I successfully migrated the vnics and management VMK on ESXi02. At this point the VEMs are still connected and everything is happy. Now here is the strange part, after migrating the vnics on ESXi01(hosting the VSM), the vmk(ESXi01 management) and the virtual machine network for the VSM to the N1kv I loose connectivity to the VSM. Also the VSM can no longer communicate with the ESXi hosts or vCentre. I have configured vlan 25 as the system vlan for the ethernet and vethernet port-profiles and added the capability L3control command to the vethernet port-profile. As a test I created a new vethernet port-profile(
VSM_MGMT) moved the VSM management(MGMT0) to another IP subnet on vlan 27. So now both vlans are trunked to the 5ks where the inter-vlan routing take place. This worked fine and the VSM was able to communicate with the ESXi hosts and vCentre.
So my question is if the VSM management interface (MGMT0) and ESXi vmks are on the same subnet/vlan should layer 3 mode work? in this case my conclusion is no. Has anybody else found this true????
port-profile type ethernet UPLINKS
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 23,25,27
channel-group auto mode on mac-pinning
no shutdown
system vlan 25,27
state enabled
port-profile type vethernet MANAGEMENT
capability l3control
vmware port-group
switchport mode access
switchport access vlan 25
pinning id 0
no shutdown
system vlan 25
state enabled
port-profile type vethernet VSM_MGMT
vmware port-group
switchport mode access
switchport access vlan 27
pinning id 0
no shutdown
system vlan 27
state enabled
