Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Nexus 1000v: VEM ports in blocking state after reboot of ESX host

 

Introduction


This document describes a problem where all VEM ports of a Nexus 1000v switch move to blocking state after a proper reboot of ESX host. The external Virtual Supervisor Module (VSM) manages the Cisco Nexus 1000V Series switch VEMs. The Virtual Ethernet Module (VEM) executes inside the Cisco Nexus 1000V Series switch hypervisor.

 

Nexus 1000v VSM and VEM

Cisco Nexus 1000V has 2 parts:

Virtual supervisor module (VSM) - This is the control software of the Cisco Nexus 1000V distributed virtual switch. It runs on a virtual machine (VM) and is based on NX-OS software.
Virtual Ethernet module (VEM) - This is the part of Cisco Nexus 1000V that actually switches data traffic. It runs on a VMware ESX 4.0 host. Several VEMs are controlled by one VSM. All the VEMs that form a switch domain should be in the same virtual Data Center as defined by VMware VirtualCenter..

 

Problem


The system was properly shutdown in the order VSM-secondry, VSM-primary, the second ESX host and at last the first ESX host. After the system was brought back up it is seen that all VEM ports are blocked.

~ # vemcmd show port

  LTL   VSM Port  Admin Link  State  PC-LTL  SGID  Vem Port  Type
   18                UP   UP    F/B*      0          vmnic1
   49                UP   UP    FWD       0            vmk0
   50              DOWN   UP    BLK       0        os-vm2.eth0
   51              DOWN   UP    BLK       0        nexus1kv-vsm-2.eth2
   52              DOWN   UP    BLK       0        nexus1kv-vsm-2.eth1
   53              DOWN   UP    BLK       0        nexus1kv-vsm-2.eth0

* F/B: Port is BLOCKED on some of the vlans.
       One or more vlans are either not created or
       not in the list of allowed vlans for this port.

Please run "vemcmd show port vlans" to see the details.

~ # vemcmd show port vlans
                          Native  VLAN   Allowed
  LTL   VSM Port  Mode    VLAN    State* Vlans
   18              T          1   FWD    189,191
   49              A        189   FWD    189
   50              A          1   BLK    1
   51              A          1   BLK    1
   52              A          1   BLK    1
   53              A          1   BLK    1

 

Nexus 1000v Configuration

 

nexus1kv# show running-config

version 4.2(1)SV2(1.1a)
svs switch edition essential
========= snip ==================
ip access-list snmp-ro
  10 permit ip 10.10.10.0/24 any
vem 3
  host vmware id 34333535-3533-435a-4a37-323730325332
vem 4
  host vmware id 34333535-3533-435a-4a37-323730325334
========= snip ==================
vrf context management
  ip route 0.0.0.0/0 10.10.10.1
vlan 1,181,189,191
vlan 181
  name Clients
vlan 189
  name Mgmt
vlan 191
  name Control
port-channel load-balance ethernet source-virtual-port-id
port-profile default max-ports 32
port-profile type ethernet Unused_Or_Quarantine_Uplink
  vmware port-group
  shutdown
  description Port-group created for Nexus1000V internal usage. Do not use.
  state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
  vmware port-group
  shutdown
  description Port-group created for Nexus1000V internal usage. Do not use.
  state enabled
port-profile type ethernet Uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 181,187,191
  no shutdown
  system vlan 187,191
  state enabled
port-profile type vethernet VMkernel
  capability l3control
  vmware port-group
  switchport mode access
  switchport access vlan 187
  no shutdown
  system vlan 187
  state enabled
port-profile type vethernet VLAN181-Clients
  vmware port-group
  switchport mode access
  switchport access vlan 181
  no shutdown
  state enabled
port-profile type vethernet Control
  capability l3control
  vmware port-group
  switchport mode access
  switchport access vlan 191
  no shutdown
  system vlan 191
  state enabled
system storage-loss log time 30
vdc nexus1kv id 1
  limit-resource vlan minimum 16 maximum 2049
  limit-resource monitor-session minimum 0 maximum 2
  limit-resource vrf minimum 16 maximum 8192
  limit-resource port-channel minimum 0 maximum 768
  limit-resource u4route-mem minimum 1 maximum 1
  limit-resource u6route-mem minimum 1 maximum 1
interface mgmt0
  ip address 10.10.10.50/24
interface Vethernet1
  inherit port-profile VMkernel
  description VMware VMkernel, vmk0
  vmware dvport 32 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 001B.7830.52B4
interface Vethernet2
  inherit port-profile VMkernel
  description nexus1kv-vsm-1, Network Adapter 2
  vmware dvport 35 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 0050.5687.3AB1
interface Vethernet3
  inherit port-profile Control
  description nexus1kv-vsm-1, Network Adapter 1
  vmware dvport 160 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 0050.5687.7941
interface Vethernet4
  inherit port-profile Control
  description nexus1kv-vsm-1, Network Adapter 3
  vmware dvport 162 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 0050.5687.7391
interface Vethernet5
  inherit port-profile VLAN181-Clients
  description centos-vm2, Network Adapter 1
  vmware dvport 64 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 0050.5687.6A8C
interface Vethernet6
  inherit port-profile VLAN181-Clients
  description centos-vm3, Network Adapter 1
  vmware dvport 65 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 0050.5687.041A
interface Vethernet7
  inherit port-profile Control
  description nexus1kv-vsm-2, Network Adapter 3
  vmware dvport 163 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 0050.5687.0027
interface Vethernet8
  inherit port-profile VMkernel
  description nexus1kv-vsm-2, Network Adapter 2
  vmware dvport 34 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 0050.5687.6DF9
interface Vethernet9
  inherit port-profile Control
  description nexus1kv-vsm-2, Network Adapter 1
  vmware dvport 161 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 0050.5687.4380
interface Vethernet10
  inherit port-profile VMkernel
  description VMware VMkernel, vmk0
  vmware dvport 33 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
  vmware vm mac 001B.7830.128E
interface Ethernet3/2
  inherit port-profile Uplink
interface Ethernet4/2
  inherit port-profile Uplink
============= snip =================

 

Explanation

 

VEMs will retain their "last known" configuration pulled from a VSM until it's reboot.  This is expected behavior.  If you shut down both VSMs, the VEM will continue to operate fine, but if you then reboot the VEM, when it comes up, all interfaces that were not pre-configured as "system vlans" will stay down until the VEM inserts as a module to the VSM.  This is a security mechanism - VEMs must check in with the VSM for any configuration changes before bringing any ports up.

 

Resolution

 

Checked the logs and found the following message:

%VEM_MGR-SLOT11-1-VEM_SYSLOG_ALERT: sfswitchdata : L3 Control and System VLAN configurations not applied on vethernet port. VMware Port[50331670] DVPortGroup[dvportgroup-25]. L3 Control can be applied only on VMKernel port.

Originally was using the following port profile

port-profile type vethernet system
  capability l3control
  vmware port-group
  switchport mode access
  switchport access vlan 100
  no shutdown
  system vlan 100
  state enabled

Created dedicated port-profile without capability l3control specially for vcenter and vsm veths

port-profile type vethernet 100-agromov
  vmware port-group
  switchport mode access
  switchport access vlan 100
  no shutdown
  system vlan 100
  state enabled

No ports were blocked after applying this port profile.

This document is created from following discussion

https://supportforums.cisco.com/discussion/11808356/nexus1000v-vem-ports-are-blocking

 

Related Information


Nexus 1000v Switch - VSM and VEM Connectivity Issues
Nexus 1000V Recovery from Failed VEM

3394
Views
5
Helpful
0
Comments