Cisco Support Community
Showing results for 
Search instead for 
Did you mean: 

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Nexus 1000V VEM Issues

I am having some problems with VSM/VEM connectivity after an upgrade that I'm hoping someone can help with.


I have a 2 ESXi host cluster that I am upgrading from vSphere 5.0 to 5.5u1, and upgrading a Nexus 1000V from SV2(2.1) to SV2(2.2).  I upgraded vCenter without issue (I'm using the vCSA), but when I attempted to upgrade ESXi-1 to 5.5u1 using VUM it complained that a VIB was incompatible.  After tracing this VIB to the 1000V VEM, I created an ESXi 5.5u1 installer package containing the SV2(2.2) VEM VIB for ESXi 5.5 and attempted to use VUM again but was still unsuccessful


I removed the VEM VIB from the vDS and the host and was able to upgrade the host to 5.5u1.  I tried to add it back to the vDS and was given the error below:


vDS operation failed on host esxi1, Received SOAP response fault from [<cs p:00007fa5d778d290,>]: invokeHostTransactionCall
Received SOAP response fault from [<cs p:1f3cee20, TCP:localhost:8307>]: invokeHostTransactionCall
An error occurred during host configuration. got (vim.fault.PlatformConfigFault) exception


I installed the VEM VIB manually at the CLI with 'esxcli software vib install -d /tmp/' and I'm able to add to to the vDS, but when I connect the uplinks and migrate the L3 Control VMKernel, I get the following error where it complains about the SPROM when the module comes online, then it eventually drops the VEM.


2014 Mar 29 15:34:54 n1kv %VEM_MGR-2-VEM_MGR_DETECTED: Host esxi1 detected as module 3
2014 Mar 29 15:34:54 n1kv %VDC_MGR-2-VDC_CRITICAL: vdc_mgr has hit a critical error: SPROM data is invalid. Please reprogram your SPROM!
2014 Mar 29 15:34:54 n1kv %VEM_MGR-2-MOD_ONLINE: Module 3 is online
2014 Mar 29 15:37:14 n1kv %VEM_MGR-2-VEM_MGR_REMOVE_NO_HB: Removing VEM 3 (heartbeats lost)
2014 Mar 29 15:37:19 n1kv %STP-2-SET_PORT_STATE_FAIL: Port state change req to PIXM failed, status = 0x41e80001 [failure] vdc 1, tree id 0, num ports 1, ports  state BLK, opcode MTS_OPC_PIXM_SET_MULT_CBL_VLAN_BM_FOR_MULT_PORTS, msg id (2274781), rr_token 0x22B5DD
2014 Mar 29 15:37:21 n1kv %VEM_MGR-2-MOD_OFFLINE: Module 3 is offline



I have tried gracefully removing ESXi-1 from the vDS and cluster, reformatting it with a fresh install of ESXi 5.5u1, but when I try to join it to the N1KV it throws the same error.

Everyone's tags (2)
New Member

I am seeing the same thing. 

I am seeing the same thing.  However I have hosts that were installed as 5.5U1 and upgraded from 5.1U1.   I only saw the modules disconnect when I added a 5.1 host to the 1KV.  It was not too long after the module came online that all the modules dropped.  I was only able to get them to come back up by adding the system-vlan command to the control port-profile (we are running in L2 mode).


Cisco Employee



The SET_PORT_STATE_FAIL message is usually thrown when there is a communication issue between the VSM and the VEM while the port-channel interface is being programmed. 

What is the uplink port profile configuration? 

Other hosts are using this uplink port profile successfully?

The upstream configuration on an affected and a working host is the same? (ie control VLAN allowed where necessary)

Per kpate's post, control VLAN needs to be a system VLAN on the uplink port profile.

The VDC SPROM message is a cosmetic defect




New Member

I determined the issue a few

I determined the issue a few weeks ago, the culprit was Jumbo Frames which I had correctely turned on end to end but I learned my NIC's didn't support it.  I went back to a host that was still on a previous version of the VEM and tried MTU 9000 pings succesfully, but when I added a -d to the command it failed, indicating I never really had Jumbo Frames working.  The new version of the VEM must be more intolerant of this misconfig than the old version.  I have since turned off Jumbo Frames everywhere and am successfully keeping the VEM's connected to the VSM's.

Thanks for the KB on the SPROM cosmetic issue, I had originally thought that was the problem but now I know it's not.



CreatePlease to create content