Nexus 1000V Jumbo Frames MTU 9000

Unanswered Question
Jan 13th, 2010

I have an ESX host with 4 NICs and I added all four NICs to the 1000V DVS. All of the four NICs are part of a single port-profile (system-uplink). After I did that, everything works fine except the datastores are now not accessible. The datastores is a NFS mount which uses MTU 9000 and is mounting on a vlan. Before moving to 1000V DVS, the 4 NICs were configured with MTU 9000 in the standard vSwitch0. According to the Cisco config guide, MTU can't be set on vethernet interfaces. But the four NICs (eth3/3 - 3/6) are configured with system-uplink port-profile and MTU can't be set neither. The config guide is here http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_0_4_s_v_1_2/interface/configuration/guide/n1000v_if_2basic.html#wp1221750

I'd like to know if anyone came across a similar situation. I'd hate to break the 4 NICs and move 1 or 2 NICs just to get it the MTU size correctly. I'm attaching part of the config for review. VLAN 142 is the storage vlan.

version 4.0(4)SV1(2)

vem 3
  host vmware id 35303738-3634-xxxxxxxxxxxxxxxxxxxx
vlan 1,142,221,223-224
vlan 101
  name control
vlan 102
  name packet
vlan 100
  name management
port-profile type vethernet Control
  vmware port-group
  switchport mode access
  switchport access vlan 101
  no shutdown
  state enabled
port-profile type vethernet Packet
  vmware port-group
  switchport mode access
  switchport access vlan 102
  no shutdown
  state enabled
port-profile type ethernet Unused_Or_Quarantine_Uplink
  description Port-group created for Nexus1000V internal usage. Do not use.
  vmware port-group
  shutdown
  state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
  description Port-group created for Nexus1000V internal usage. Do not use.
  vmware port-group
  shutdown
  state enabled
port-profile type vethernet data_vlan224
  vmware port-group
  switchport mode access
  switchport access vlan 224
  no shutdown
  state enabled
port-profile type ethernet data-uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 142,221,223-224
  channel-group auto mode on sub-group cdp
  no shutdown
  state enabled
port-profile type vethernet data_vlan223
  vmware port-group
  switchport mode access
  switchport access vlan 223
  no shutdown
  state enabled
port-profile type vethernet service_console
  vmware port-group
  switchport mode access
  switchport access vlan 221
  no shutdown
  state enabled
port-profile type vethernet storage
  vmware port-group
  switchport mode access
  switchport access vlan 142
  no shutdown
  state enabled
port-profile type ethernet system-uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 100-102,142,221,223-224
  channel-group auto mode on sub-group cdp
  no shutdown
  system vlan 100-102,142,221
  state enabled

interface port-channel1
  inherit port-profile system-uplink

interface port-channel2
  inherit port-profile system-uplink

interface Ethernet3/3
  inherit port-profile system-uplink

interface Ethernet3/4
  inherit port-profile system-uplink

interface Ethernet3/5
  inherit port-profile system-uplink

interface Ethernet3/6
  inherit port-profile system-uplink

interface Vethernet1
  inherit port-profile storage
  description VMware VMkernel, vmk1
  vmware dvport 160

interface Vethernet2
  inherit port-profile mgmt_network
  description VMware VMkernel, vmk0
  vmware dvport 352

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
jean-louis.auzepy Mon, 01/25/2010 - 05:05

Hello

did you try to set up the system MTU first?

Nexus1000v(config)# system jumbomtu 9000

Regards

Jlouis

felixjai Mon, 01/25/2010 - 07:30

Hi Jean-louis,

I believe we have resolved the issue. I found that the MTU setting needs to be set on the Port-channel instead of to the Ethernet interface.

I have another question. I have some Nexus 1000V instance that was installed for demo purpose, but I need to uninstall it from vCenter, but I keep getting error when object was still in used. I checked all hosts were removed and couldn't figure out what else needs to remove. Do you have a link for uninstalling? Thanks.

Felix Lai | Network & Security Architect | CCIE CCSP JNCIS-FWV JNCIA-SSL | Northeast | Presidio Networked Solutions

One Penn Plaza, Suite 1924, New York NY 10119 | [email protected]

D: 212.652.5717 | C: 917.805.1431 | F: 212.244.1685 www.presidio.com

jean-louis.auzepy Mon, 01/25/2010 - 07:39

Funny enough, i was trying to do exactly the same thing today.

After a lot of pain i managed to get rid of them using the troubleshooting guide:

You are logged in to the vSphere Client and the Nexus 1000V.


Step 1 From the vSphere Client, choose
Inventory

Networking
.


Step 2 Select the Nexus 1000V DVS and select the
Hosts
tab.


Step 3 Right-click each host, and choose
Remove from Distributed Virtual Switch.


Step 4 On the VSM, enter the
show vmware vc extension-key
command to verify the extension key.


Step 5 Delete the extension key that is present on vCenter Server using the Managed Object Browser (MOB) Unregister Extension API, as follows:


a. Go to the extension manager [https:///mob/?moid=ExtensionManager]


b. Click
Unregister Extension
[https:///mob/?moid=ExtensionManager&method=unregisterExtension]


c. Paste Cisco_Nexus_1000V_495736846 (your extension key attached to the DVS) and click
Invoke Method.


Step 6 From the Nexus 1000V Connection Configuration mode, remove the DVS from the vCenter Server as follows:.


n1000v# conf t
n1000v(config)# svs connection VC
n1000v(config-svs-conn)# no vmware dvs

If that doesn't help, may be you can post your error message

good luck

jlouis

felixjai Mon, 01/25/2010 - 07:44

thanks for the quick reply. What if the VSM is no longer there or was deleted before removing the SVS connection?

jean-louis.auzepy Mon, 01/25/2010 - 08:06

I might be wrong but i think your only way is to recreate a Nexus1K using the same extension key, register it again and unregister properly

I couldn't find any other way to remove it from the DVS. I m sure a lot of people will have the same problem and we should have the option on the next version(if we re lucky)

Sorry

Jlouis

ryan.lambert Mon, 04/19/2010 - 11:07

Just as a sidenote here (and a word of caution), there's currently an issue where the VEM modules will not retain the 9000 MTU setting upon reboot/restart, and will boot themselves @ 1500 MTU until they get their instructions from the VSM. This may cause some headaches for things like mounting network storage, etc... so just a variable to consider. In addition, it can be particularly ugly if your VSM storage is network-attached.

As I understand it, it is slated to be fixed in the next version of Nexus 1000v software later this month.

There are a lot of excellent posts out there regarding MTU, yet for weeks my colleagues and I continue to be confused about MTU on Nexus100v.

The confusions were around:

-How to set jumbo on N1kv

-Relationship between "system mtu" and "mtu"

-If "system mtu" is needed

-How to set "system mtu"

I wrote a short summary, based on multiple conversations with Cisco, and finally the definitive source. Hope it helps.

http://www.seanxwang.com/2010/07/nexus1000v-when-to-use-system-mtu.html

Actions

This Discussion

Related Content