cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4178
Views
0
Helpful
8
Replies

Jumbo Frames size

Walter Dey
VIP Alumni
VIP Alumni

Can someone please clarify the secrets about Jumbo Frame Size.

Here are some findings

on UCS QoS, max is 9216 (eg. best effort QoS)

on a ethernet vnic, max is 9000

Setup UCS - N5K - Netapp for NFS, L2 no routing, Netapp vfiler interface 9000

vmkping with -d (don't fragment) shows 8972 max end to end (vswitch is setup with 9000, vmk interface as well)

On another UCS domain, which we believe is setup identically, we get a max of 8944. 

vmkping between 2 ESXi on the same UCS domain also shows 8944 (yes, its L2 switched on the FI, setup of ESXi are identical )

Ping from a 6500 to UCS max 8972, from same 6500 to Netapp 9000 ?

I know that essentially the source application determines the max size used ? any explanation of the algorithm ?

8 Replies 8

richbarb
Cisco Employee
Cisco Employee

Hello wdey,

The general rule is 9216 in the network devices (swicthes and routers) and 9000 in the server side, just because some operational systems doesn't support mtu greater than 9000. Another reason to leave the mtu bigger in the network is because the packet coming from the server side could be encapsulated by some protocol (VxLAN, NVGRE, Tunneling, etc.).

Why the ping doesn't work with 9000 packet size?

This packet size configured by the ping, is without the header, when putting the header (IP 20Bytes + ICMP 8 Bytes = 9000 - 28 = 8972)  in the packet the server interface can't forward it without fragmentation.

The application usually do not control the mtu, this is a S.O. function, if you are using UCS you can configure the MTU there or also overwrite by the S.O. installed (not recommended). Few very specific applications are different.

I'm not sure if I helped.

Thanks Richard

S.O. = ? OS ?

This explains the 8972, which I hoped was solving the issue;

But then we discovered in the DR site (exactly the same hardware / software) a max of 8944, (=8972-28);

Specially weird: even between 2 ESXi in the same UCS domain, we see 8944 ? (and it's 100% sure not leaving the UCS)

Could it be due to some VNtag overhead ?

Hello wdey,

Sorry about that, S.O or O.S, I mean operating system.

I can't identify where exactly is the problem, I don't believe the VN-tag could be a issue, VN_tag just increased 6 bytes in the header frame.

Which virtual switch are you using in your ESX enviroment?

OK, with ping the overhead was 20 (IP) plus 8 (ICMP) = 8972 bytes.

Does it mean one should set MTU on the vnic's in the SP to 9000 - 20 (TCP) - 20 (IP) = 8960 bytes ?

No, you can leave the vnic's mtu as 9000. The rest of your network you will set to 9216 (if cisco) including the UCS QoS configuration.

The mtu as 9216 in the network can handle any overhead you can take in your servers.

Thanks Richard

Just a feedback, in

Cisco UCS Manager GUI Configuration Guide, Release 2.2

First Published: December 11, 2013

On p 272, 284, 288,

This is not correct, the max MTU value can only be set to 9000 ! NOT 9216

Hi wdey,

I didn't said that UCS doesn't support 9216 in the vnic, it supports but the vnic configuration reflects the operating system compatibility.

The MTU as 9000 is just because most part of operating systems follow the 9000 in the configuration, the rest of the network you can leave as 9216.

VMware:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007654

Windows:

http://www.altaro.com/hyper-v/how-to-adjust-mtu-jumbo-frames-on-hyper-v-and-windows-server-2012/

Linux:

http://tacticalvim.wordpress.com/2013/03/07/setting-up-jumbo-frames-in-rhelcentos/

You agree that with UCSM the MTU for vnic can only be set to 9000 ! not 9216, as the documentation claims; in the LAN QoS section you can set it to 9216.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: