Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

VIP Green

Jumbo Frames size

Can someone please clarify the secrets about Jumbo Frame Size.

Here are some findings

on UCS QoS, max is 9216 (eg. best effort QoS)

on a ethernet vnic, max is 9000

Setup UCS - N5K - Netapp for NFS, L2 no routing, Netapp vfiler interface 9000

vmkping with -d (don't fragment) shows 8972 max end to end (vswitch is setup with 9000, vmk interface as well)

On another UCS domain, which we believe is setup identically, we get a max of 8944. 

vmkping between 2 ESXi on the same UCS domain also shows 8944 (yes, its L2 switched on the FI, setup of ESXi are identical )

Ping from a 6500 to UCS max 8972, from same 6500 to Netapp 9000 ?

I know that essentially the source application determines the max size used ? any explanation of the algorithm ?

Everyone's tags (2)
8 REPLIES
Cisco Employee

Jumbo Frames size

Hello wdey,

The general rule is 9216 in the network devices (swicthes and routers) and 9000 in the server side, just because some operational systems doesn't support mtu greater than 9000. Another reason to leave the mtu bigger in the network is because the packet coming from the server side could be encapsulated by some protocol (VxLAN, NVGRE, Tunneling, etc.).

Why the ping doesn't work with 9000 packet size?

This packet size configured by the ping, is without the header, when putting the header (IP 20Bytes + ICMP 8 Bytes = 9000 - 28 = 8972)  in the packet the server interface can't forward it without fragmentation.

The application usually do not control the mtu, this is a S.O. function, if you are using UCS you can configure the MTU there or also overwrite by the S.O. installed (not recommended). Few very specific applications are different.

I'm not sure if I helped.

VIP Green

Jumbo Frames size

Thanks Richard

S.O. = ? OS ?

This explains the 8972, which I hoped was solving the issue;

But then we discovered in the DR site (exactly the same hardware / software) a max of 8944, (=8972-28);

Specially weird: even between 2 ESXi in the same UCS domain, we see 8944 ? (and it's 100% sure not leaving the UCS)

Could it be due to some VNtag overhead ?

Cisco Employee

Jumbo Frames size

Hello wdey,

Sorry about that, S.O or O.S, I mean operating system.

I can't identify where exactly is the problem, I don't believe the VN-tag could be a issue, VN_tag just increased 6 bytes in the header frame.

Which virtual switch are you using in your ESX enviroment?

VIP Green

Jumbo Frames size

OK, with ping the overhead was 20 (IP) plus 8 (ICMP) = 8972 bytes.

Does it mean one should set MTU on the vnic's in the SP to 9000 - 20 (TCP) - 20 (IP) = 8960 bytes ?

Cisco Employee

Jumbo Frames size

No, you can leave the vnic's mtu as 9000. The rest of your network you will set to 9216 (if cisco) including the UCS QoS configuration.

The mtu as 9216 in the network can handle any overhead you can take in your servers.

VIP Green

Jumbo Frames size

Thanks Richard

Just a feedback, in

Cisco UCS Manager GUI Configuration Guide, Release 2.2

First Published: December 11, 2013

On p 272, 284, 288,

This is not correct, the max MTU value can only be set to 9000 ! NOT 9216

Cisco Employee

Jumbo Frames size

Hi wdey,

I didn't said that UCS doesn't support 9216 in the vnic, it supports but the vnic configuration reflects the operating system compatibility.

The MTU as 9000 is just because most part of operating systems follow the 9000 in the configuration, the rest of the network you can leave as 9216.

VMware:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007654

Windows:

http://www.altaro.com/hyper-v/how-to-adjust-mtu-jumbo-frames-on-hyper-v-and-windows-server-2012/

Linux:

http://tacticalvim.wordpress.com/2013/03/07/setting-up-jumbo-frames-in-rhelcentos/

VIP Green

Jumbo Frames size

You agree that with UCSM the MTU for vnic can only be set to 9000 ! not 9216, as the documentation claims; in the LAN QoS section you can set it to 9216.

2315
Views
0
Helpful
8
Replies