New generation networking equipments like servers and Nexus switches support jumbo mtu so to send a big chunk of data in one go. This improves overall network throughput if all equipments support jumbo mtu size. However if jumbo mtu is not supported by all switches in the path it may result in packet fragmentation or in some cases packet drops. If the DF (Don't Fragment) bit within the IP header is not set by the server, the router will fragment the packets and forward on. This will impact both the router performance and the throughput that the servers are able to achieve between each other. If the server Operating System supports and is configured for Path MTU Discovery, it will set the DF bit in the IP packet header causing the Nexus to return an ICMP Type 3 Code 4 (Fragmentation Needed and Don't Fragment was Set). This will result in the hosts reducing the maximum packet size they sends.
There are two servers on two different subnets/vlans and they want to use jumbo frames to tranfer data. The servers are actually connected on two Nexus 5000 switches and these are layer 2 so the default gateways are on the Nexus 7000 switches.
For Nexus 5K release prior to NX-OS 5.0(2)N1(1), an MTU mismatch is a Type 1 inconsistency and the vPC secondary peer will disable vPC member ports. As part of NX-OS 5.0(2)N1(1) Support for Type 2 vPC Consistency Checks was added which changed a number of parameters from Type 1 to Type 2 inconsistency as so downtime is less likely in this case. Nexus 5K doesn't show the configured 9216-byte MTU in the output of commands such as show interface. You can see that the policy has been taken using the show queuing interface command though.
Note: For L2, you cannot set the MTU per interface but only system wide. The mtu is set under network-qos which can only be attached to system qos. For L3 you can apply it to just one or a group-of interface, instead of the "system qos" just use "interface ethx/y".
MTU Size Considerations
For Layer 3 interfaces, you can configure the MTU to be between 576 and 9216 bytes (even values are required). For Layer 2 interfaces, you can configure the MTU to be either the system default MTU (1500 bytes) or the system jumbo MTU size (which has the default size of 9216 bytes). You can change the system jumbo MTU size, but if you change that value, you should also update the Layer 2 interfaces that use that value so that they use the new system jumbo MTU value. If you do not update the MTU value for Layer 2 interfaces, those interfaces will use the system default MTU (1500 bytes).
For Jumbo mtu to work properly it should be configured on all interfaces, including physical interface and vlan interface (on Nexus 7K). If you have port channel configured then it should also have jumbo mtu enabled. Make sure that all possible paths between any jumbo enabled end-points are all jumbo enabled.
Nexus 5K config:
policy-map type network-qos JUMBO_MTU class type network-qos class-default mtu 9216
system qos service-policy type network-qos JUMBO_MTU