Jumbo support on the Nexus N7K-C7009

[Difan Zhao]

Hi experts,

Forgive me if it is silly question...

The server team has two servers on two different subnets/vlans and they want to use jumbo frames to tranfer data. The are actually connected on two N5Ks and 5Ks are layer 2 so the default gateways are on the 7Ks. I think I have found the configurations that I need to apply on the 5Ks and 7Ks to allow the jumbo frames on layer2 links

-- 5K --

policy-map type network-qos JUMBO_MTU

  class type network-qos class-default

    mtu 9216

!

system qos

  service-policy type network-qos JUMBO_MTU

-- 7K --

system jumbomtu 9216

!

interface ethernet x/y

mtu 9216

!

My question is, do I need to configure MTU on the vlan interfaces to avoid fragmentation as well on the 7Ks?

!

interface vlan xxx

mtu 9216

!

So could you go through my configuration and see if I am missing anything?

Thanks a bunch!

[Steve Fuller · Accepted solution]

Hi,

You will need the mtu 9216 command on the VLAN interface of the Nexus 7K. If you don't add this the Nexus 7K will do one of two things:

• If the DF (Don't Fragment) bit within the IP header is not set by the server, the router will fragment the packets and forward on. This will impact both the router performance and the throughput that the servers are able to achieve between each other. You definitely want to avoid this.
• If the server Operating System supports and is configured for Path MTU Discovery, it will set the DF bit in the IP packet header causing the Nexus to return an ICMP Type 3 Code 4 (Fragmentation Needed and Don't Fragment was Set). This will result in the hosts reducing the maximum packet size they sends.

The other thing to note with your setup is that the Nexus 5K doesn't show the configured 9216-byte MTU in the output of commands such as show interface. You can see that the policy has been taken using the show queuing interface command though.

Regards

[Difan Zhao]

Thank you Steve.

It is a good point to check on the server whether they have PMD enabled or not. If they do, depending on where they are sending the probe traffic, the MTU might still be set back to 1500 because that is the setting we have for everywhere else...

I am also aware that the first router does the fragmentation and the last router does the assembly. In this case, just being curious, does the 7K (with default 1500 MTU on the vlan interface) defrag and then assemble right away?

One last question but also the most important one, when I change MTU with my script, will that cause inconsistency on the vPCs either glabally or on interfaces..? Will it cause any down time?

Thanks!

[Steve Fuller]

I am also aware that the first router does the fragmentation and the last router does the assembly. In this case, just being curious, does the 7K (with default 1500 MTU on the vlan interface) defrag and then assemble right away?

The above statement is not actually correct. A router may fragment the packets, but it is the receiving host that does the reassembly.

One last question but also the most important one, when I change MTU with my script, will that cause inconsistency on the vPCs either glabally or on interfaces..? Will it cause any down time?

When changing MTU you certainly need to be mindful of vPC. If you have a Nexus 5K release prior to NX-OS 5.0(2)N1(1) then an MTU mismatch is a Type 1 inconsistency and the vPC secondary peer will disable vPC member ports so you need to be especially careful. As part of NX-OS 5.0(2)N1(1) Support for Type 2 vPC Consistency Checks was added which changed a number of parameters from Type 1 to Type 2 inconsistency as so downtime is less likely in this case.

The Virtual Port Channel Operations guide has more details on the operation of vPC and is probably worth reviewing so you fully understand the implications.

Regards

[Difan Zhao]

Thanks!

[Difan Zhao]

Actually another question, on the 7K, do I apply the "MTU 9216" configuration on the port-channel interfaces that are connected with 5Ks or do I configure it on the component physical interfaces or both? Do I apply it on the peer-link port-channel as well? thanks!

[Steve Fuller]

Hi,

The increased MTU setting will need to be applied to both the physical and port-channel interfaces. Whether the peer link will require the MTU change as well will depend upon whether the peer-link could be used to carry jumbo frames.

When I increase the MTU I do so on all interfaces that could possibly be in the path of the traffic, both under normal operating conditions and when a link or device has failed and the network converges around the failure.

I've seen quite a number of problems caused by partialy enabling of jumbo frames i.e., only some links in the network being enabled, so I make sure that all possible paths between any jumbo enabled end-points are all jumbo enabled.

Regards

[Difan Zhao]

My understanding is that peer-link will be used when failure happens to one side of the vpc links, true? So I shoud set MTU on those links as well, correct? Thanks!

[Steve Fuller]

Hi Difan,

That's correct. The vPC peer-link is used under certain failure conditions so it should also have the MTU increased on this link.

Regards

[Peter Koltl]

You will need the mtu 9216 command on the VLAN interface of the Nexus 7K. If you don't add this the Nexus 7K will do one of two things:
If  the DF (Don't Fragment) bit within the IP header is not set by the  server, the router will fragment the packets and forward on. 

It is a Layer3 interface, so a bit different. To my knowledge the IP fragmentation mechanism is performed by IP (L3) process, not a switching process. The switching process drops the ingress frame on a physical interface if the payload is larger than the MTU. The L3 process fragments or drops the egress packet on a Layer3 interface if the packet is larger than the MTU (assuming it had not been already dropped on ingress interface).

[Steven Doremus]

One last question but also the most important one, when I change MTU with my script, will that cause inconsistency on the vPCs either glabally or on interfaces..? Will it cause any down time?

Thanks!

To verify enter the following cmd on the N7k.

Show vpc consistency global | interface port-channel channel-number | vlans | vpc number } where VPC will be the Port Channel interface.

[Steven Doremus]

Just remember when you set jumbo frames on 5k via system QOS if you enter the following command you will see the reflected change.

show queueing interface \interface name

[JHarris6117]

Quick clarity question:

On a 7K, in regards to L2 interfaces only...

1) If you configure "system jumbomtu 9216", will this set all layer 2 interfaces (including port channel's and their member interfaces) to MTU of 9216?  or is per port configuration still required?

2) If you have to configure the service-policy on the 7K in order to increase MTU on 7K attached fexes (config below), does this affect only fex interfaces, or is this a global 7K setting as it is in the 5K/6K?

**Config sample start**

policy-map type network-qos default-nq-8e-4q8q-policy template 8e-4q8q
 class type network-qos c-nq-8e-4q8q
  match cos 0-7
   congestion-control tail-drop threshold burst-optimized
    mtu 9216

system qos
service-policy type network-qos default-nq-8e-4q8q-policy

**Config Sample End**

3) Content below eludes to a need to configure both a port-channel interface and a member interface for increased MTU.  Shouldn't the configuration of the port-channel parent push the increased MTU to the member interfaces?

Thanks