Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

Jumbo support on the Nexus N7K-C7009

Hi experts,

Forgive me if it is silly question...

The server team has two servers on two different subnets/vlans and they want to use jumbo frames to tranfer data. The are actually connected on two N5Ks and 5Ks are layer 2 so the default gateways are on the 7Ks. I think I have found the configurations that I need to apply on the 5Ks and 7Ks to allow the jumbo frames on layer2 links

-- 5K --

policy-map type network-qos JUMBO_MTU

  class type network-qos class-default

    mtu 9216

!

system qos

  service-policy type network-qos JUMBO_MTU

-- 7K --

system jumbomtu 9216

!

interface ethernet x/y

mtu 9216

!

My question is, do I need to configure MTU on the vlan interfaces to avoid fragmentation as well on the 7Ks?

!

interface vlan xxx

mtu 9216

!

So could you go through my configuration and see if I am missing anything?

Thanks a bunch!

1 ACCEPTED SOLUTION

Accepted Solutions

Jumbo support on the Nexus N7K-C7009

Hi,

You will need the mtu 9216 command on the VLAN interface of the Nexus 7K. If you don't add this the Nexus 7K will do one of two things:

  • If the DF (Don't Fragment) bit within the IP header is not set by the server, the router will fragment the packets and forward on. This will impact both the router performance and the throughput that the servers are able to achieve between each other. You definitely want to avoid this.
  • If the server Operating System supports and is configured for Path MTU Discovery, it will set the DF bit in the IP packet header causing the Nexus to return an ICMP Type 3 Code 4 (Fragmentation Needed and Don't Fragment was Set). This will result in the hosts reducing the maximum packet size they sends.

The other thing to note with your setup is that the Nexus 5K doesn't show the configured 9216-byte MTU in the output of commands such as show interface. You can see that the policy has been taken using the show queuing interface command though.

Regards

9 REPLIES

Jumbo support on the Nexus N7K-C7009

Hi,

You will need the mtu 9216 command on the VLAN interface of the Nexus 7K. If you don't add this the Nexus 7K will do one of two things:

  • If the DF (Don't Fragment) bit within the IP header is not set by the server, the router will fragment the packets and forward on. This will impact both the router performance and the throughput that the servers are able to achieve between each other. You definitely want to avoid this.
  • If the server Operating System supports and is configured for Path MTU Discovery, it will set the DF bit in the IP packet header causing the Nexus to return an ICMP Type 3 Code 4 (Fragmentation Needed and Don't Fragment was Set). This will result in the hosts reducing the maximum packet size they sends.

The other thing to note with your setup is that the Nexus 5K doesn't show the configured 9216-byte MTU in the output of commands such as show interface. You can see that the policy has been taken using the show queuing interface command though.

Regards

New Member

Jumbo support on the Nexus N7K-C7009

Thank you Steve.


It is a good point to check on the server whether they have PMD enabled or not. If they do, depending on where they are sending the probe traffic, the MTU might still be set back to 1500 because that is the setting we have for everywhere else...

I am also aware that the first router does the fragmentation and the last router does the assembly. In this case, just being curious, does the 7K (with default 1500 MTU on the vlan interface) defrag and then assemble right away?

One last question but also the most important one, when I change MTU with my script, will that cause inconsistency on the vPCs either glabally or on interfaces..? Will it cause any down time?

Thanks!

Jumbo support on the Nexus N7K-C7009

I am also aware that the first router does the fragmentation and the last router does the assembly. In this case, just being curious, does the 7K (with default 1500 MTU on the vlan interface) defrag and then assemble right away?

The above statement is not actually correct. A router may fragment the packets, but it is the receiving host that does the reassembly.

One last question but also the most important one, when I change MTU with my script, will that cause inconsistency on the vPCs either glabally or on interfaces..? Will it cause any down time?

When changing MTU you certainly need to be mindful of vPC. If you have a Nexus 5K release prior to NX-OS 5.0(2)N1(1) then an MTU mismatch is a Type 1 inconsistency and the vPC secondary peer will disable vPC member ports so you need to be especially careful. As part of NX-OS 5.0(2)N1(1) Support for Type 2 vPC Consistency Checks was added which changed a number of parameters from Type 1 to Type 2 inconsistency as so downtime is less likely in this case.

The Virtual Port Channel Operations guide has more details on the operation of vPC and is probably worth reviewing so you fully understand the implications.

Regards

New Member

Jumbo support on the Nexus N7K-C7009

Thanks!

New Member

Re: Jumbo support on the Nexus N7K-C7009

Actually another question, on the 7K, do I apply the "MTU 9216" configuration on the port-channel interfaces that are connected with 5Ks or do I configure it on the component physical interfaces or both? Do I apply it on the peer-link port-channel as well? thanks!

Jumbo support on the Nexus N7K-C7009

Hi,

The increased MTU setting will need to be applied to both the physical and port-channel interfaces. Whether the peer link will require the MTU change as well will depend upon whether the peer-link could be used to carry jumbo frames.

When I increase the MTU I do so on all interfaces that could possibly be in the path of the traffic, both under normal operating conditions and when a link or device has failed and the network converges around the failure.

I've seen quite a number of problems caused by partialy enabling of jumbo frames i.e., only some links in the network being enabled, so I make sure that all possible paths between any jumbo enabled end-points are all jumbo enabled.

Regards

New Member

Jumbo support on the Nexus N7K-C7009

My understanding is that peer-link will be used when failure happens to one side of the vpc links, true? So I shoud set MTU on those links as well, correct? Thanks!

Jumbo support on the Nexus N7K-C7009

Hi Difan,

That's correct. The vPC peer-link is used under certain failure conditions so it should also have the MTU increased on this link.

Regards

Silver

Jumbo support on the Nexus N7K-C7009

You will need the mtu 9216 command on the VLAN interface of the Nexus 7K. If you don't add this the Nexus 7K will do one of two things:

If  the DF (Don't Fragment) bit within the IP header is not set by the  server, the router will fragment the packets and forward on.

It is a Layer3 interface, so a bit different. To my knowledge the IP fragmentation mechanism is performed by IP (L3) process, not a switching process. The switching process drops the ingress frame on a physical interface if the payload is larger than the MTU. The L3 process fragments or drops the egress packet on a Layer3 interface if the packet is larger than the MTU (assuming it had not been already dropped on ingress interface).

1889
Views
5
Helpful
9
Replies
CreatePlease login to create content