Full switch outage after enabling jumbo frames (mtu 9216)

[joshabaire]

I attempted to enable jumbo frames on a Nexus 5010 (NX-OS version 4.2(1)N1(1)).  I created the policy map below and lost access to the switch.

policy-map type network-qos jumbo

class type network-qos class-default

mtu 9216

After recovery I see from the logs that all vlans and interface were suspended.  I've attempted to look for reasons for a compatibility issue but I am unable to find what is checked and what could have been incompatible.  The other troubling thing is the adjacent switch suspended its interfaces too but no change was done there.  Can anyone explain what I've done wrong and what I need to look out for so that this does not happen again?

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1,10,601 on Interface port-channel1 are being suspen

ded. (Reason: QoSMgr Network QoS configuration incompatible)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-5-IF_TRUNK_DOWN: Interface port-channel1, vlan 1,10,601 down

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 10 on Interface port-channel508 are being suspended.

(Reason: Global compat check failed)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 10 on Interface port-channel507 are being suspended.

(Reason: Global compat check failed)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 10 on Interface port-channel506 are being suspended.

(Reason: Global compat check failed)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 10 on Interface port-channel505 are being suspended.

(Reason: Global compat check failed)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 601 on Interface port-channel18 are being suspended.

(Reason: Global compat check failed)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-5-IF_DOWN_INACTIVE: Interface port-channel508 is down (Inactive)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-5-IF_DOWN_INACTIVE: Interface port-channel507 is down (Inactive)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-5-IF_DOWN_INACTIVE: Interface port-channel506 is down (Inactive)

2011 Nov 22 23:43:09 phx-ipcg1dwfcma %ETHPORT-5-IF_DOWN_INACTIVE: Interface port-channel505 is down (Inactive)

[Alexander Maroukian · Accepted solution]

Hi,

Most probably this have happened because you have changed the compatibility requirements of the port-channel. Because on the Nexus 5000 for Layer 2 the MTU is changed on the global basis it had this effect.

You can find more info on this on the following link:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/421_n1_1/Cisco_n5k_layer2_config_gd_rel_421_n1_1_chapter7.html#con_1158953

Please rate helpful answers.

Best regards,

Alex

[joshabaire]

Thats correct.  Thank You.  So whats the solution?  How can I change the MTU on these switches without losing access?  I actually resolved the issue by removing the members from the vPC uplink from the adjacent switch to the 7k.  Then when I applied the MTU change I still had access to the other switch to replicate the change.  This worked because these 5ks are not in production yet.  If this was a change to a production switch, messing with the uplinks might not be practical.

Whats the best practice here?

[Alexander Maroukian]

Well it is good idea to prepare the portchannel before implementing it. So MTU settings should be ready before this. Also you can have additional(backup) connection besides portchannel between and to the switches. In production I would do this only during the maintenance window. You can use also configure replace and if it is supported on the device.More on configure replace command:

http://www.cisco.com/en/US/docs/ios/fundamentals/configuration/guide/cf_config-rollback.html

Bets regards,

Alex

[joshabaire]

How do you prepare a portchannel?  On a 5k, the MTU is applied to a layer 2 interface by applying the policy-map.  Apply the policy, every member changes its MTU.

[Alexander Maroukian]

I mean first apply mtu and whatever changes you need globally and then make etherchannel.You can create port channel interface then add the switchport interfaces to Etherchannel.

Best regards,

Alex

[Glenn Gaines]

What if the port channels are already configured, how could this be prevented?

[Kevin Dorrell]

Bump.

I am very interested in this because our VMware team have asked me to implement jumbo frames for their vmotion, but I had not done so yet because I was afraid of the disruption.

We have two 5548s, and practicaally everything is vPC.  Even the FEXes are dual homed, and we often have extended vPC into the servers through multiple FEXes.  We have over 60 VPC port-channels so I don't really feel like dismantling all of those.

From these postings, it looks like it would be an absolute no-no to change the MTU on the fly in a vPC configuration.  I don't have a maintenance slot until October, so it looks like I'm stuck.  Has anyone any comments?

Kevin Dorrell

Luxembourg

[joshabaire]

Just following up to provide a solution for everyone.  The problem with changing MTU is that you will create a consistency problem and disrupt traffic.  Cisco has implemented syncronization via switch profiles to allow a configuration to bhe push to two switches at once.  This will allow the MTU change to be applied on the peer switch and keep the vPC consistent and thus up.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_config_sync_ops.html

[Kevin Dorrell]

Hi, and thank you for that reply.  I do already use conf sync quite extensively - somewhat over half of the lines in my configuration file are in the switch-profile - but sometimes I wish I didn't.  I have had so many problems with it, mainly due to bugs in the synchronisation database getting out of step with reality, that I'm not sure I dare to take this risk.If my VPCs went down, I could lose my entire data centre.  I think I shall discuss it with the TAC.

Thanks again

Kevin Dorrell

Luxembourg

[raul.hernandez]

Hello,

     I have a doubt related to this issue. How Cisco Nexus detects the MTU inconsistency between the physical interfaces associated to the port-channel between two Cisco Nexus switches?. CDP or LLDP (DCBXP)?

     We must implement in a live DaCe the jumbo Frames and I was thinking a way to avoiding the service disruption disabling LLDP or CDP in related interfaces. Also I´m worried about impact in other port-channel to non Cisco elements.

Thanks for your support,
Regards,

Raúl, Spain.