Ask the Expert: Troubleshooting Nexus 5000/2000 series switches

Unanswered Question
Jun 18th, 2012

Read the bioWith Prashanth Krishnappa

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn about how to troubleshoot the Nexus 5000/2000 series switches.

Prashanth Krishnappa is an escalation engineer for datacenter switching at the Cisco Technical Assistance Center in Research Triangle Park, North Carolina. His current responsibilities include escalations in which he troubleshoots complex issues related to the Cisco Catalyst, Nexus and MDS product lines as well as providing training and author documentation. He joined Cisco in 2000 as an engineer in the Technical Assistance Center. He holds a bachelor's degree in electronics and communication engineering from Bangalore University, India, and a master's degree in electrical engineering from Wichita State University, Kansas. He also holds CCIE certification (#18057).

Remember to use the rating system to let Prashanth know if you have received an adequate response. 

Prashanth might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum shortly after the event. This event lasts through June 29, 2012. Visit this forum often to view responses to your questions and the questions of other community members.

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Average Rating: 4.7 (3 ratings)
sarah.staker Tue, 06/19/2012 - 09:26

Hi Prashanth,

I have migrated my FCOE set up from a pair of 5020s to a pair of Nexus 5500s and my vFCs is not coming up. Configurations have been triple checked and they are identical - can you help?

Thanks.

prkrishn Tue, 06/19/2012 - 14:25

Hello Sarah

Unlike 50x0, in 5500s, until 5.1(3)N1(1), FCOE queues are not created by default when you enable "feature fcoe"

Make sure you have the following policies under system QoS

system qos

  service-policy type queuing input fcoe-default-in-policy

  service-policy type queuing output fcoe-default-out-policy

  service-policy type qos input fcoe-default-in-policy

  service-policy type network-qos fcoe-default-nq-policy

Thanks

-Prashanth

kamin-ganji Sun, 03/02/2014 - 00:47

Hi,

I have a couple nexus 5k which are peer together. I  linked a 3750 Switch to this peer with trunk port. I've configured all vlans on VPC peers as 3750.

I have eigrp on 3750 and I want to migrate all routing from 3750 to new nexus switches. 3750 can see nexus switches hsrp IP address on one of the vlans and vise versa.

Can I use thus vlan inteface in both sides for eigrp neighborship or I have to create L3 interface on nexus switchs instead of existing trunk port?

steffenwebb Mon, 03/03/2014 - 00:17

Hi Kamin-ganji.

I believe that you have the same restrictions on a N5K as on a N7K. But check the design guides on cisco.com. This is from the N7K guide.

http://www.cisco.com/en/US/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

Layer 3 and vPC: Guidelines and Restrictions

Attaching a L3 device (router or firewall configured in routed mode for instance) to vPC domain using a vPC is not a supported design because of vPC loop avoidance rule.

To connect a L3 device to vPC domain, simply use L3 links from L3 device to each vPC peer device.

L3 device will be able to initiate L3 routing protocol adjacencies with both vPC peer devices.

One or multiple L3 links can be used to connect to L3 device to each vPC peer device. NEXUS 7000 series support L3 Equal Cost Multipathing (ECMP) with up to 16 hardware load-sharing paths per prefix. Traffic from vPC peer device to L3 device can be load-balanced across all the L3 links interconnecting the 2 devices together.

Using Layer 3 ECMP on the L3 device can effectively use all Layer 3 links from this device to vPC domain. Traffic from L3 device to vPC domain (i.e vPC peer device 1 and vPC peer device 2) can be load-balanced across all the L3 links interconnecting the 2 entities together.

YapChinHoong_2 Thu, 06/21/2012 - 08:20

Hi Prashanth,

   Thanks for having this session. First question that I have is whether Jumbo MTU is supported across the vPC Peer Link on N5Ks? Below is the output when I tried to configure this in N7K, but I presume N5K may have the same symptom. Thanks.

RCS-WG1(config-if)# int po10

RCS-WG1(config-if)# mtu 9216

ERROR: port-channel10: Cannot configure port MTU on Peer-Link.

RCS-WG1(config-if)# sh int po10

port-channel10 is up

  Hardware: Port-Channel, address: 70ca.9bf8.eef5 (bia 70ca.9bf8.eef5)

  Description: *** vPC Peer Link ***

  MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec

  reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is trunk

  full-duplex, 10 Gb/s

  Input flow-control is off, output flow-control is off

  Switchport monitor is off

  EtherType is 0x8100

  Members in this channel: Eth3/1, Eth3/2

  Last clearing of "show interface" counters never

  30 seconds input rate 23120 bits/sec, 34 packets/sec

  30 seconds output rate 23096 bits/sec, 34 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 23.05 Kbps, 32 pps; output rate 23.06 Kbps, 31 pps

  RX

    12 unicast packets  60639 multicast packets  6 broadcast packets

    60657 input packets  5024553 bytes

    0 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    12 unicast packets  60688 multicast packets  340 broadcast packets

    61040 output packets  5568549 bytes

    0 jumbo packets

    0 output error  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble  0 output discard

    0 Tx pause

  2 interface resets

RCS-WG1(config-if)#

prkrishn Fri, 06/22/2012 - 05:22

Hello YapChinHoong

Jumbo QoS configuration on Nexus 5000 is configured per QoS class of group using QoS configurations and not per interface. When applied, the QoS setting applies to all ethernet interfaces including the peer-link

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/qos/513_n2_1/b_Cisco_Nexus_5000_QoS_Config_Guide_513_N2_1_chapter_0101.html#con_1150612

Note that when you are configuring jumbo on switches configured for FCoE, use the following

policy-map type network-qos fcoe+jumbo-policy
 class type network-qos class-fcoe
   pause no-drop
   mtu 2158
 class type network-qos class-default
   mtu 9216
   multicast-optimize
system qos
 service-policy type network-qos fcoe+jumbo-policy
YapChinHoong_2 Thu, 06/21/2012 - 08:25

Hi Prashanth,

   The 2nd question that I have is regarding the effectiveness of storm control on the Nexus platform, partcilarly N5K and N7K.

   The concern that I have is the storm control falling threshold capability as with the mid-range Catalyst platforms (eg: C3750), in which the port blocks traffic when the rising threshold is reached. The port remains blocked until the traffic rate drops below the falling threshold (if one is specified) and then resumes normal forwarding. The graph below is excerpted from OReilly – Network Warrior.

N7K and N5K never mentions anything about the falling threshold mechanism, so the graphs looks like this (as get from N7K config guide).

In the event of broadcast storms, theoretically the graph looks like this.

This means only 50% of the broadcast packets will be suppressed or dropped. Assuming 300’000 broadcast packets hitting the SVI within a second, 150’000 will be hitting the SVI, which is often sufficient to cause high CPU, switch starts not responding to UDLD from peers, peer devices blocking ports due to UDLD, and then a disastrous network meltdown.

Appreciate your comment upon this. Thanks.

prkrishn Fri, 06/22/2012 - 05:26

Hello YapChinHoong

In addition to any user configured storm control, the SVI/CPU is also protected by default Control plane policing. In lab, I tested sending 10Gig line rate broadcast in a 5500 and noticed that the switch CPU and other control plane protocols were not affected.

F340.24.10-5548-1# sh policy-map interface control-plane class copp-system-class-default

control Plane

  service-policy  input: copp-system-policy-default

    class-map copp-system-class-default (match-any)

      match protocol default

      police cir 2048 kbps , bc 6400000 bytes

        conformed 45941275 bytes; action: transmit

        violated 149875654008 bytes; action: drop<<<---------

siddhartham Mon, 06/25/2012 - 12:31

Hello Prashanth,

I have few questions about Nexus 500s and 5500s

Hardware port channel resources-

According to the below document 16 hardware PortChannels is the limit on 5020 and 5010 switches . Does the 5548s with layer 3 daughter card has any kind limitation on the hardware portchannel resources.Does a FEX (2248 or 2232) dualhomed to a 5548 with L3 consume a Hardware portchannel?

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572829-01_Design_N5K_N2K_vPC_DG.pdf

Can you please confirm the below designs- 5548 are running on 5.1(3)N1(1),  5010s are on 5.0(3)N2(1)

prkrishn Mon, 06/25/2012 - 14:53

Hello Siddhartham

55xx support 48 local port channels. But only port-channels having more than one interface in it count against this 48 limit. Since your FEX only has one interface per Nexus 5k, it

does not use up a resource.

Regarding your topologies, you are referring to the Enhanced vPC(E-vPC). E-vPC is only

supported on the 55xx platforms. So your second topology is not supported since it used Nexus 5010 as parent switch for the FEX.

Thanks

-Prashanth

siddhartham Tue, 06/26/2012 - 07:20

Thanks Prashanth.

Since the number of FEXs supported by 5548 with L3 card is eight, even if I use two 10Gig links between a FEX and each 5548, this will only consume 8 hardware portchannels out of available 48.

Do we have any limit on the 2248/2232 FEXs, will the port channel on a FEX count against the Limit of a 5500.

Siddhartha

prkrishn Tue, 06/26/2012 - 09:25

The FEX port-channels do not count against the limit of the 5500.

Leo Laohoo Tue, 06/26/2012 - 16:25
Since the number of FEXs supported by 5548 with L3 card is eight, 

"Up to 24 fabric extenders per Cisco Nexus 5548P, 5548UP, and 5596UP switch (8 fabric extenders for Layer 3 configurations)" - This line was taken from the data sheet of the B22 (Table 2).

"Up to 24 fabric extenders per Cisco Nexus 5548P, 5548UP, 5596UP switch (16 fabric extenders for L3 configurations): up to 1152 Gigabit Ethernet servers and 768 10 Gigabit Ethernet servers per switch" - This line was taken from the data sheet of the Nexus 2000 (Table 2).

Leo Laohoo Wed, 06/27/2012 - 15:21

Thanks Sid.

I just completed Nexus training so I'm eager to learn from the experts here. 

Good responses Prashanth. 

aik-chiew.goh Wed, 06/27/2012 - 03:59

Hi,

Since we are in this topic, are we able to turn "on" or "off" the L3 capabilities of the N5548UPs?

Thanks,

Dennis Goh

prkrishn Wed, 06/27/2012 - 05:59

Hello Dennis

Most of the features including L3 features are enabled using the feature command. Here is an output from a lab switch

5548-1(config)# feature ?

  bgp             Enable/Disable Border Gateway Protocol (BGP)

  cts             Enable/Disable CTS

  dhcp            Enable/Disable DHCP Snooping

  dot1x           Enable/Disable dot1x

  eigrp           Enable/Disable Enhanced Interior Gateway Routing Protocol (EIGRP)

  fcoe            Enable/Disable FCoE/FC feature

  fcoe-npv        Enable/Disable FCoE NPV feature

  fex             Enable/Disable FEX

  flexlink        Enable/Disable Flexlink

  hsrp            Enable/Disable Hot Standby Router Protocol (HSRP)

  http-server     Enable/Disable http-server

  interface-vlan  Enable/Disable interface vlan

  lacp            Enable/Disable LACP

  msdp            Enable/Disable Multicast Source Discovery Protocol (MSDP)

  npiv            Nx port Id Virtualization (NPIV) feature enable

  npv             Enable/Disable FC N_port Virtualizer

  ospf            Enable/Disable Open Shortest Path First Protocol (OSPF)

  pim             Enable/Disable Protocol Independent Multicast (PIM)

  port-security   Enable/Disable port-security

  private-vlan    Enable/Disable private-vlan

  privilege       Enable/Disable IOS type privilege level support

  rip             Enable/Disable Routing Information Protocol (RIP)

  ssh             Enable/Disable ssh

  tacacs+         Enable/Disable tacacs+

  telnet          Enable/Disable telnet

  udld            Enable/Disable UDLD

  vpc             Enable/Disable VPC (Virtual Port Channel)

  vrrp            Enable/Disable Virtual Router Redundancy Protocol (VRRP)

  vtp             Enable/Disable Vlan Trunking Protocol (VTP)

So if a feature is not needed, you can turn it off using "no feature".

Thanks

-Prashanth

aik-chiew.goh Wed, 06/27/2012 - 06:52

Hi Prashanth,

Thank you for your reply. Does that mean that if I do not turn on any L3 features on the Nexus 5500, I can be able to scale up to 24 FEXs dual-homed?

P.s. yup, I have L3 daughterboard installed

prkrishn Wed, 06/27/2012 - 07:00

Hi Dennis

The switch is considered Layer 3 only when you install the L3 licenses. So if you do not need the L3 features at this time and want to use it as L2 switch with upto 24 FEX, you could uninstall all the L3 licenses.

Thanks

-Prashanth

aik-chiew.goh Wed, 06/27/2012 - 07:13

Hi Prashanth,

Correct me if I'm wrong, What I do to be able to scale up to 24 FEXs is by backing up my licenses, then uninstalling the L3 license. And if one day in the future I need this L3 feature, I just restore my license from the backup that I've made?

Btw, how do I restore backed up licenses on the N5500? I can't seem to find any guides on it.

Thanks in advance.

prkrishn Wed, 06/27/2012 - 07:38

Hello Dennis

Just installing license using "clear license" command should be enough. Here is an example from my lab switch.

5548-1# clear license ?

  MDS20101212190435659.lic

  MDS20110331181304989.lic

  MDS20110331181349133.lic

  WORD                      License file to be uninstalled

But if you want to back up all the license files, you can do that too. Here is how you do it.

1)copy license bootflash:file-name.tar

2)Then issue "clear license" like above.

If you need to, reinstall the license at some point in the future

1)tar extract bootflash:file-name.tar

2)Install the license back using "install license bootflash:" command.

siddhartham Fri, 06/29/2012 - 10:44

Hi Prashanth,

Question about Enhanced vPC-

"The Dual-homed FEX topology can also be deployed for servers that have multiple NICs

but do not support 802.3ad"

Does the above statement mean we can't use LACP to dual home a server to dual homed FEXs?

Siddhartha

tenaro.gusatu.novici Wed, 07/04/2012 - 07:17

Hi,

here is one simple question: if I have Nexus 5k connected to another cisco device that also supports vPC, will CDP work if port on one side belongs to vPC while on another side is configured as simple access port?

Thanks,

Tenaro

aaditi_hirave Tue, 08/21/2012 - 03:32

Is there any Nexus Switch Emulator/Manager available which can be used for getting switch statistics?

Leo Laohoo Wed, 08/22/2012 - 00:16
Is there any Nexus Switch Emulator/Manager available which can be used for getting switch statistics?

not sure but try DCNS.

pdervaux Tue, 01/29/2013 - 02:37

Hi Dennis,

I'm curently facing a similar problem. Do you het an answer or solve this issue?

The Cisco answer is diabling Layer 3 features but do we need to rmove the license and / or the Layer 3 card?

Tanks,

Pascal

aik-chiew.goh Tue, 01/29/2013 - 05:46

Hello Pascal,

Newer versions of NX-OS supports up to 16 EvPCs (my customer uses 10 FEXs so far) hence I did not attempt to try out removing the licenses. I have yet to try removing the license as my project timeline grew shorter and I was unable to test anymore. If you have the time, do give Prashant's guides a try and let us know If I have a chance to deal with N55XX with L3 module, I will attempt to try this out

Thanks

Dennis Goh

aik-chiew.goh Sun, 08/26/2012 - 19:26

Is this post still active?

Would like to ask further on L3 modules. Lets say I'm performing an NX-OS upgrade on a Nexus5500 series which is equipped with L3 module. I have L3 license installed, however I do not use any L3 capabilities at all.

Whenever I perform an ISSU check for upgrading, the output will always be "disruptive" which I suspect is due to the reason that I have an L3 module installed. I know for a fact that my configurations on a Nexus5500 unit without an L3 module will be non-disruptive. Can I safely tell the customer that their production network will not encounter any downtime if I perform an NX-OS upgrade?

Thanks in advance!

Dennis Goh

samuelmendieta Mon, 09/24/2012 - 12:33

Hi Prashanth. A pleasure to greet, my name is Samuel. I am trying to set certain SNMP commands in the Nexus 7000 in our network, to monitor the computer for any problems and do not know where to configure them.

On a Cisco router I can configure it normally, but, in the Nexus does not.

The commands are as follows on a Cisco router:

access-list 80 permit xx.xx.xx.xx 0.0.0.255                                      

access-list 81 permit xx.xx.xx.xx 0.0.0.255                                        

snmp-server community bMc-pupl!c RO 80                                            

snmp-server community bMc-pr!V4t3 RW 81                                            

snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart  

snmp-server enable traps envmon fan shutdown supply temperature                    

snmp-server enable traps syslog                                                        

snmp-server host xx.xx.xx.xx bMc-pupl!c

¿As I can do the same in the NEXUS?

They help me with this problem please

thanks

prkrishn Thu, 09/27/2012 - 15:21

Hi Samuel

This event is closed. But I will try to answer it. I do see an option for use-acl in my switch.

20-11-7010-N7k-Core-A(config)# snmp-server community public use-acl ?

  WORD  Acl name to filter snmp requests (Max Size 32)

Thanks

-Prashanth

marc.russo Tue, 09/25/2012 - 13:10

Hi,

A quick question about port-channel load balancing algorithms on a Nexus 5020.  Can I change the algorithm on the fly without adverse affects?  I need to change from "source dest IP" to "source dest port" but I am worried about what happens to my port channels while I make this change.

Thanks

Marc

prkrishn Thu, 09/27/2012 - 15:22

Hi Marc

This event is closed but I will try to answer your question. You can make load-balancing change with no impact

Thanks

-Prashanth

mithun.ghosh Mon, 10/01/2012 - 00:49

hi ,

i have two Cisco Nexus  5k's in vpc and and cisco fex 2k and cisco 2960 switch as access layer in downstream.

configured vtp on both 5k's and whenever i am trying to add or delete vlan on N5K server switch the revision number does not increase and the vlans not propagating to the down stream switches.

can you please help me in this case

regards ,

Mithun ghosh

gonzalo.diaz.meza Fri, 04/26/2013 - 04:50

Hi,

I am trying to configure FHRP with two Nexus 5K With no vPC in SVI's but i cant make one of them "passive". It seems to be an issue with incoming hellos because i have connected a 3750 to both of them in the same vlan and i have runn debugs. The test fails with VRRP in SVIs and HSRP in SVIs

This test is with only one 5k and a 3750 with HSRP

Run in N5K

interface Vlan1712

  no shutdown

  description LAN DMZ PrivateCORP

  ip address 10.208.204.2/24

  hsrp 111

    authentication text test

    preempt

    priority 150

    ip 10.208.204.1

N5K_Esperanza_1#

N5K_Esperanza_1#

N5K_Esperanza_1#

N5K_Esperanza_1# sh hsrp brief

                     P indicates configured to preempt.

                     |

Interface   Grp Prio P State    Active addr      Standby addr     Group addr

Vlan1712    111 150  P Active   local            unknown          10.208.204.1    (conf)

N5K_Esperanza_1#

Debug

N5K_Esperanza_1# 2013 Apr 25 21:14:14.812585 hsrp: Vlan1712[111/V4]: Hello out Active pri 150 ip 10.208.204.1

2013 Apr 25 21:14:14.812605 hsrp: Vlan1712[111/V4]: hel 3 hol 10 auth test

2013 Apr 25 21:14:17.812686 hsrp: Vlan1712[111/V4]: Hello out Active pri 150 ip 10.208.204.1

2013 Apr 25 21:14:17.812708 hsrp: Vlan1712[111/V4]: hel 3 hol 10 auth test

2013 Apr 25 21:14:20.812683 hsrp: Vlan1712[111/V4]: Hello out Active pri 150 ip 10.208.204.1

2013 Apr 25 21:14:20.812705 hsrp: Vlan1712[111/V4]: hel 3 hol 10 auth test

Run in 3750 connevted via trunk to N5K

interface Vlan1712

ip address 10.208.204.10 255.255.255.0

standby 111 ip 10.208.204.1

standby 111 priority 190

standby 111 preempt

standby 111 authentication test

end

Debug

SW2_3750_LAB#

Apr 25 18:14:20: HSRP: Vl1712 Grp 111 Hello  out 10.208.204.10 Active  pri 190 vIP 10.208.204.1

Apr 25 18:14:20: HSRP: Vl1712 Grp 111 Hello  in  10.208.204.2 Active  pri 150 vIP 10.208.204.1

Apr 25 18:14:20: HSRP: Vl1712 Grp 111 Hello  out 10.208.204.10 Active  pri 190 vIP 10.208.204.1

SW2_3750_LAB#

It seem to be working OK in 3750 but N5K thinks he is the only one int the LAN and assumes itself as Active

NX-OS is

n5000-uk9.5.1.3.N2.1.bin

Hope you can help

Best Regards

Gonzalo Díaz

thath02@gsc Fri, 10/25/2013 - 07:52

hi

Hope you can assist on nexus 5020.

we had a faulty psu from nexus 2232 Fex. A replacement was done. But fex did not restart after replacement. Had to reseat the psu and then fex restarted. Is this a default behaviour. Thank you.

Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html

Copyright (c) 2002-2012, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html.

Software

BIOS: version 1.5.0

loader: version N/A

kickstart: version 5.1(3)N1(1a)

system: version 5.1(3)N1(1a)

power-seq: Module 1: version v1.2

BIOS compile time: 11/30/10

kickstart image file is: bootflash:///n5000-uk9-kickstart.5.1.3.N1.1a.bin

kickstart compile time: 2/7/2012 23:00:00 [02/08/2012 18:49:30]

system image file is: bootflash:///n5000-uk9.5.1.3.N1.1a.bin

system compile time: 2/7/2012 23:00:00 [02/08/2012 23:44:33]


PID: N2200-PAC-400W

steffenwebb Sun, 11/24/2013 - 05:42

Hi,

I'm using the B22 FEX module in our HP C7000 chassis. Somehow, i'm recieving alot of messages in my log saying : Interface Ethernet xx/1/xx is down (Error disabled. Reason:ekeying triggered). My Fex is connected to a Nexus5596 : 5.2(1)N1(4). I did not see that kind of messages on my old 5010 switches.

What is the reason (all though i suspect servers booting) and can one disable the messages?

Regard Steffen Webb

gnijs Thu, 02/13/2014 - 06:29

Same here. any way of disabling these messages ? even shutdown of interface won't help

Actions

Login or Register to take actions

This Discussion

Posted June 18, 2012 at 8:27 AM
Stats:
Replies:38 Avg. Rating:4.66667
Views:12626 Votes:0
Shares:1

Related Content

Discussions Leaderboard