Cisco Nexus 5596 does not boots up after NX-OS upgrade 5.1.3.N1.1 to 6.0.2.N2.5 [CSCun66310]

[ajay pandey]

Hello Friends,

Just want to save your precious time & effort, while doing Nexus 5596 upgrades so wanted to share some useful info which you can have a look prior to your upgrades of Cisco Nexus 5596 device.

I recently ran into trouble when upgarding one of datacenter switch Nexus 5596 does not boots up and after investigation from Cisco TAC it comes out to be Bug documented below.

https://tools.cisco.com/bugsearch/bug/CSCun66310/?reffering_site=dumpcr

My Upgrade path of device was : 5.1(3)N1(1)  to  6.0(2)N2(5).

Since this upgarde involves upgrade of Bios and Power sequensor, I was having some doubts to get this succesfully upgraded,  But there was more twist involved in this upgrade which I came to know after contacting Cisco TAC for my died Nexus 5596 device and got info that if you have below impacted version in your Nexus 5596 device and you are planning to do Upgrade, then you migh RUN into this serious BUG aftre which device won't come up at all and only Option left is to replace with RMA device.

PID                            Impacted                   Hardware versions      Updated Versions

UCS-FI-6296UP                                            1.0                                       1.1

N5K-C5596UP                                              1.0                                        1.1

N5K-C5596T                                                 1.1                                         1.2

You can check hardware version using below command from your device.

5596# show sprom sup | inc H/W

H/W Version     : 1.1

So please be carefull when planning your Nexus 5596 upgrade and verify above things as this Bug is not present anywhere in Upgrade docuemntations unfortunately.

Hope this will help and save someone's precious Time.

[nsti_0682]

I also hit this bug going from 5.1(3)N1(1) -->5.2(1)N1(4). The switch was bricked after it rebooted.

[IT Department]

I bricked two 5596-UP's last night thanks to this bug.

Why isn't there something in the upgrade documentation stating you must have a specific hardware version 1.1 before upgrading?  When it goes through the pre-upgrade checks why doesn't it check the hardware version and stop this?  

This is $100,000 worth of metal for the scrap heap.  Unacceptable Cisco.

[ajay pandey]

Hi,

Please check the BUG info and upgrade the H/W version, sometimes due to BUG we face such issues.

One more thing, please always follow ISSU path while doing an upgrade, even if you are doing disruptive upgrade, please try to follow ISSU path to avoid any Major issues.

This is what I leaned after my experience with N5K upgrades.

Regards

Ajay

[vveen]

Today "upgraded" our switch (N5k-C5596UP) to from 5.1(2)N1 to 7.0(5) and after reboot no life from the switch either. Even no console response. show sprom sup | inc H/W shows 1:0.

Only thing we could do is RMA the switch. So what must I do with these switches? Have still to do 3 of them.

[Danilo Braga · Accepted solution]

This bug affected me. Before you upgrade the equipament, do the follow:

This is the procedure, there shouldn't be any affectation to apply.
Attached to the case is the dplugin for each NXOS, see the name related to each version.
 
 
Step1
 
-------------------------------
Copy dplugin and update file,  Load Plugin
    Copy the ucd-update.tar to the bootflash
    Copy the debug plugin to the bootflash
    Load the image via the load command
 
 
Note you can access the command line without exiting from the debug plugin via vsh command
 
Example:
 
n5k-1# dir bootflash: | grep debug
    3023275    Jan 23 15:28:48 2013  ucd-update.tar
    283423     Jan 23 15:12:28 2013  nuova-or-dplug-mzg.5.2.1.N1.5.bin
n5k-1# load bootflash:nuova-or-dplug-mzg.5.2.1.N1.5.bin
Loading plugin version 5.2(3a)
###############################################################
  Warning: debug-plugin is for engineering internal use only!
  For security reason, plugin image has been deleted.
###############################################################
Successfully loaded debug-plugin!!!
Linux(debug)#
 
 
 
Step2
 
-------------------------------
After load the dplugin file in NX-OS, please follow the steps to update the TI module.
 
 
Copy the ucd-update.tar from bootflash to tmp directory and untar the file
 
Linux(debug)# cp /bootflash/ucd-update.tar /tmp
 
Linux(debug)# cd /tmp/
 
Linux(debug)# tar -xvf ucd-update.tar
 
ucd.bin
 
Address_91_A.csv
 
Address_92_A.csv
 
Linux(debug)#
 
Step 2
 
----------------------------------
 
Run the ucd.bin file:
 
Linux(debug)# ./ucd.bin
 
 
 
You will see the prompt about updated version 1.1.
 
--------------------------------------------------
 
main: Successfully upgraded TI inst 2 to version 5.
 
New SPROM HW version 1.1. Reload will be required to reflect in "show sprom" output.
 
Linux(debug)# exit

The problem is how to get the plugin. Only Cisco can provide this.

[ryan_bell]

We had a power outage that bounced our 5596's and one out of two would not power back up. Per Cisco, we too hit this bug.

My question for the community, is there ANY way to get back into the failed switch to pull its config? I have a backup of coarse but some changes have been made since the last backup and it would take some time to track them all down since I have quite a few 2k's attached to the 5596's.

Thanks,

Ryan

[korobtsov.v]

I had the same problem, the switch would not boot after poweroff. After a lot of experimenting with turning the power on and off, I managed to boot it.
The starting procedure is as follows:
1. Connect only one power cable and wait about 2-5 minutes.
2. Disconnect the power cable, wait for the switch coolers to stop completely.
3. Reconnect only one power cable! It is very important to run the switch with only one power cable connected.
After that, the switch should boot up, at least in my case it helped.
Hope this helps someone else.