Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Matt Breitbach · ‎01-17-2014

I'm attempting to upgrade a pair of Nexus 5596's in my lab to the latest code available, and I'm getting an error on one of the switches (the other switch upgraded just fine.

Software

BIOS: version 3.6.0

loader: version N/A

kickstart: version 6.0(2)N1(2)

system: version 6.0(2)N1(2)

Power Sequencer Firmware:

Module 1: version v5.0

Microcontroller Firmware: version v1.0.0.2

SFP uC: Module 1: v1.1.0.0

QSFP uC: Module not detected

BIOS compile time: 05/09/2012

kickstart image file is: bootflash:///n5000-uk9-kickstart.6.0.2.N1.2.bin

kickstart compile time: 3/14/2013 1:00:00 [03/14/2013 08:53:55]

system image file is: bootflash:///n5000-uk9.6.0.2.N1.2.bin

system compile time: 3/14/2013 1:00:00 [03/14/2013 11:28:50]

Hardware

cisco Nexus 5596 Chassis ("O2 48X10GE/Modular Supervisor")

Intel(R) Xeon(R) CPU with 8262944 kB of memory.

Processor Board ID FOC17360JQS

Device name: Nexus5596.A2-C06

bootflash: 2007040 kB

Nexus5596.A2-C06# sh file bootflash:n5000-uk9-kickstart.6.0.2.N2.3.bin md5sum

ff2693db82a73668d0b8a07606fcfe54

Nexus5596.A2-C06# sh file bootflash:n5000-uk9.6.0.2.N2.3.bin md5sum

7cff5f41ffaee8655104af09fa8324d5

I'm getting the following error :

Nexus5596.A2-C06# sh install all impact kickstart bootflash:n5000-uk9-kickstart.6.0.2.N2.3.bin system bootflash:n5000-uk9.6.0.2.N2.3.bin

Verifying image bootflash:/n5000-uk9-kickstart.6.0.2.N2.3.bin for boot variable "kickstart".

[####################] 100% -- SUCCESS

Verifying image bootflash:/n5000-uk9.6.0.2.N2.3.bin for boot variable "system".

[####################] 100% -- SUCCESS

Verifying image type.

[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n5000-uk9.6.0.2.N2.3.bin.

[# ] 0% -- FAIL. Return code 0x404F0003 (SRG file not present/cannot be opened).

Pre-upgrade check failed. Return code 0x40930012 (SRG extraction failed).

Does anyone have any idea what could be causing this? I got the error on one switch, but no errors or issues at all on the other switch. I've tried restarting the switch to no avail.

Reza Sharifi · ‎01-18-2014

It appears you are loading two different images:

good switch

bootflash:///n5000-uk9.6.0.2.N1.2.bin

bad switch

bootflash:/n5000-uk9.6.0.2.N2.3.bin

Can you try the good switch image on the other one?

HTH

Matt Breitbach · ‎01-18-2014

All of the information posted above is from one switch. The switch is currently running 6.0.2.N1.2, and I'm trying to upgrade to 6.0.2.N2.3. The other switch has already upgraded to 6.0.2.N2.3.

I've run and re-run this on this switch many times, and the only thing that differs between this switch and the switch that succeeded is that "verifying image type" takes quite some time on the switch that upgraded properly, and it goes very quickly on the switch that is having problems.

Amit Singh · ‎01-18-2014

There is a bug seen on this in previous releases on this behaviour. It looks like your flash disk is full with some /tmp files.

Let's check your disk space with command: " sh system internal flash "

Look at the /var/tmp, is it full ? or do you see another directory full?

Some of the temp folders get cleaned up with a reload, otherwise we can
load a debug-plugin to clean the partition that has been filled.

If we dont want to load a debug plugin, You can use the hidden CLI:


show system internal dir /var/tmp


and to delete the unwanted files (make sure you don't delete directories).


filesys delete /var/tmp/filename(s)

Hope this helps.

Cheers,
-amit singh

Matt Breitbach · ‎01-18-2014

Yeah, I had read about that one, but I'm not seeing it as an issue right now, unless I'm missing something. Here's the output of the disk space report :

Nexus5596.A2-C06# sh system internal flash

Mount-on 1K-blocks Used Available Use% Filesystem

/ 409600 72812 336788 18 /dev/root

/proc 0 0 0 0 proc

/post 2048 4 2044 1 none

/var 409600 72812 336788 18 none

/sys 0 0 0 0 none

/isan 1536000 765456 770544 50 none

/var/tmp 307200 100 307100 1 none

/var/sysmgr 1024000 4 1023996 1 none

/nxos/tmp 40960 32 40928 1 none

/var/sysmgr/ftp 409600 68 409532 1 none

/var/sysmgr/ftp/cores 102400 0 102400 0 none

/callhome 61440 0 61440 0 none

/dev/shm 524288 284536 239752 55 none

/volatile 153600 0 153600 0 none

/debug 20480 0 20480 0 none

/dev/mqueue 0 0 0 0 none

/debugfs 0 0 0 0 nodev

/mnt/plog 49024 1360 47664 3 /dev/mtdblock2

/mnt/cfg/0 113794 4347 103572 5 /dev/sda5

/mnt/cfg/1 113794 4347 103572 5 /dev/sda6

/var/sysmgr/startup-cfg 409600 2860 406740 1 none

/dev/pts 0 0 0 0 devpts

/mnt/pss 113802 5402 102524 6 /dev/sda4

/bootflash 1612212 562324 967988 37 /dev/sda3

/proc/bus/usb 0 0 0 0 none

Matt Breitbach · ‎01-18-2014

Also, here's the contents of var/tmp

Nexus5596.A2-C06# show system internal dir /var/tmp

./ 680

../ 260

mysrg 1984

upgrade_status_file 1

mylogs 760

bootloader_ver 0

m2rib_ascii.3623 34

_clis_parse_tree_.log 2706

fcoe_mgr_init.log 15815

CSCub19397_debug 237

fcoe_mgr 0

fc-pcm-log-4225 757

fabric_start_cfg_mgr.log 761

issu_helper.log 224

stp.log.1 4316

fwm1.out 0

ntpd_1 0

ntp_client_1 0

igmp_restart.log 0

cfs_mac.log 332

radius_debug.log 232

security_stderr 77

security_stdout 0

aaa_debug.log 519

lcc.log 694

security_debug.log 647

cmp_slot_id.1 0

mvsh.3260 0

lockcisco 0

.flexlm/ 60

logs@ 20

first_setup.log 4745

boot_uptime.log 56

vsh/ 40

Amit Singh · ‎01-19-2014

Hi Matt,

I have not tried this ever but have you tried deleting the unwanted files from the var/tmp directory above? Tried deleting few of these except the directories and see if that helps. Also, try to remove some of the files from the flash on the switch, the unwanted ones.

In case that still does not help, tried a different NX-OS version or open a TAC SR.

Cheers,

-amit singh

Matt Breitbach · ‎01-20-2014

Just an update - I tested on two additional 5596's, and those both upgraded fine.

I loaded the images onto a USB drive and actually plugged in to the system (rather than SFTP'ing them), tested again and it still failed.

I reloaded the switch, renamed images to .old, uploaded new copies, tried again, still got the same error.

I reloaded the switch a third time, and lo and behold it worked flawlessly. I'm not exactly sure why it took three reloads of the swtich to get that to function, but it did. I'm thankful that this was in a lab environment, but I am a little concerned about putting this into production now. I don't really want to have to reload a switch 3 times just to get it to take an OS update.

Amit Singh · ‎01-20-2014

Hi Matt, Many thanks for the update. I read it internally and one of the engineer suggested that by re-booting the box deletes the tmp files in the memory. I would not do that in a production environment as you said. BTW, did you try deleteting the files in tmp directory manaually if that helped at all or not?

Glad it worked for you but yes that needs to be fixed in a better way. I will try raising it with some engineers.

Cheers,

-amit singh

Matt Breitbach · ‎01-20-2014

I did not remove any files from the /var/tmp directory, as I was not sure what ones were vital, and which ones could be removed. FYI this is what the flash system looks like after the upgrade.

Nexus5596.A2-C06# sh ver

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public

License. A copy of the license is available at

http://www.gnu.org/licenses/gpl.html.

Software

BIOS: version 3.6.0

loader: version N/A

kickstart: version 6.0(2)N2(3)

system: version 6.0(2)N2(3)

Power Sequencer Firmware:

Module 1: version v5.0

Microcontroller Firmware: version v1.0.0.2

SFP uC: Module 1: v1.1.0.0

QSFP uC: Module not detected

BIOS compile time: 05/09/2012

kickstart image file is: bootflash:///n5000-uk9-kickstart.6.0.2.N2.3.bin

kickstart compile time: 12/17/2013 2:00:00 [12/17/2013 12:52:59]

system image file is: bootflash:///n5000-uk9.6.0.2.N2.3.bin

system compile time: 12/17/2013 2:00:00 [12/17/2013 16:02:31]

Hardware

cisco Nexus 5596 Chassis ("O2 48X10GE/Modular Supervisor")

Intel(R) Xeon(R) CPU with 8253856 kB of memory.

Processor Board ID FOC17360JQS

Device name: Nexus5596.A2-C06

bootflash: 2007040 kB

Kernel uptime is 0 day(s), 0 hour(s), 52 minute(s), 23 second(s)

Last reset at 128479 usecs after Tue Apr 28 20:57:01 2009

Reason: Reset due to upgrade

System version: 6.0(2)N1(2)

Service:

plugin

Core Plugin, Ethernet Plugin, Fc Plugin

Nexus5596.A2-C06# sh system internal flash

Mount-on 1K-blocks Used Available Use% Filesystem

/ 409600 74520 335080 19 /dev/root

/proc 0 0 0 0 proc

/sys 0 0 0 0 none

/isan 1536000 763412 772588 50 none

/var/tmp 307200 92 307108 1 none

/var/sysmgr 1024000 24152 999848 3 none

/nxos/tmp 40960 52 40908 1 none

/var/sysmgr/ftp 409600 72 409528 1 none

/var/sysmgr/ftp/cores 102400 0 102400 0 none

/callhome 61440 0 61440 0 none

/dev/shm 524288 285464 238824 55 none

/volatile 153600 0 153600 0 none

/debug 20480 0 20480 0 none

/dev/mqueue 0 0 0 0 none

/debugfs 0 0 0 0 nodev

/mnt/plog 49024 1380 47644 3 /dev/mtdblock

2

/mnt/cfg/0 113794 4304 103615 4 /dev/sda5

/mnt/cfg/1 113794 4304 103615 4 /dev/sda6

/var/sysmgr/startup-cfg 409600 2856 406744 1 none

/dev/pts 0 0 0 0 devpts

/mnt/pss 113802 5833 102093 6 /dev/sda4

/bootflash 1612212 562348 967964 37 /dev/sda3

/usbslot1 982016 258816 723200 27 /dev/sdb1

/proc/bus/usb 0 0 0 0 none

/cgroup 0 0 0 0 vdccontrol

Nexus5596.A2-C06# sh sys internal dir /var/tmp

./ 1020

../ 260

system.bootstrap.cfg.log 0

sysmgr_startcfg_flash.lock 0

dhcp_snoop_startup_cfg.trm 0

dhcp_snoop_startup_cfg.dat 0

dhcp_snoop_startup_cfg.map 0

dhcp_snoop_startup_cfg.pss 848

upgrade_status_file 1

install_all_cli.log 244

_clis_parse_tree_.log 2706

stp.log.1 3907

ntpd_1 0

fabric_start_cfg_mgr.log 359

ntp_client_1 0

fs_upg_fspf_hs_time.txt 10

fwm1.out 0

radius_debug.log 232

cfs_mac.log 360

fcoe_mgr 0

aaa_debug.log 407

security_stderr 77

security_stdout 0

igmp_restart.log 0

fs_upg_lacp_hs_time.txt 10

CSCub19397_debug 240

security_debug.log 656

fs_upg_ctrl_up_time.txt 10

issu_helper.log 1109

lcc.log 694

fc-pcm-log-3378 829

libusd_0x30b.3343 0

libusd_0x2fd.3339 0

libusd_0x2e7.3344 0

cmp_slot_id.1 0

mvsh.3326 0

libusd_0x2a4.3314 0

libusd_0x323.3312 0

lockcisco 0

.flexlm/ 60

dhcp_snoop_running_cfg.trm 0

dhcp_snoop_running_cfg.dat 0

dhcp_snoop_running_cfg.map 0

dhcp_snoop_running_cfg.pss 848

assoc_mgr_cnv.log 366

infraconvertlibpsscheck 0

infraconvertcheck 0

logs@ 20

first_setup.log 3050

boot_uptime.log 56

vsh/ 40

Matt Breitbach · ‎01-21-2014

So, I'm now upgrading the 5548's that I have in the same lab, and I'm getting the same error. I'd like to try nuking some of the files in the /var/tmp folder, but I'm trying to figure out what ones to delete. Current file structure looks like this :

Nexus5548.A-C04# sh sys int dir /var/tmp

./ 820

../ 260

mysrg 1984

upgrade_status_file 1

util_cli_history_admin 0

cfg_status.log 7

exclude 0

sysmgr_startcfg_flash.lock 0

m2rib_ascii.3996 68

_clis_parse_tree_.log 2706

fcoe_mgr_init.log 15815

CSCub19397_debug 237

fcoe_mgr 0

fc-pcm-log-4917 757

fabric_start_cfg_mgr.log 1496

issu_helper.log 224

stp.log.1 4316

fwm1.out 0

ntpd_1 0

cfs_mac.log 332

ntp_client_1 0

igmp_restart.log 0

radius_debug.log 232

security_stderr 77

security_stdout 0

security_debug.log 647

aaa_debug.log 0

libusd_0x30b.3252 0

lcc.log 694

libusd_0x2fd.3248 0

libusd_0x2e7.3253 0

mvsh.3237 0

libusd_0x2a4.3226 0

libusd_0x323.3224 0

cmp_slot_id.1 0

lockcisco 0

.flexlm/ 60

logs@ 20

first_setup.log 0

boot_uptime.log 56

vsh/ 40

Amit Singh · ‎01-21-2014

Hi Matt,

Are you upgrading to the same NX-OS? What if you try to use a different NX-OS? Do you see the same symptoms?

Try deleting the .log files and see if this helps. Do not delete any directories.

Cheers,

-amit singh

Matt Breitbach · ‎01-21-2014

I am upgrading to 6.0(2)N2(3) - only difference is that I'm upgrading from 6.0(2)N2(2) rather than 6.0(2)N1(2). I'll delete logs and see what we see.

Matt Breitbach · ‎01-21-2014

Deleted all .log files, still having the exact same error. Here's the current directory listing :

Nexus5548.A-C04# sh sys int dir /var/tmp

./ 860

../ 260

mysrg 1984

mylogs 330

bootloader_ver 0

upgrade_status_file 1

util_cli_history_admin 0

cfg_status.log 0

exclude 0

sysmgr_startcfg_flash.lock 0

m2rib_ascii.3996 68

_clis_parse_tree_.log 0

fcoe_mgr_init.log 0

CSCub19397_debug 237

fcoe_mgr 0

fc-pcm-log-4917 757

fabric_start_cfg_mgr.log 0

issu_helper.log 0

stp.log.1 0

fwm1.out 0

ntpd_1 0

cfs_mac.log 0

ntp_client_1 0

igmp_restart.log 0

radius_debug.log 0

security_stderr 77

security_stdout 0

security_debug.log 0

aaa_debug.log 0

libusd_0x30b.3252 0

lcc.log 0

libusd_0x2fd.3248 0

libusd_0x2e7.3253 0

mvsh.3237 0

libusd_0x2a4.3226 0

libusd_0x323.3224 0

cmp_slot_id.1 0

lockcisco 0

.flexlm/ 60

logs@ 20

first_setup.log 0

boot_uptime.log 0

vsh/ 40

Amit Singh · ‎01-23-2014

Just an update on this. We are working with TAC and engineering on this.

Cheers,

-amit singh