Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. If you'd prefer to explore, try our test area to get started. And see here for current known issues.

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

I'm attempting to upgrade a pair of Nexus 5596's in my lab to the latest code available, and I'm getting an error on one of the switches (the other switch upgraded just fine.

Software

  BIOS:      version 3.6.0

  loader:    version N/A

  kickstart: version 6.0(2)N1(2)

  system:    version 6.0(2)N1(2)

  Power Sequencer Firmware:

             Module 1: version v5.0

  Microcontroller Firmware:        version v1.0.0.2

  SFP uC:    Module 1: v1.1.0.0

  QSFP uC:   Module not detected

  BIOS compile time:       05/09/2012

  kickstart image file is: bootflash:///n5000-uk9-kickstart.6.0.2.N1.2.bin

  kickstart compile time:  3/14/2013 1:00:00 [03/14/2013 08:53:55]

  system image file is:    bootflash:///n5000-uk9.6.0.2.N1.2.bin

  system compile time:     3/14/2013 1:00:00 [03/14/2013 11:28:50]

Hardware

  cisco Nexus 5596 Chassis ("O2 48X10GE/Modular Supervisor")

  Intel(R) Xeon(R) CPU         with 8262944 kB of memory.

  Processor Board ID FOC17360JQS

  Device name: Nexus5596.A2-C06

  bootflash:    2007040 kB

Nexus5596.A2-C06# sh file bootflash:n5000-uk9-kickstart.6.0.2.N2.3.bin md5sum

ff2693db82a73668d0b8a07606fcfe54

Nexus5596.A2-C06# sh file bootflash:n5000-uk9.6.0.2.N2.3.bin md5sum

7cff5f41ffaee8655104af09fa8324d5

I'm getting the following error :

Nexus5596.A2-C06# sh install all impact kickstart bootflash:n5000-uk9-kickstart.6.0.2.N2.3.bin system bootflash:n5000-uk9.6.0.2.N2.3.bin

Verifying image bootflash:/n5000-uk9-kickstart.6.0.2.N2.3.bin for boot variable "kickstart".

[####################] 100% -- SUCCESS

Verifying image bootflash:/n5000-uk9.6.0.2.N2.3.bin for boot variable "system".

[####################] 100% -- SUCCESS

Verifying image type.

[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n5000-uk9.6.0.2.N2.3.bin.

[#                   ]   0% -- FAIL. Return code 0x404F0003 (SRG file not present/cannot be opened).

Pre-upgrade check failed. Return code 0x40930012 (SRG extraction failed).

Does anyone have any idea what could be causing this?  I got the error on one switch, but no errors or issues at all on the other switch.  I've tried restarting the switch to no avail. 

17 REPLIES
VIP Super Bronze

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

It appears you are loading two different images:

good switch

bootflash:///n5000-uk9.6.0.2.N1.2.bin

bad switch

bootflash:/n5000-uk9.6.0.2.N2.3.bin

Can you try the good switch image on the other one?

HTH

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

All of the information posted above is from one switch.  The switch is currently running 6.0.2.N1.2, and I'm trying to upgrade to 6.0.2.N2.3.  The other switch has already upgraded to 6.0.2.N2.3.

I've run and re-run this on this switch many times, and the only thing that differs between this switch and the switch that succeeded is that "verifying image type" takes quite some time on the switch that upgraded properly, and it goes very quickly on the switch that is having problems.

Cisco Employee

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

There is a bug seen on this in previous releases on this behaviour. It looks like your flash disk is full with some /tmp files.

Let's check your disk space with command: " sh system internal flash "

Look at the /var/tmp, is it full ? or do you see another directory full?

Some of the temp folders get cleaned up with a reload, otherwise we can
load a debug-plugin to clean the partition that has been filled.

If we dont want to load a debug plugin, You can use the hidden CLI: show system internal dir /var/tmp and to delete the unwanted files (make sure you don't delete directories). filesys delete /var/tmp/filename(s) Hope this helps.

Cheers,
-amit singh
New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Yeah, I had read about that one, but I'm not seeing it as an issue right now, unless I'm missing something.  Here's the output of the disk space report :

Nexus5596.A2-C06# sh system internal flash

Mount-on                  1K-blocks      Used   Available   Use%  Filesystem

/                            409600     72812      336788     18   /dev/root

/proc                             0         0           0      0   proc

/post                          2048         4        2044      1   none

/var                         409600     72812      336788     18   none

/sys                              0         0           0      0   none

/isan                       1536000    765456      770544     50   none

/var/tmp                     307200       100      307100      1   none

/var/sysmgr                 1024000         4     1023996      1   none

/nxos/tmp                     40960        32       40928      1   none

/var/sysmgr/ftp              409600        68      409532      1   none

/var/sysmgr/ftp/cores        102400         0      102400      0   none

/callhome                     61440         0       61440      0   none

/dev/shm                     524288    284536      239752     55   none

/volatile                    153600         0      153600      0   none

/debug                        20480         0       20480      0   none

/dev/mqueue                       0         0           0      0   none

/debugfs                          0         0           0      0   nodev

/mnt/plog                     49024      1360       47664      3   /dev/mtdblock2

/mnt/cfg/0                   113794      4347      103572      5   /dev/sda5

/mnt/cfg/1                   113794      4347      103572      5   /dev/sda6

/var/sysmgr/startup-cfg      409600      2860      406740      1   none

/dev/pts                          0         0           0      0   devpts

/mnt/pss                     113802      5402      102524      6   /dev/sda4

/bootflash                  1612212    562324      967988     37   /dev/sda3

/proc/bus/usb                     0         0           0      0   none

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Also, here's the contents of var/tmp

Nexus5596.A2-C06# show system internal dir /var/tmp

                                                                ./         680

                                                               ../         260

                                                            mysrg         1984

                                              upgrade_status_file            1

                                                           mylogs          760

                                                   bootloader_ver            0

                                                 m2rib_ascii.3623           34

                                            _clis_parse_tree_.log         2706

                                                fcoe_mgr_init.log        15815

                                                 CSCub19397_debug          237

                                                         fcoe_mgr            0

                                                  fc-pcm-log-4225          757

                                         fabric_start_cfg_mgr.log          761

                                                  issu_helper.log          224

                                                        stp.log.1         4316

                                                         fwm1.out            0

                                                           ntpd_1            0

                                                     ntp_client_1            0

                                                 igmp_restart.log            0

                                                      cfs_mac.log          332

                                                 radius_debug.log          232

                                                  security_stderr           77

                                                  security_stdout            0

                                                    aaa_debug.log          519

                                                          lcc.log          694

                                               security_debug.log          647

                                                    cmp_slot_id.1            0

                                                        mvsh.3260            0

                                                        lockcisco            0

                                                          .flexlm/          60

                                                             logs@          20

                                                  first_setup.log         4745

                                                  boot_uptime.log           56

                                                              vsh/          40

Cisco Employee

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Hi Matt,

I have not tried this ever but have you tried deleting the unwanted files from the var/tmp directory above? Tried deleting few of these except the directories and see if that helps. Also, try to remove some of the files from the flash on the switch, the unwanted ones.

In case that still does not help, tried a different NX-OS version or open a TAC SR.

Cheers,

-amit singh

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Just an update - I tested on two additional 5596's, and those both upgraded fine.

I loaded the images onto a USB drive and actually plugged in to the system (rather than SFTP'ing them), tested again and it still failed.

I reloaded the switch, renamed images to .old, uploaded new copies, tried again, still got the same error.

I reloaded the switch a third time, and lo and behold it worked flawlessly.  I'm not exactly sure why it took three reloads of the swtich to get that to function, but it did.  I'm thankful that this was in a lab environment, but I am a little concerned about putting this into production now.  I don't really want to have to reload a switch 3 times just to get it to take an OS update.

Cisco Employee

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Hi Matt, Many thanks for the update. I read it internally and one of the engineer suggested that by re-booting the box deletes the tmp files in the memory. I would not do that in a production environment as you said. BTW, did you try deleteting the files in tmp directory manaually if that helped at all or not?

Glad it worked for you but yes that needs to be fixed in a better way. I will try raising it with some engineers.

Cheers,

-amit singh

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

I did not remove any files from the /var/tmp directory, as I was not sure what ones were vital, and which ones could be removed.  FYI this is what the flash system looks like after the upgrade.

Nexus5596.A2-C06# sh ver

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html

Copyright (c) 2002-2013, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software are covered under the GNU Public

License. A copy of the license is available at

http://www.gnu.org/licenses/gpl.html.

Software

  BIOS:      version 3.6.0

  loader:    version N/A

  kickstart: version 6.0(2)N2(3)

  system:    version 6.0(2)N2(3)

  Power Sequencer Firmware:

             Module 1: version v5.0

  Microcontroller Firmware:        version v1.0.0.2

  SFP uC:    Module 1: v1.1.0.0

  QSFP uC:   Module not detected

  BIOS compile time:       05/09/2012

  kickstart image file is: bootflash:///n5000-uk9-kickstart.6.0.2.N2.3.bin

  kickstart compile time:  12/17/2013 2:00:00 [12/17/2013 12:52:59]

  system image file is:    bootflash:///n5000-uk9.6.0.2.N2.3.bin

  system compile time:     12/17/2013 2:00:00 [12/17/2013 16:02:31]

Hardware

  cisco Nexus 5596 Chassis ("O2 48X10GE/Modular Supervisor")

  Intel(R) Xeon(R) CPU         with 8253856 kB of memory.

  Processor Board ID FOC17360JQS

  Device name: Nexus5596.A2-C06

  bootflash:    2007040 kB

Kernel uptime is 0 day(s), 0 hour(s), 52 minute(s), 23 second(s)

Last reset at 128479 usecs after  Tue Apr 28 20:57:01 2009

  Reason: Reset due to upgrade

  System version: 6.0(2)N1(2)

  Service:

plugin

  Core Plugin, Ethernet Plugin, Fc Plugin

Nexus5596.A2-C06# sh system internal flash

Mount-on                  1K-blocks      Used   Available   Use%  Filesystem

/                            409600     74520      335080     19   /dev/root

/proc                             0         0           0      0   proc

/sys                              0         0           0      0   none

/isan                       1536000    763412      772588     50   none

/var/tmp                     307200        92      307108      1   none

/var/sysmgr                 1024000     24152      999848      3   none

/nxos/tmp                     40960        52       40908      1   none

/var/sysmgr/ftp              409600        72      409528      1   none

/var/sysmgr/ftp/cores        102400         0      102400      0   none

/callhome                     61440         0       61440      0   none

/dev/shm                     524288    285464      238824     55   none

/volatile                    153600         0      153600      0   none

/debug                        20480         0       20480      0   none

/dev/mqueue                       0         0           0      0   none

/debugfs                          0         0           0      0   nodev

/mnt/plog                     49024      1380       47644      3   /dev/mtdblock

2

/mnt/cfg/0                   113794      4304      103615      4   /dev/sda5

/mnt/cfg/1                   113794      4304      103615      4   /dev/sda6

/var/sysmgr/startup-cfg      409600      2856      406744      1   none

/dev/pts                          0         0           0      0   devpts

/mnt/pss                     113802      5833      102093      6   /dev/sda4

/bootflash                  1612212    562348      967964     37   /dev/sda3

/usbslot1                    982016    258816      723200     27   /dev/sdb1

/proc/bus/usb                     0         0           0      0   none

/cgroup                           0         0           0      0   vdccontrol

Nexus5596.A2-C06# sh sys internal dir /var/tmp

                                                                ./        1020

                                                               ../         260

                                         system.bootstrap.cfg.log            0

                                       sysmgr_startcfg_flash.lock            0

                                       dhcp_snoop_startup_cfg.trm            0

                                       dhcp_snoop_startup_cfg.dat            0

                                       dhcp_snoop_startup_cfg.map            0

                                       dhcp_snoop_startup_cfg.pss          848

                                              upgrade_status_file            1

                                              install_all_cli.log          244

                                            _clis_parse_tree_.log         2706

                                                        stp.log.1         3907

                                                           ntpd_1            0

                                         fabric_start_cfg_mgr.log          359

                                                     ntp_client_1            0

                                          fs_upg_fspf_hs_time.txt           10

                                                         fwm1.out            0

                                                 radius_debug.log          232

                                                      cfs_mac.log          360

                                                         fcoe_mgr            0

                                                    aaa_debug.log          407

                                                  security_stderr           77

                                                  security_stdout            0

                                                 igmp_restart.log            0

                                          fs_upg_lacp_hs_time.txt           10

                                                 CSCub19397_debug          240

                                               security_debug.log          656

                                          fs_upg_ctrl_up_time.txt           10

                                                  issu_helper.log         1109

                                                          lcc.log          694

                                                  fc-pcm-log-3378          829

                                                libusd_0x30b.3343            0

                                                libusd_0x2fd.3339            0

                                                libusd_0x2e7.3344            0

                                                    cmp_slot_id.1            0

                                                        mvsh.3326            0

                                                libusd_0x2a4.3314            0

                                                libusd_0x323.3312            0

                                                        lockcisco            0

                                                          .flexlm/          60

                                       dhcp_snoop_running_cfg.trm            0

                                       dhcp_snoop_running_cfg.dat            0

                                       dhcp_snoop_running_cfg.map            0

                                       dhcp_snoop_running_cfg.pss          848

                                                assoc_mgr_cnv.log          366

                                          infraconvertlibpsscheck            0

                                                infraconvertcheck            0

                                                             logs@          20

                                                  first_setup.log         3050

                                                  boot_uptime.log           56

                                                              vsh/          40

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

So, I'm now upgrading the 5548's that I have in the same lab, and I'm getting the same error.  I'd like to try nuking some of the files in the /var/tmp folder, but I'm trying to figure out what ones to delete.  Current file structure looks like this :

Nexus5548.A-C04# sh sys int dir /var/tmp

                                                                ./         820

                                                               ../         260

                                                            mysrg         1984

                                              upgrade_status_file            1

                                           util_cli_history_admin            0

                                                   cfg_status.log            7

                                                          exclude            0

                                       sysmgr_startcfg_flash.lock            0

                                                 m2rib_ascii.3996           68

                                            _clis_parse_tree_.log         2706

                                                fcoe_mgr_init.log        15815

                                                 CSCub19397_debug          237

                                                         fcoe_mgr            0

                                                  fc-pcm-log-4917          757

                                         fabric_start_cfg_mgr.log         1496

                                                  issu_helper.log          224

                                                        stp.log.1         4316

                                                         fwm1.out            0

                                                           ntpd_1            0

                                                      cfs_mac.log          332

                                                     ntp_client_1            0

                                                 igmp_restart.log            0

                                                 radius_debug.log          232

                                                  security_stderr           77

                                                  security_stdout            0

                                               security_debug.log          647

                                                    aaa_debug.log            0

                                                libusd_0x30b.3252            0

                                                          lcc.log          694

                                                libusd_0x2fd.3248            0

                                                libusd_0x2e7.3253            0

                                                        mvsh.3237            0

                                                libusd_0x2a4.3226            0

                                                libusd_0x323.3224            0

                                                    cmp_slot_id.1            0

                                                        lockcisco            0

                                                          .flexlm/          60

                                                             logs@          20

                                                  first_setup.log            0

                                                  boot_uptime.log           56

                                                              vsh/          40

Cisco Employee

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Hi Matt,

Are you upgrading to the same NX-OS? What if you try to use a different NX-OS? Do you see the same symptoms?

Try deleting the .log files and see if this helps. Do not delete any directories.

Cheers,

-amit singh

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

I am upgrading to 6.0(2)N2(3) - only difference is that I'm upgrading from 6.0(2)N2(2) rather than 6.0(2)N1(2).  I'll delete logs and see what we see.

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Deleted all .log files, still having the exact same error.  Here's the current directory listing :

Nexus5548.A-C04#  sh sys int dir /var/tmp

                                                                ./         860

                                                               ../         260

                                                            mysrg         1984

                                                           mylogs          330

                                                   bootloader_ver            0

                                              upgrade_status_file            1

                                           util_cli_history_admin            0

                                                   cfg_status.log            0

                                                          exclude            0

                                       sysmgr_startcfg_flash.lock            0

                                                 m2rib_ascii.3996           68

                                            _clis_parse_tree_.log            0

                                                fcoe_mgr_init.log            0

                                                 CSCub19397_debug          237

                                                         fcoe_mgr            0

                                                  fc-pcm-log-4917          757

                                         fabric_start_cfg_mgr.log            0

                                                  issu_helper.log            0

                                                        stp.log.1            0

                                                         fwm1.out            0

                                                           ntpd_1            0

                                                      cfs_mac.log            0

                                                     ntp_client_1            0

                                                 igmp_restart.log            0

                                                 radius_debug.log            0

                                                  security_stderr           77

                                                  security_stdout            0

                                               security_debug.log            0

                                                    aaa_debug.log            0

                                                libusd_0x30b.3252            0

                                                          lcc.log            0

                                                libusd_0x2fd.3248            0

                                                libusd_0x2e7.3253            0

                                                        mvsh.3237            0

                                                libusd_0x2a4.3226            0

                                                libusd_0x323.3224            0

                                                    cmp_slot_id.1            0

                                                        lockcisco            0

                                                          .flexlm/          60

                                                             logs@          20

                                                  first_setup.log            0

                                                  boot_uptime.log            0

                                                              vsh/          40

Cisco Employee

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Just an update on this. We are working with TAC and engineering on this.

Cheers,

-amit singh

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Excellent - I opened a tac case and referenced this thread.  TAC case that is open is 628898063

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

Matt, have you heard anything new on this? I'm getting the exact same error "SRG file not present/cannot be opened".

We are on 6.0.2.N2.1.

New Member

Nexus 5596 Upgrade to 6.0(2)N2(3) failure.

I worked through TAC, and they believe that there was an open Telnet session locking a file in the bootflash.  I'm not sure I totally agree, and it's happened twice on all 6 5500's that I have in our lab.  Check and see if there are any open SSH/Telnet sessions and if possible, kill them.

It seems as though whatever has the lock will release itself after some period of time, and I've not found a single solution short of rebooting the switch to clear the issue directly.

2002
Views
10
Helpful
17
Replies