cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
9950
Views
5
Helpful
12
Replies

How to recover from a software failure on the 6120 Fabric Interconnect

Darren Lynn
Cisco Employee
Cisco Employee

Hi all,

Had a software failure while upgrading the fabric interconnects. Does anyone know the process to recover from a software failure.

Ive copied up the kickstart and system images, from the loader prompt ive booted the kickstart image and then loaded the system image but the fabric interconnect still reboots to the "switch(boot)" prompt.

Thanks in Advance

Darren

1 Accepted Solution

Accepted Solutions

You're going to need the UCSM bin file as well. You can download this from CCO.

"ucs-manager-k9.1.3.1c.bin"

1. Once booted to the "switch(boot)" prompt copy all the files to /bootflash


switch(boot)# conf t

    switch(boot)(config)# interface mgmt 0

    switch(boot)(config-if)# ip address [IP address] [Mask]

    switch(boot)(config-if)# no shut

    switch(boot)(config-if)# exit

    switch(boot)(config)# ip default-gateway [Gateway IP]

    switch(boot)(config)# exit

    switch(boot)# copy scp://username@x.x.x.x/images/kickstart-image-version.bin bootflash:

    switch(boot)# copy scp://username@x.x.x.x/images/system-image-version.bin bootflash:

    switch(boot)# copy scp://username@x.x.x.x/images/ucs-manager-version.bin bootflash:

2. Rename ucs-manager-version.bin file to the name that the reserved system image name "nuova-sim-mgmt-nsg.0.1.0.001.bin".

     switch(boot)# copy bootflash:ucs-manager-verison.bin bootflash:nuova-sim-mgmt-nsg.0.1.0.001.bin

3. Reboot

    switch(boot)# exit

4. CTRL-L to break into loader and boot into kickstart and system  images at the same time, space between

    loader> boot kickstart-image-version.bin system-image-version.bin 

5.  When you get the system back up, re-join your cluster (or restore your config) and ensure you set the boot variables as previusly mentioned.

Let me know how you go with the above.

Robert

View solution in original post

12 Replies 12

Robert Burns
Cisco Employee
Cisco Employee

The full restore procedure is found here:

http://www.cisco.com/en/US/partner/docs/unified_computing/ucs/sw/cli/config/guide/1.3.1/CLI_Config_Guide_1_3_1_chapter38.html

The reason your system is still not loading the system software is because the boot variables haven't been set.  Once you've loaded your kickstart and system image from the Loader prompt restored your config (or at least setup basic Mgmt IP access), and open up UCSM (via GUI) and set the boot variables by:

Navigate to the Equipment Tab - Firmware Management - and then "Activate" the Kickstart and System software versions (corresponding to the same versions you currently have running) for both FI's.  Though there will be no actual new firmware activated, this will set your boot variables.

Let me know if you have any issues.

Robert

Hi Robert,

Thanks for the tips, but unfortunately im still stuck on the switch(boot): prompt.

The bootflash: only has 2 files in it, the kickstart and system image.

Each time i get the FI to the boot prompt, i can ping the device, but not access via the GUI. (i dont have a local copy of UCSM installed)

What files should i have in the bootflash: for the FI to correctly load.

below is a log of the FI as it loads the kickstart image from the Loader prompt.

loader> boot bootflash:ucs-6100-k9-kickstart.4.1.3.N2.1.3c.bin                
Booting kickstart image: bootflash:ucs-6100-k9-kickstart.4.1.3.N2.1.3c.bin....
....................................................................Image verif
ication OK

Starting kernel...
Usage: init 0123456SsQqAaBbCcUu
INIT: version 2.85 booting
Starting Nexus5010 POST...
  Executing Mod 1 1 SEEPROM Test......done
  Executing Mod 1 1 GigE Port Test.......done
  Executing Mod 1 1 Inband GigE Test.....done
  Executing Mod 1 1 NVRAM Test....done
  Executing Mod 1 1 PCIE Test..............................done
  Mod 1 1 Post Completed Successfully
  Mod 2 Post Completed Successfully
POST is completed
Creating /callhome..
Mounting /callhome..
Creating /callhome done.
Callhome spool file system init done.
Checking all filesystems..... done.
Warning: switch is starting up with default configuration
Checking NVRAM block device ... done
The startup-config won't be used until the next reboot.
.
Loading system software
INIT: Sending processes the KILL signalkernel daemon...don
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
switch(boot)#

From here i can the load the system image, but it always fails and drops me back to the same prompt, output show below

Uncompressing system image: bootflash:/ucs-6100-k9-system.4.1.3.N2.1.3c.bin

Loading plugin 0: core_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/0, tar_log = /isan/plugin_extract_log/0
Loading plugin 1: eth_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/1, tar_log = /isan/plugin_extract_log/1
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0.0.0 is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0 is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/plugin_exclude.conf is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0.0.0 is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0 is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/plugin_exclude.conf is excluded from linking
Loading plugin 2: fc_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/2, tar_log = /isan/plugin_extract_log/2
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
INIT: Sending processes the TERM signal /boot/lib/ is excluded from lin
INIT: (boot)# l
INIT: Sending processes the KILL signaldaemon...don
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
switch(boot)#

please let me know if you require any further information.

Thanks in Advance

Darren

You're going to need the UCSM bin file as well. You can download this from CCO.

"ucs-manager-k9.1.3.1c.bin"

1. Once booted to the "switch(boot)" prompt copy all the files to /bootflash


switch(boot)# conf t

    switch(boot)(config)# interface mgmt 0

    switch(boot)(config-if)# ip address [IP address] [Mask]

    switch(boot)(config-if)# no shut

    switch(boot)(config-if)# exit

    switch(boot)(config)# ip default-gateway [Gateway IP]

    switch(boot)(config)# exit

    switch(boot)# copy scp://username@x.x.x.x/images/kickstart-image-version.bin bootflash:

    switch(boot)# copy scp://username@x.x.x.x/images/system-image-version.bin bootflash:

    switch(boot)# copy scp://username@x.x.x.x/images/ucs-manager-version.bin bootflash:

2. Rename ucs-manager-version.bin file to the name that the reserved system image name "nuova-sim-mgmt-nsg.0.1.0.001.bin".

     switch(boot)# copy bootflash:ucs-manager-verison.bin bootflash:nuova-sim-mgmt-nsg.0.1.0.001.bin

3. Reboot

    switch(boot)# exit

4. CTRL-L to break into loader and boot into kickstart and system  images at the same time, space between

    loader> boot kickstart-image-version.bin system-image-version.bin 

5.  When you get the system back up, re-join your cluster (or restore your config) and ensure you set the boot variables as previusly mentioned.

Let me know how you go with the above.

Robert

Hi Robert,

The process you outlined has worked in recovering the FI's.. however.. im unable to http or https to the Cluster address after running through the initial express setup on both FI's.. would you have any idea?

I can ping the cluster address

Do i need to activate the new firmware via the CLI?

both FI's look ready to go as per the below output

show system version expand
UCSM:
    Running-Vers: 1.3(1c)
    Activate-Status: Ready

Fabric Interconnect A:
    Running-Kern-Vers: 4.1(3)N2(1.3c)
    Running-Sys-Vers: 4.1(3)N2(1.3c)
    Startup-Kern-Vers:
    Startup-Sys-Vers:
    Act-Kern-Status: Ready
    Act-Sys-Status: Ready
    Bootloader-Vers:   v1.3.0(09/08/09)

Fabric Interconnect B:
    Running-Kern-Vers: 4.1(3)N2(1.3c)
    Running-Sys-Vers: 4.1(3)N2(1.3c)
    Startup-Kern-Vers:
    Startup-Sys-Vers:
    Act-Kern-Status: Ready
    Act-Sys-Status: Ready
    Bootloader-Vers:   v1.3.0(09/08/09)

Cheers

Darren

Robert,

I wanted to thank you for your assistance with this. I also wanted to add the additional processess i went through to rectify the issues

1. Boot the Fabric Interconnect (FI) to the "switch(boot)" prompt. If presented with the "loader" prompt and no kickstart image in the Bootflash: you can boot the switch to the "switch(boot)" prompt by booting from a TFTP server

2. Configure the MGMT 0 interface on the FI

switch(boot)#conf t

switch(boot)(config)#interface mgmt 0

switch(boot)(config-if)#ip address [IP address] [Mask]

switch(boot)(config-if)#no shut

switch(boot)(config-if)#exit

switch(boot)(config)#ip default-gateway [Gateway IP]

switch(boot)(config)#exit

switch(boot)#

3. Copy the Kickstart and System Images to the Bootflash: and also copy the UCS Manager Version to the Bootflash: (i used FTP, but SCP, TFTP, FTP is available)

switch(boot)#copy ftp://username@[FTP IP Address]/kickstart-image-version.bin bootflash:

switch(boot)#copy ftp://username@[FTP IP Address]/system-image-version.bin bootflash:

switch(boot)#copy ftp://username@[FTP IP Address]/ucs-manager-version.bin bootflash:

4. Rename the ucs-manager-version.bin file to the name that the reserved system image uses "nuova-sim-mgmt-nsg.0.1.0.001.bin" (Huge thx to Robert for this step)

switch(boot)#copy bootflash:ucs-manager-version.bin bootflash:nuova-sim-mgmt-nsg.0.1.0.001.bin

5. Reboot the FI

switch(boot)#exit

6. In my experience, the switch rebooted to the loader screen, if it doesnt, you can press CTRL-L (as per Roberts instructions) or CRTL-1 or CRTL-SHIFT-R (as per the configuration guide 1.3(1c). At this prompt boot into the kickstart and system images at the same time. Remember to put a space between the file names

loader> boot kickstart-image-version.bin system-image-version.bin

6. Once the FI reboots the option to start the setup is presented. selct the GUI option and then enter the FI details

IP Address:

Subnet Mask:

Default Gateway:

7. Open a web browser and connect via HTTPS: to the IP address configured in the setup

8. Launch the Express Setup option from the browser and complete the Express Setup.

9. I found from here that i then couldnt launch the UCS manager as after the Express setup was completed, it weas inaccessible. I then connected to the FI via console and could log in. From here i uploaded the UCS-Bundle-Version.bin file

FI-A#scope firmware

FI-A /Firmware # download image ftp://username@[IP Address]/UCS-Bundle-Version.bin file

password: FTP Password

Check the image has complete its download

FI-A /Firmware # show image

10. Open a web browser and connect via HTTPS: to the MGMT 0 IP address abd press Launch.

From here you should be able to enter into the UCS manager and see the FI. You're not done here, the FI still needs to have the newly loaded firmware set as the startup image.

11. Navigate to the Equipment Tab >> Firmware Management and then Activate the Kickstart and System software versions that were uploaded earlier in the process. There is no new firmware to update, this will just set the boot variables for the FI so that it boots correctly and doesnt stop at the "loader" prompt again.

Outside of this, I had a few other issues with multiple FI's that failed and then resetting the cluster information, but this process is in the configuration guides.

Once again a HUGE thanks to Robert for his imput earlier that got me underway.

Cheers

Darren

I had the same problem. The guide work fine for me, except for point 11

"11. Navigate to the Equipment Tab >> Firmware Management and then  Activate the Kickstart and System software versions that were uploaded  earlier in the process. There is no new firmware to update, this will  just set the boot variables for the FI so that it boots correctly and  doesnt stop at the "loader" prompt again."

The Firmware quickstart and system are configured, but I don't know how to configure automatic boot?

Thanks

Cristhian Santelices O.

Cristhian,

Even though the variables may be set in UCSM, activate both FI's again.  This should re-assign the NXOS boot variables and fix your issue.

Have you already done this twice?

If not, let me know.

Regards,

Robert

Hello Robert

This is my screenshot

I don't know how to Activate the Kickstart and System software versions??

The UCS boot to:

Loader Version pr-1.3

loader>

and manually I boot

loader>boot ucs-6100-k9-kickstart.4.1.3.N2.1.3m.bin ucs-6100-k9-system.4.1.3.N2.1.3m.bin

I don't know how to activate boot through GUI or CLI ?

Thanks and regards

Cristhian

Hi Cristhian,

From that screen shot, you need to click on "activate firmware" from the menu bar.


From the next screen you can select the startup versions for each of your Fabric Interconnects.

Cheers

Darren

Hi Darren

I try to activate firmware, but the combox not available to select a "Startup Version" of firmware for UCS and not show any firmware available.

You can help me about
Thanks and regards

Hi,

Can i assume that there is nothing in the drop down box for the start up files for you to select?

Are the files visible on the FI via SSH?

If there isnt, the only thing i could suggest is trying to upload the files again, or to go to the TAC for further assistance

Hi Darren

I uploaded de kickstart and system image via GUI and I have boot the system automatically.

Thanks again

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: