×

Warning message

  • Cisco Support Forums is in Read Only mode while the site is being migrated.
  • Cisco Support Forums is in Read Only mode while the site is being migrated.

Cisco 4500X IOS upgrade through ISSU

Unanswered Question
Apr 17th, 2014
User Badges:

Hi,

I am having 2 number of cisco 4500x switch and configured with VSS

so one switch is active and another switch is standby.

I am panning to upgrade IOS through ISSU

i read in document that it required auto boot enable in switch.

My switch current Configuration register = 0x2101

do i need to change config register or this will ok. If need to change then what will be auto boot and after IOS upgrade do i need to change it again.

Please help....

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (4 ratings)
Loading.
Leo Laohoo Thu, 04/17/2014 - 17:07
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Configuration Register value of 0x2101 is, by default, the setting when the appliance or supervisor is shipped out.  The last octet of "1" basically tells the appliance to IGNORE the boot variable string and boot the first valid IOS (from top to bottom) found in the bootflash.  

The easiest and fastest way to do an IOS upgrade on a VSS pair is very simple.  

 

1.  Copy the IOS from the TFTP server to the active unit using the command "copy tftp://<TFTP IP address>/IOS_filename.bin bootflash:";

2.  Copy the IOS from the TFTP server to the standby unit using the command "copy tftp://<TFTP IP address>/IOS_filename.bin slavebootflash:";

3.  Remove the old boot variable string:  no boot system flash

4.  Change to the new boot variable string:  boot system flash bootflash:IOS_filename.bin

5.  Change the config-registry value to 0x2102:  config-registry 0x2102

6.  Reboot. 

 

Now be aware that, like any 4500 supervisor card and 6500 supervisor card, if you change the config-registry to 0x2102 and you DO NOT HAVE a boot variable string (or a valid one), the your switch will reboot into ROMmon.  So please take good care in entering the right boot variable string.

 

Hope this helps.  

 

Please don't forget to rate our useful posts.  

Leo Laohoo Thu, 04/17/2014 - 21:16
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

I would like to present to you Option 2.  

 

Now, if you do not want to change the config-registry value to 0x2102, this means that you would like config-registry value of 0x2101, there's another option around this.  

 

And the process is almost similar.  

 

1.  Copy the IOS from the TFTP server to the active unit using the command "copy tftp://<TFTP IP address>/IOS_filename.bin bootflash:";

2.  Copy the IOS from the TFTP server to the standby unit using the command "copy tftp://<TFTP IP address>/IOS_filename.bin slavebootflash:";

3.  RENAME the old IOS in both units:  

Active:  rename bootflash:OLD_IOS_filename.bin bootflash:OLD_IOS_filename.bin.BAK

Standby:  rename slavebootflash:OLD_IOS_filename.bin slavebootflash:OLD_IOS_filename.bin.BAK

4.  Reboot.

 

You need to rename the old IOS because if you don't then there's a chance that one of the VSS pairs will boot the OLD IOS if the file is found to be above the new IOS file.

Tarunkumar Vyas Fri, 04/18/2014 - 01:49
User Badges:

Hi Leo,

while giving below command on active switch.

Switch# issu loadversion [active-slot] active-image-new [standby-slot] standby-image-new

I am getting error "Active config-register does not have 0x0002 as the low order nible"

Leo Laohoo Fri, 04/18/2014 - 18:10
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Tarun, 

 

I cannot comment ISSU because I've attempted to use this process in the past and they never worked.  

 

The process I've detailed above is a known "workaround" to stop either one or two of your supervisors going into a boot-error-crash loop.  

Tarunkumar Vyas Fri, 04/18/2014 - 23:23
User Badges:

Thnx Leo, I have upgraded IOS individually in both of switch and working fine.

 

Leo Laohoo Sat, 04/19/2014 - 17:11
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Happy to hear it's working Tarun. 

 

Tell us, which option did you use?  Did you use the Option 1 (change the config-register to 0x2102) or use Option 2 (maintain the config-register of 0x2101)?

Leo Laohoo Tue, 04/22/2014 - 14:50
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Ok, thanks for the update Tarun.

afsharki2 Tue, 05/10/2016 - 10:16
User Badges:

Hey Leo,  I was thinking about doing our 4500x vss pair upgrade through ISSU.  Did you say that you were having trouble with ISSU, so you did the upgrade on each switch individually?  Did you take down the network?

Leo Laohoo Tue, 05/10/2016 - 14:30
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Did you take down the network?

Whichever process used, it will ALWAYS take down the network.  So the answer to the question is, yes, I took down the network. 

Did you say that you were having trouble with ISSU, so you did the upgrade on each switch individually?

Each switch pair was upgraded individually and both units were reloaded simultaneously.

afsharki2 Tue, 05/10/2016 - 14:37
User Badges:

Really??  Why would ISSU state in their documentation that "as long as SSO and NSF are enabled, you don't have to take down the network and you can individually perform the updates"



Leo Laohoo Tue, 05/10/2016 - 15:00
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

you can individually perform the updates

This is correct.  Anyone can copy the IOS into the box and, in an ideal condition, without an outage.  However, when ISSU is performed the secondary line card reboots and the primary line card stays online.  When this occurs, single-homed devices or devices with incorrectly configured etherchannel will fail.  

When the secondary line card goes online, the primary will failover.  During this failover, the same scenario above will be repeated again, i.e. single-homed devices or devices with incorrectly configured etherchannel will fail.

Some might say, sure they have theirs upgraded without any impact, however, I am not taking any chances.  The first time I did an ISSU, and we followed the procedure to the letter, my VSS pair went into a boot-error-crash loop and it took out our entire wireless network.  The only way to stop this was to pull out the supervisor card or power down the chassis.  I ain't repeating that nightmare again. 

afsharki2 Tue, 05/10/2016 - 16:10
User Badges:

Man, that sucks!  I have given everyone the notice that this might potentially happen, so I'm at least anticipating it.  I'm doing the upgrade for 3 Distribution switches that are VSS pairs.  Every single edge switch has one uplink on each of the VSS pairs, so everything is fully redundant.  Our WLC is connected to another 4500x that connects to all our Distribution switches...fully redundant also.  Fingers x'd

Leo Laohoo Tue, 05/10/2016 - 17:42
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Man, that sucks!

We had proper change control in place and had anticipated a "moderate" outage.  Took all of us by surprise.  

So from now on, anyone (in our office) who utters the abbreviation FSU, eFSU and ISSU will be greeted with a well-aimed Citrix rocket and followed by threats of bodily harm ... using toothpicks & pitchforks.

paul driver Wed, 05/11/2016 - 05:36
User Badges:
  • Green, 3000 points or more

Hello

We have a issue with the 4500's at present regards a previous planned power outage , that when vss cam back online active/standby mode etc..

The inter-vlan routing wasn't working, even though the advertised subnets were being propagated into the wan bgp rtrs., and could be seen through our entire estate, but they were not reachable even the SVI weren't not responding

The VSL was up and seemed okay and all virtual switching logs seemed okay, its only when I shutdown each individual SVI and re-enabled it that vlan was reachable.

Even after full vss shelf reload still the same issue.......I have not implemented dual-active detection at this time but given the way this problem is occurring I don't think it would be beneficial anyway-  I am now thinking a possible ios bug - 

Ill keep you posted after the cisco tac review

res

Paul

Leo Laohoo Wed, 05/11/2016 - 15:18
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Ill keep you posted after the cisco tac review

Paul, 

I'm keen to know the result or findings from the TAC review. 

By the way, what IOS is this VSS pair running on?

afsharki2 Mon, 05/16/2016 - 11:54
User Badges:

Just wanted to share my experience with ISSU on Friday.  Everything went relativity well, and since my network is substantially smaller than yours, I had a lot less checks and things to worry about.

I checked to see if we are fully multi-homed to our access layer, as you were also telling me.  check!

 The distribution upgrade went well, but our services switch which is trunked, multi-homed to our WLC's did not go as well.  Took down the wireless for about 15 min.  The APs dissociated than re-associated and were back online in about 15 min.  We have about 1500 APs.  


There is a new caveat with ISSU, that was not mentioned anywhere in my predecessors documentation or online Cisco docs:

In order to begin the loadversion process, you must take the image path out of the "boot system flash" statement on your 4500x; it cannot be pointing to the new image until after you have finished the upgrade process on the VSS.  I had to open a TAC case because I kept getting "boot variable is invalid" message.  Documents online just said to do a redudnacy force-switchover to remedy that problem and that did not work.  Other than the wireless going down for about 15 min, everything else stayed online.  

Now, I have to try to find out why the wireless went down: our WLC are configured with all our T1/x/x interfaces to go the primary WLC and all the T2/x/x interfaces to go the standby WLC on our service switch vss pair that I upgraded on Friday.

Leo Laohoo Mon, 05/16/2016 - 13:31
User Badges:
  • Super Gold, 25000 points or more
  • Hall of Fame,

    The Hall of Fame designation is a lifetime achievement award based on significant overall achievements in the community. 

  • Cisco Designated VIP,

    2017 LAN, Wireless

Now, I have to try to find out why the wireless went down

I'm a wireless guy too.  Happy to help troubleshoot.  Maybe the APs went to another controller and this controller have different firmwares? 


Glad to see the upgrade went well.  

Hello
Could you solve this issue. It looks like Bug CSCur09885

Description

Upon VSS-standby reload, the route entries corresponding the L2 Multichassis Ether Channel are not installed at VSS-standby. Partial packet drops are observed.

Symptom:
For IP packets passing through the L2 Multichassis Ether Channel, partial packet drops are noticed.

Conditions:
On a VSS, L2 Multichassis Ether Channel, must be configured with trunk mode. The issue is seen after the VSS-standby switch reloads and becomes part of the VSS again.

Workaround:
Issue "shut" followed by "no shut" on the SVI interface on which 'packet drops' are being noticed.

 
I have also problems after booting both of the Switches individualy with "redundancy reload peer " and after this switch was up again and in Standby hot state, booting active one with "redundancy force-switchover"
This will reload the active unit and force switchover to standby[confirm]Preparing for switchover..

During the process of coming up of this switch I lost connection to some ip's, e.g I can ping some in another vlan and some not. Made test from another pc, there I can ping the pc's not working with the other pc and can't ping the others.

I did also a reload of the whole vss stack wirth reload command. But still same issue.

After disabling all the redundant links to server and access switches on standby VSS switch, everything is workin again. When I enable the second link e.g. to an access Switch connected with Etherchannel, connection to all devices and also management ip on this switch is lost again:

Actual os 03.07.02.E  I have not yet upgrade, because I can not interrupt connections at the moment.

Regards
Peter






Luis Villanueva... Fri, 04/18/2014 - 21:23
User Badges:
  • Cisco Employee,

Hello Tarun,

 

Please find below the steps to perform the ISSU:

 

ISSU Prerequisites

Before one can perform an ISSU, there are a few prerequisites one must verify for a successful ISSU. The following list explains what is initially required.

• Must be using a redundant Cisco Catalyst 4500 switch with symmetric hardware (that is, supervisors, memory, rommon, NFL daughter card, and so on).

• Both new and old Cisco IOS Software images must be preloaded to the file system on both supervisors.

• SSO must be configured and working properly.

• Config register must be configured to autoboot (that is, the value should have a "2" in the lowest byte).

45010R-203# sh bootvar | i register

Configuration register is 0x2102

Standby Configuration register is 0x2102

Several commands are available to verify if SSO is enabled:


4510R-203# sh module | b Redundancy

Mod  Redundancy role     Operating mode      Redundancy status

----+-------------------+-------------------+-------------------

 1   Standby Supervisor   SSO                  Standby hot        

 2   Active Supervisor    SSO                 Active

45010R-203# sh redundancy states 

       my state = 13 -ACTIVE 

     peer state = 8   -STANDBY HOT 

           Mode = Duplex

           Unit = Secondary

        Unit ID = 2

Redundancy Mode (Operational) =  Stateful Switchover

Redundancy Mode (Configured)  =  Stateful Switchover

Redundancy State              =  Stateful Switchover

             <snip>

4507R-ISSU# sh run | b redundancy

redundancy

 mode  sso

 

 

As a step prior to the beginning of the ISSU process, the new version of the Cisco IOS Software image needs to be loaded into both the active and standby supervisors' file systems. Both active and standby supervisor need to contain both the new and old images in the file system. In order to store both new and old images, the supervisors should be upgraded to contain sufficient amounts of flash memory prior to the ISSU process.

The new images can be downloaded into both supervisors using commands such as:

copy tftp: bootflash:

copy tftp: slavebootflash: 

The example below illustrates this verification:

4510R-203#dir

Directory of bootflash:/

    

1  -rwx 13636500 Sep 6 2006 03:18:58 -08:00 cat4500-entservices-mz.122-31.SGA

2  -rwx 13747611 Sep 9 2006 03:19:58 -08:00 cat4500-entservices-mz.122-31.SGA1

4510R-203#dir slavebootflash:

Directory of slavebootflash:/

1  -rwx 13636500 Sep 6 2006 03:18:58 -08:00 cat4500-entservices-mz.122-31.SGA

2  -rwx 13747611 Sep 9 2006 03:19:58 -08:00 cat4500-entservices-mz.122-31.SGA1 

Once this check is verified, one can now proceed with the ISSU process.

 

The ISSU process is started by typing the "issu loadversion" command on the active supervisor. This command directs the active supervisor to begin the ISSU process. The active supervisor, through intersupervisor communications, checks that the requested image has been downloaded into both the active and standby supervisors' file systems. If the required images are not present, the command is rejected, and an appropriate warning is generated.

If the "issu loadversion" command is successful, the switch transitions into the "Load Version" ISSU state. The standby supervisor will reset and boot with the new version of the Cisco IOS Software image loaded into the file system.

The following actions take place when the command is implemented:

1. The standby supervisor (B) is reset.

2. The standby supervisor (B) is booted with the new Cisco IOS Software image: Release 12.2(31)SGA1.

3. If both Cisco IOS Software images are declared as compatible, the standby supervisor moves into SSO mode and is fully stateful for all compatible clients and applications. Compatibility allows for in-service software upgrade or downgrade between two versions to succeed with minimal service effect.

4. If both Cisco IOS Software images are incompatible, the system moves into RPR mode, and the ISSU process is terminated with an appropriate message to the user. Images are declared incompatible when "required" clients or applications are not interoperable between two Cisco IOS Software releases.

5. Standby "B" reaches the standby HOT state.

6. The user has an option to abort the ISSU process by issuing the "issu abortversion" command.

7. The "issu loadversion" command also supports a "forced" option that allows the operator to force the system into entering RPR mode when incompatibility is detected.

Note: When performing an ISSU, disable manual switchovers. Performing manual switchovers during the issu process is strongly discouraged. The current implementation does not prevent it, but it does display a warning to the user.

An example of the CLI for implementing the issu loadversion command is displayed below.

On the active supervisor, one would issue the following command:

4510R-203#issu loadversion 1 bootflash:cat4500-entservices-mz.122-31.SGA1 2 slavebootflash: cat4500-entservices-mz.122-31.SGA1

Syntax - issu loadversion active-slot active-image-new standby-slot standby-image-new

 

 

The second step of the ISSU process is to perform the issu runversion CLI.

The user can issue the " issu runversion" command when:

1. The ISSU state is "Load Version"; this can be verified with the "show issu state detail" CLI.

2. The standby supervisor is running the new version of the software.

3. The standby supervisor has moved into the "Standby Hot " state.

The following actions take place when the " issu runversion" command is executed:

1. A switchover occurs; that is, the standby (B) becomes the new active, and the old active (A) is rebooted and comes up as a standby.

2. A timer called "Rollback Timer" is started with a previously configured value.

3. Move both supervisors to "Run Version" state.

4. If the command "issu acceptversion" is not issued before the "Rollback timer" fires, then the entire ISSU process is aborted via the automatic rollback.

5. If the active supervisor console connectivity is established and the "issu acceptversion" command is issued, then the rollback timer is stopped.

6. The user has an option to abort the ISSU process by issuing the "issu abortversion" command.

An example of the CLI for implementing the issu runversion command is displayed below:

On the active supervisor, one would issue the following command:

4510R-203#issu runversion 2 slavebootflash:cat4500-entservices-mz.122-31.SGA1

Syntax - issu runversion standby-slot [standby-image-new]

 

 

Prior to issuing the `issu acceptversion' command the system will be counting down the rollback timer. If `issu acceptversion' is not completed before rollback timer expires an automatic abort will occur. This command stops the "Rollback Timer." This command serves as a feedback mechanism. This is an optional command and can be skipped in the ISSU process with the "issu commitversion" CLI.

If this command is not issued within 45 minutes (default) from the time the standby supervisor moves into the "Standby Hot" state, it is assumed that the new active supervisor is not reachable and the entire ISSU process is rolled back to the previous version of the software. The acceptversion is not intended for long-term network operation. It is also important to note that none of the features available on the new version will work yet.

The following actions take place when the command is implemented:

1. The "Rollback Timer" is terminated. This means that the rollback timer is not looked at anymore. Therefore, the system can run in this state for an extended period.

2. The user has an option to abort the ISSU process by issuing the command "issu abortversion."

Aborting the ISSU process now causes the newly active supervisor (B) to fail over to the standby supervisor (A) running the old image and will also cause the rebooting supervisor (B) to load the original image. The issu acceptversion halts the rollback timer and helps ensure the ISSU process is not automatically aborted during the process.

An example of the CLI for implementing the issu acceptversion command is displayed below:

On the "New" active supervisor, one would issue the following command:

4510R-203#issu acceptversion 2

% Rollback timer stopped. Please issue the commitversion command.

Syntax - issu acceptversion active-slot-number

 

This is the last stage of the ISSU procedure. Once the user is satisfied with the new version of software, this must be committed by issuing the "issu commitversion" command. This command resets the standby supervisor and boots it with a new version of the software (same as the active supervisor). This concludes the ISSU process, and the new version of software is permanently committed on both supervisors. Since this is the conclusion of the ISSU process, the system can not be reverted back to the previous version of the software from this point onward as a part of this upgrade cycle. However, if for any reason users wish to go back to the previous version of the software, they can do so by starting a new upgrade/downgrade process.

The following actions take place if the command is implemented:

1. The standby supervisor (A) is reset and booted with the new version of Cisco IOS Software image.

2. The standby supervisor (A) moves into the "Standby Hot" state in SSO mode and is fully stateful for all clients/applications that are compatible.

3. Both supervisors are moved into "Final State," which is the same as "Initial State."

4. Users can initiate switchovers from this point onward.

An example of the CLI for implementing the issu commitversion command is displayed below:

4510R-203#issu commitversion 1

Syntax - issu commitversion standby-slot-number

 

 

ISSU Process: issu abortversion

One can abort the ISSU process at any stage manually (prior to issuing the issu commitversion command) by issuing the exec-level issu abortversion command. The ISSU process also aborts on its own if the software detects a failure.

If a user aborts the process after issuing the issu loadversion command, then the standby supervisor engine is reset and reloaded with the original software.

If the process is aborted after a user enters either the issu runversion or issu acceptversion command, then a second switchover is performed to the new standby supervisor engine that is still running the original software version.

The supervisor engine that had been running the new software is reset and reloaded with the original software version. The command is accepted only in "Load Version" or "Run Version" states. In "Load Version" state, the active supervisor is running an old image and the standby supervisor is running new image.

Syntax - issu abortversion active-slot [active-image-new]

 

Let me know if you have any questions.

 

Actions

This Discussion