Solved: ASA 5520 Active/Standby Memory Upgrade

max.pierson · ‎07-22-2010

Hi All,

We're about to undertake the 8.2 -> 8.3 upgrade. I understand that failover config will not work if the pair has different memory. For those that have done this already, what was your plan on inserting the new memory?? Obviously, we want to minimize downtime.

Thanks,

M

bragan · ‎07-22-2010

We just did this...here's what I did

First, upgrade the ASDM (on both units) to the latest version (6.31) before starting on either unit. The ASDM is backwards-compatible, so there's no problems using it with 8.2. Then follow these steps:

1. Turn the primary unit off. The standby unit will take over the duties.

2. Upgrade the memory and reinstall in the rack but do not power the unit yet.

3. Power off the standby and power up the primary unit. The only downtime is the primary unit rebooting.

4. Upgrade the memory in the standby unit, reinstall in the rack and power up.

After both units are back up and running, I did the 8.3 upgrade through ASDM and changed the boot directory to the new version. Then it's just a simple reboot of both units to use the new version. I was suprised at how easy the process was.

View solution in original post

Kureli Sankar · ‎08-11-2010

These steps should do the trick to do hitless upgrade on a failover pair with memory upgrade.

1. Install the image on one unit (say secondary/standby) and set the boot variable, and then power it off.

2. Install the memory, and then power unit back on.

3. It will boot up and convert the config and then do the failover config sync.
at this point it will bring the config from the active unit which will be 8.2 config but it will convert it as it receives.
4. Once complete, this unit will be running 8.3 as a standby.

5. We can make it Active and then perform the same functions on the peer unit.

Regarding nat statements that may not convert over properly, we did have some known issues that were solved in 8.3.2 code.

Pls. read open and resolved caveats are here:

I have listed some related to NAT which are resolved in the 8.3.2 code

CSCte73170

CSCte74686

CSCte74866

CSCte75201

CSCte82378

CSCte92758

CSCte95091

CSCte96408

CSCtf63794

CSCtf57830 Incorrect real ip translation of ACE after 8.3.1 upgrade

-KS

View solution in original post

August Ritchie · ‎07-22-2010

First and foremost, a downtime window is recommended for this as it is a major upgrade.

According to documentation, using different amounts of RAM shouldn't cause consequences, however, it is not recommended.

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/ha_overview.html#wp1077536

So you could in theory, upgrade secondary RAM, failover, then upgrade primary ram. When doing this though, I would suggest you totally disconnect the secondary. Upgrade the RAM, and then reconnect just the failover link to make sure that it syncs correctly and failover is established.

After the RAM is installed you can do the following:

1 - Download the new software to both units, and specify the new image to
load with the boot system command

2 - Force a failover to the secondary ASA by issuing the no failover active
command on the primary ASA or power off the primary ASA. This causes the
secondary PIX to become active.

2 - Disconnect all network cables from the primary ASA.

3 - Power on or reload the primary ASA and verify the new version, license
keys and features, configuration and so on.

4 - Power off the primary ASA.

5 - Re-connect all cables to the primary ASA.

6 - Quickly power off the secondary ASA, then immediately power on primary
ASA. Verify that the primary ASA is now passing traffic running the new
version of code.

Note: Your downtime occurs while the primary is booting.

7 - Once the primary ASA is up, it becomes active and passes traffic.

8 - Repeat steps two through seven for the secondary ASA.

9 - Power on the secondary ASA. It becomes the standby unit. Wait two
minutes and verify that the secondary ASA is in standby mode and that all
interfaces have a status of Normal. Both ASA firewalls are now running the
upgraded version and back to normal operation.

bragan · ‎07-22-2010

We just did this...here's what I did

First, upgrade the ASDM (on both units) to the latest version (6.31) before starting on either unit. The ASDM is backwards-compatible, so there's no problems using it with 8.2. Then follow these steps:

1. Turn the primary unit off. The standby unit will take over the duties.

2. Upgrade the memory and reinstall in the rack but do not power the unit yet.

3. Power off the standby and power up the primary unit. The only downtime is the primary unit rebooting.

4. Upgrade the memory in the standby unit, reinstall in the rack and power up.

After both units are back up and running, I did the 8.3 upgrade through ASDM and changed the boot directory to the new version. Then it's just a simple reboot of both units to use the new version. I was suprised at how easy the process was.

max.pierson · ‎07-23-2010

Thanks for the response guys. Since we have a very tight change control (for those that don't speak network), I would rather take the known downtime so they can expect it. Telling them I can do this with no hit to services, and then taking it down would look bad. This is the methodology I had in mind anyways, but I wanted to make sure no one has been able to do it without taking any downtime.

Thanks again guys,

M

David White · ‎07-24-2010

Hi Max,

Just so you know, you can upgrade from 8.2 to 8.3 with zero downtime while performing the memory upgrade. We support this and we test for it.

Just install the image on one unit and set the boot variable, and then power it off. Install the memory on it, and then power it back on. It will boot up and convert the config and then do the failover config sync. Once complete, this unit will be running 8.3 as a standby. Next, make it Active and then perform the same functions on the peer.

Note that we do support mis-matched memory for short durations during the upgrade process so that you can perform zero downtime upgrades.

Sincerely,

David.

aidacruises · ‎07-24-2010

Hello,

we allready accomplished the RAM exchange successfully and are now on the way to move from 8.0.4 to 8.3.1 -- are we able to use the zero downtime feature without a need of intermediate software versions?

If I read http://www.cisco.com/en/US/docs/security/asa/asa83/configuration/guide/admin_swconfig.html#wp1192014 correctly we will need at least the same major version or the latest version of the last major (here 8.2?). Or can we also "jump" from 8.0 to 8.3 (without any risk)?

Thank you &

Kind regards

Mathias

David White · ‎07-24-2010

Hi Mathias,

You should upgrade from 8.0 to 8.2 (which can be done with zero

downtime), and then from 8.2 to 8.3, which can again be done with zero

downtime.

I would not recommend upgrading directly from 8.0 to 8.3.

Sincerely,

David.

dbuttry · ‎07-23-2010

Have any of you had any issues with NAT and the new upgrade? I know that NAT has been totally rewritten and it is suposed to convert them for you. I however have hundreds of Static Policy NATs to support many LAN-2-LAN VPNs. I am a bit worried about that. I am thinking of trying to get into a Cisco LAB and try it out ahead of time. Thoughts?

Thanks.

Kureli Sankar · ‎08-11-2010

These steps should do the trick to do hitless upgrade on a failover pair with memory upgrade.

1. Install the image on one unit (say secondary/standby) and set the boot variable, and then power it off.

2. Install the memory, and then power unit back on.

3. It will boot up and convert the config and then do the failover config sync.
at this point it will bring the config from the active unit which will be 8.2 config but it will convert it as it receives.
4. Once complete, this unit will be running 8.3 as a standby.

5. We can make it Active and then perform the same functions on the peer unit.

Regarding nat statements that may not convert over properly, we did have some known issues that were solved in 8.3.2 code.

Pls. read open and resolved caveats are here:

I have listed some related to NAT which are resolved in the 8.3.2 code

CSCte73170

CSCte74686

CSCte74866

CSCte75201

CSCte82378

CSCte92758

CSCte95091

CSCte96408

CSCtf63794

CSCtf57830 Incorrect real ip translation of ACE after 8.3.1 upgrade

-KS

aidacruises · ‎08-11-2010

Last week I tried the zero downtime upgrade from 8.2.2 to 8.3.1. Installed the new OS images on both devices and reloaded the secondary (failover/standby) device. After this the device got unreachable. Checked via console I've got a converting issue of some config lines which resulted a reload - config upgrade failed - reboot cycle. While reverting to 8.2.2 nothing bad happend for sure.

Now I plan for tomorrow to reload both devices in parallel and move from 8.2.2 to 8.3.2. Hopefully this will be successfully.

Also I've done some test with another device and config from 8.2.2 -> 8.3.1 -> 8.3.2 which was mostly successfull. Had one NAT issue with some asymetric NAT stated in the logging (which wasn't here before) so I guess this will not happen when going directly from 8.2.2 to 8.3.2 since the closed bugs above.

Last point what was really upset is there the convert process generates a lot of new ACLs and NAT rules so that a complex firewall setup is getting really into a mess, especially when using network-groups before. I'm wondering if nobody complains about that. From my point of view the convert process should mostly leave all rules as before or generate mostly similar rules not such a huge amount of "new" rules.

So far, hope the best for tomorrow night....

max.pierson · ‎08-12-2010

We finally got our 5520's upgraded. It didn't go as smoothly as we would have liked. We first tried to down the secondary unit, upgrade the memory in it, bring it back up and then failover to it. When we failed over to the standby with 2gb of ram, it would not forward packets (from the standby, I could ping inside hosts and outside hosts, but no hosts could ping out). We ended up rolling back the change and calling TAC to see what they had to say. TAC did say that there was not any documentation in regards to the procedure (upgrade a failover pair memory), but he did say that Cisco recommends upgrading the secondary unit first just as we did.

The next night, we decided to go all out and try what a few others have suggested and failover to standby, upgrade memory and OS on primary unit. Then rack primary back up, turn the secondary (active) unit off, and then turn on the primary and wait for it to boot (which caused a downtime of however long it took for the primary unit to boot). Then take the secondary unit offline, wash, rinse, repeat.

After both units were back online, we had a few nat messages after the migration, but everything seemed to work without any problems. Since it was late, we didn't test extensively, we didn't notice anything as being down. Turns out that almost every nat rule that was created during the migration had _unidirectional_ added to the end of it. There's no documentation of why that happened either.

http://www.cisco.com/en/US/docs/security/asa/asa83/upgrading/migrating.html

After making the statements bidirectional, all was well.

I think alot of our headache would have never happened if Cisco would have had this documented. We have mixed feelings about how the new statements and functions work and need to be configured. We also feel that the creation of all of the new objects is too much fluff and clutters up the config.

back to my coffee now

max

JMCNEL · ‎11-08-2010

how does the memory upgrade and upgrading from 8.2 to 8.3 affect the IPS devices inside the ASA?

aidacruises · ‎11-08-2010

The memory upgrade as well as the ASA OS update does not affect the IPS modules in there function, there work mostly as a standalone device with a shared backplane to intercept / redirect traffic in the ASA chassis.