Solved: Upgrade of MDS 9222i, SANOS 3.2(3a) to 5.0.1(a)

karllewis · ‎08-12-2010

Hi folks -

I have four MDS 9222i, currently running 3.2(3a). All four have the same specs:

dc-cis9222i-01# sh module
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
1 22 4x1GE IPS, 18x1/2/4Gbps FC/Sup2 DS-X9222I-K9 active *

Mod Sw Hw World-Wide-Name(s) (WWN)
--- -------------- ------ --------------------------------------------------
1 3.2(3a) 1.1 20:01:00:0d:ec:81:cb:c0 to 20:12:00:0d:ec:81:cb:c0

Mod      Application Image Description       Application Image Version
-------- -----------------------------       -------------------------
1        SSI linecard image (Packaged in SAN-OS) 3.2(3a)

Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 00-17-94-ee-5b-64 to 00-17-94-ee-5b-6c JAE1212BPPK

I'd like to upgrade all of them to the latest SANOS (unless someone has a good reason why I should not). I'm managing them with Fabric Manager 3.3(5) Standalone. I've read the Release Notes for Cisco Fabric Manager Release 5.0(1a) and the Cisco MDS 9000 NX-OS Release 5.0(1a) and SAN-OS 3.3(x) Software Upgrade and Downgrade Guide, and can see that my hardware is on the support list for 5.0(1a), so I think I'm ready.

Unless I misread this, I have upgrade FM from 3.3.5 to 4.2.5 in one step, then upgrade from 4.2.5 to 5.0.1 in a second step. I think I get that.

What I'm confused about is, what do about firmware and kickstart versions on the MDS as I go through each version of FM? Each 9222i is running 3.2.3. now, and the upgrade guide says that each switch can do a non-disruptive upgrade from 3.2.3 to 3.3.5, then to 4.2.5 and finally to 5.0.1.

Do my upgrade steps look like this:

1) upgrade switches with m9200-s2ek9-kickstart-mz.3.3.5.bin and m92000-s2ke9-mz.3.3.5.bin

2) upgrade FM from 3.3.5 to 4.2.5, after the switches are on 3.3.5

3) upgrade switches with m9200-s2ek9-kickstart-mz.4.2.5.bin and m92000-s2ke9-mz.4.2.5.bin

4) upgrade FM from 4.2.5 to 5.0.1

5) upgrade switches with m9200-s2ek9-kickstart-mz.5.0.1a.bin and m92000-s2ke9-mz.5.0.1a.bin

I don't believe I have any extra services/features enabled on the switches. I'm using FCIP, but no FICON, SANTap, SME or DMM.

Thanks!

mattkauffmann · ‎08-19-2010

I asked them that because we'll have to do the same thing. I was told that 3.3.5 was the oldest version they tested with 5.0.1 and IVR which we use quite a bit but that's SAN-OS versions. I think you can lag your SAN-OS a little with FM i.e. 5.0.1(a) will manage older versions but . . . you probably don't want to tary too long. Are you running FM stand alone or are you using a licensed server and doing all the fabric monitoring and collecting?? We will be running 5.0.1(a) with 3.3.5 for a little while until we swap out older edge switches for 9148's that only run 5.x and in order to manage them we'll have to have FM server at 5.x a well.

View solution in original post

Gary Ross · ‎08-19-2010

Matt & Karl,

You can run FM 5.0.1a and manage switches running older SAN-OS/NX-OS code just not the other way around. I've never done

such a large jump in upgrading FM but I don't see why this would be a problem. If the features aren't available on the switches,

they won't be available in FM.

I was surprised to see that Gen 1 modules are no longer supported on NX-OS 5.x but I knew this day was coming (we have Gen 1

modules in all 4 of our 9222i's but our 4 9513's are all Gen 2). Whew!

Looking at upgrading the cores (9513's) to 5.0.1a for an ASIC feature I've been looking for and upgrading our 9222i's to 4.2.5,

until we can remove the Gen. 1 modules, next week sometime. FM 5.0.1a is already installed with the 9513's at 4.1.3a

and the 9222i's at 4.1.1b. NX-OS 4.1.x code was very buggy for us.

Karl, since you mentioned that your running FM standalone, I'm also assuming your DB is PostgreSQL 8.2.

Hope this helps.

Gary

View solution in original post

mattkauffmann · ‎08-16-2010

Karl,

it's not apples to apples for hardware but we just tested going from 3.3.5 to 5.0.1 on our 9134's and it was a two step process if you want it to be non-disruptive. When we tried to go from 3.3.5 to 5.0.1 our 9134's sho install all impact told us we needed to reboot. If we went from 3.3.5 to 4.2.5 and then to 5.0.1 those two steps were NDU. Now I know 9222i's are more core-ish then edge-ish like the 9134's but since they have the single supvervisor I would think they would act the same. As far as your finding on the FM server that's the path we're going to take, 3.3.5 to 4.2.5 then to 5.0.1 I think that's more an Oracle thing then a Cisco thing (if you're using the Oracle database backend for FM Server.) Hope this helps.

karllewis · ‎08-16-2010

Thanks, Matt - this is tremendously helpful! I have just one more question: In your upgrades, was there every a point that FM could not manager a switch below X.Y.Z level of code? Basically, I thinking of upgrading FM from 3 to 4 to 5 all at once, then going back and upgrading the switches along the NDU path later. Otherwise, if, say, FM 5.0.1 can't manage a 3.3.2 switch, then I have to upgrade FM to 4.2.5,then upgrade the switches, then come back to FM and upgrade to 5.0, then back to the switches, etc.

I can't find anything in the documentation either way, so I'm just not. If you've never heard of that either, I'll just do the full FM upgrade, then do the switch upgrades. Thanks so much for any more input you can supply!

Karl

mattkauffmann · ‎08-19-2010

I asked them that because we'll have to do the same thing. I was told that 3.3.5 was the oldest version they tested with 5.0.1 and IVR which we use quite a bit but that's SAN-OS versions. I think you can lag your SAN-OS a little with FM i.e. 5.0.1(a) will manage older versions but . . . you probably don't want to tary too long. Are you running FM stand alone or are you using a licensed server and doing all the fabric monitoring and collecting?? We will be running 5.0.1(a) with 3.3.5 for a little while until we swap out older edge switches for 9148's that only run 5.x and in order to manage them we'll have to have FM server at 5.x a well.

Gary Ross · ‎08-19-2010

Matt & Karl,

You can run FM 5.0.1a and manage switches running older SAN-OS/NX-OS code just not the other way around. I've never done

such a large jump in upgrading FM but I don't see why this would be a problem. If the features aren't available on the switches,

they won't be available in FM.

I was surprised to see that Gen 1 modules are no longer supported on NX-OS 5.x but I knew this day was coming (we have Gen 1

modules in all 4 of our 9222i's but our 4 9513's are all Gen 2). Whew!

Looking at upgrading the cores (9513's) to 5.0.1a for an ASIC feature I've been looking for and upgrading our 9222i's to 4.2.5,

until we can remove the Gen. 1 modules, next week sometime. FM 5.0.1a is already installed with the 9513's at 4.1.3a

and the 9222i's at 4.1.1b. NX-OS 4.1.x code was very buggy for us.

Karl, since you mentioned that your running FM standalone, I'm also assuming your DB is PostgreSQL 8.2.

Hope this helps.

Gary

karllewis · ‎08-20-2010

Thanks for the tip, Gary - I'm doing just that. I opted to upgrade to FM 4.2.5 and keep the switches below that. I did choose to upgrade one switch to 5.0.1a, but I'm managing it from the command line, and this switch has no hosts on it. When I get my downtime window next week, I'll upgrade FM to 5.0.1a and get all my switches onto the same code.

Are your Gen 1 modules the 14+2 MPS blades? Thanks again!

Gary Ross · ‎08-20-2010

Karl,

The Gen. 1 module is a 32port SSM. The Storage Architect purchased this blade not knowing that the 9222i already had the ability to do SANtap. To be honest, I didn't know that the 9222i had this capability either until we put SANtap into production.

Good luck with the upgrades.

Gary

karllewis · ‎08-20-2010

Thanks, Matt -

Your comments were extremely helpful! I'm running FM standalone. I opted to upgrade FM to 4.2, then I began updating switches to later versions of code. As I said, everything was running 3.2.3a for the longest time. I just upgraded everything to atleast 3.3.5 for now. One switch on one fabric has storage only, no hosts. I chose to upgrade that one to 5.0.1a, just to see. So far, I have no issues with FM managing all of the switches on two different codes. I have a data center outage next week, so I will upgrade FM and switches to 5.0.1a ahead of that. Thanks for your help!

Karl

Zubair.Sayed_2 · ‎12-14-2010

Hi All.

This post is very useful. However I have some questions I would like to ask.

Firstly we are upgrading our MDS 9222i SAN switches from m9200-s2ek9-mz.3.2.3a.bin to m9200-s2ek9-mz.3.3.5a.bin.

We are trying to perform this upgrade with no disruption if possible. Reading Cisco information I came across the following:

Software Upgrade Methods

You can upgrade software without any disruptions using the Cisco MDS SAN-OS software designed for mission-critical high availability environments. To realize the benefits of nondisruptive upgrades on the Cisco MDS 9500 Directors, we highly recommend that you install dual supervisor modules.

You can upgrade any switch in the Cisco MDS 9000 Family using one of the following methods:

•Automatic - you can use the Fabric Manager Software Install Wizard for Cisco MDS SAN-OS switches as described in the "Using the Software Install Wizard" section.

•Manual - for information on manual upgrades, see the Cisco MDS 9000 Family CLI Configuration Guide or the Cisco MDS 9020 Switch Configuration Guide and Command Reference.

In some cases, regardless of which process you use, the software upgrades may be disruptive. These exception scenarios can occur under the following conditions:

•A single supervisor module system with kickstart or system image changes.

•A dual supervisor module system with incompatible system software images

My question is, how do I go about finding out whether I have:

•A single supervisor module system with kickstart or system image changes.

•A dual supervisor module system with incompatible system software images

Thanks

Z

mattkauffmann · ‎12-14-2010

Zubair,

I can tell you what I know. I have some 9222i's but I have not put them into service but I do have a pair of 9216 (same basic setup though Gen 1) and they only have one supervisor so code updates are disruptive because the switch has to reboot. It's pretty quick, probably on the order of 30-45 seconds. Now the Gen 2 edge switches like the 9124's etc do restart but the outage is much quicker ~ 5 seconds I believe

sho system reset-reason

----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) ---

1) At 333640 usecs after Sun Aug 1 06:24:17 2010

Reason: Reset due to upgrade

Service:

Version: 3.3(1c)

2) No time

Reason: Unknown

Service:

Version: 3.3(1c)

3) No time

Reason: Unknown

Service:

Version: 3.3(1c)

4) At 654670 usecs after Sun Dec 14 01:18:19 2008

Reason: Reset due to upgrade

Service:

Version: 3.1(2)

What does the sho install all impact output say for that switch?

Michael Brown · ‎12-14-2010

The 9222i can be upgraded non-disruptively if you follow the guidelines on CCO. Here is the link to the 5.0.1(a) release notes. Table 11 shows the path that you need take with regards to complete the upgrade non-disruptively. Basically its --> 3.3, then --> 4.1 or 4.2, then --> 5.0. It also lists all the features that may be impacted by the upgrade.

The 9222i and the 9124/9134 and new 9148 all support ISSU (In Service Software Upgrade). What they do is freeze the control plane for <80 seconds, while the sup is upgraded. If the default FSPF timers are in place, would prevent any ISL failures. When the control plane is frozen, no new devices can login, no zone changes should be attempted.

http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/5_0/release/notes/nxos/mds_nxos_rel_notes_501a.html#wp698225

Remember that the Gig E ports will flap when the box is upgraded, so if you have FCIP running and there is not a port channel with links from line card 1 and line card 2 combined, your FCIP tunnel will flap. Best practice for FCIP tunnels is to build a port channel with links on different line cards, then when the upgrade is happening on line card 1, the traffic will be up on the link from line card 2, and if it all works right, the link on line card 1 will be back up before the upgrade flaps the link on line card 2. This would prevent an FCIP outage.

-Hope this helps clarify ISSU,

Mike

I did forget to mention that if you upgrade the swtich using the 'install all' command from the CLI, if you are connected via telnet or SSH, the connection will drop as part of the upgrade when the sup is upgraded, or switches over in the case of the 95xx. This does not indicate that FC traffic was impacted.

Zubair.Sayed_2 · ‎12-20-2010

Thanks Guys, very useful info.

MB - I am probably going to follow your steps, however I am a little confused as to what platform I select when downloading the latest software.

I get the following 4 options:

1. Software on Chassis

2. Cisco MDS 9000 32-Port Storage Services Module

3. Cisco MDS 9000 18/4-Port Multiservice Module

4. Cisco MDS 9000 16-Port Storage Services Node

How do I check which platform I am running?

When I last checked the software version was at 3.2, I notice you mentioned 5.0.1.

Regards

Zubair

Michael Brown · ‎12-20-2010

Zubair,

The choices you are seeing are related to the inteligent modules. For the 9222i chassis you will need the followoing images.

Under 'software on chassis' select the 'SAN-OS System Software' and get the 3.3.(5a) image.

Under 'software on chassis' select the 'SAN OS Kickstart' and get the 3.3(5a) kickstart image.

Then under 'software on chassis' select the 'NX-OS System software' and get one of the 4.2 images and the 5.0(1a) image as well.

Then under 'software on chassis' sellect the NX-OS Kickstart' and get the 4.2 kickstart for the same image number that you used in the step above, and then get the 5.0(1a) kick start.

Then the commands to do the install would be

install all kickstart bootflash: system bootflash:

Follow the prompts. When this completes...you can then copy the 4.2 kickstart and main image to bootflash, and repeat the above command using the 4.2 image names.

When that completes, repeat the steps with the 5.0(1a) images.

Cisco SAN-OS is the name for software versions 3.x and below. They changed the name of the software to NX-OS starting with version 4 of the software.

From the show module output in the original post, you only need the main and kickstart chassis images as there is no intelligent linecard in slot 2.

I think the command Matt was suggesting is this.

show install all impact kickstart bootflash:m9500-sf2ek9-kickstart-mzg.5.0.1a.bin system bootflash:m9500-sf2-fcoe-ek9-mzg.5.0.1a.bin

In this example, the 5.0.1a kickstart and main images would need to be on the bootflash: in the the swtich...this command would extract all the software components and check with each line card and then post a report as to what would be disruptive and what would be non-disruptive. The install all command I listed above will do the exact same thing, but at the end of the install commands you will be promped to hit a Y to continue the install...or N to cancel the install. The 'show install all impact' will simply run the check and post the report.

Hope this helps,

Mike

Zubair.Sayed_2 · ‎01-04-2011

Hi Michael Brown

Please ignore my last post, the switch had to be physically rebooted for some reason. I performed the same upgrade process you advised on switch 2 and it worked 100%. Thanks alot for your help.

I am not having an NTP reset issue on the switch 2 for some reason, funny enough no NTP issues on switch 1 post upgrade.

I will raise a new post regarding this issue.

Thanks.

Zubair

Zubair.Sayed_2 · ‎12-26-2010

Hi MB,

Thanks for all the help and useful info thusfar.

I downloaded the 6 images as you suggested. Copied them to SAN 1 switch bootflash: folder.

Ran the show install all impact using the kickstart and system image to upgrade first to version 3.3(5a).

Everything was going ok and then suddenly switch seemed to have rebooted and now we are unable to connect.

This is the sh hardware from switch 2, they are exactly the same switches. Not sure if I missed something in my previous post but so far it seems i did everything correctly...

Any other checks or suggestions before trying the upgrade again???

SAN2# sh hardware

Cisco Storage Area Networking Operating System (SAN-OS) Software

TAC support: http://www.cisco.com/tac

The copyrights to certain works contained herein are owned by

other third parties and are used and distributed under license.

Some parts of this software may be covered under the GNU Public

License or the GNU Lesser General Public License. A copy of

each such license is available at

http://www.gnu.org/licenses/gpl.html and

http://www.gnu.org/licenses/lgpl.html

Software

BIOS: version 1.0.12

kickstart: version 3.2(3a)

system: version 3.2(3a)

BIOS compile time: 09/10/07

kickstart image file is: bootflash:/m9200-s2ek9-kickstart-mz.3.2.3a.bin

kickstart compile time: 1/10/2008 9:00:00 [02/03/2008 08:21:02]

system image file is: bootflash:/m9200-s2ek9-mz.3.2.3a.bin

system compile time: 1/10/2008 9:00:00 [02/03/2008 09:12:58]

Hardware

cisco MDS 9222i ("4x1GE IPS, 18x1/2/4Gbps FC/Sup2")

Motorola, ppc8548 (e500) with 1032388 kB of memory.

Processor Board ID JAE1234SFTE

bootflash: 1023120 kB

SAN2 kernel uptime is 441 days 4 hours 48 minute(s) 23 second(s)

Last reset at 351685 usecs after Thu Oct 8 19:19:02 2009

Reason: Kernel Panic

System version: 3.2(3a)

Service:

--------------------------------

Switch hardware ID information

--------------------------------

MDS Switch is booted up

Switch type is "MDS 2 Slot Chassis"

Model number is DS-C9222I-K9

H/W version is 1.0

Part Number is 73-11019-01

Part Revision is A1

Manufacture Date is Year 12 Week 34

Serial number is FOX1234HBQK

CLEI code is COM9310ARA

--------------------------------

Chassis has 2 slots for Modules

--------------------------------

Module in slot 1 is ok

Module type is "4x1GE IPS, 18x1/2/4Gbps FC/Sup2"

No submodules are present

Model number is DS-X9222I-K9

H/W version is 1.2

Part Number is 73-11018-06

Part Revision is B0

Manufacture Date is Year 12 Week 34

Serial number is JAE1234SFTE

CLEI code is COUIAMCCAA

Module in slot 2 is ok

Module type is "1/2/4 Gbps FC Module"

No submodules are present

RAM size is 0 (kb)

Model number is DS-X9124

H/W version is 1.7

Part Number is 73-9622-52

Part Revision is C0

Manufacture Date is Year 12 Week 50

Serial number is JAE12502Z4X

CLEI code is COUCABUCAB

---------------------------------------

Chassis has 2 Slots for Power Supplies

---------------------------------------

PS in slot A is ok

Power supply type is "800.10W 110v AC"

Model number is DS-CAC-845W

H/W version is 1.2

Part Number is 341-0052-03

Part Revision is B0

Manufacture Date is Year 12 Week 27

Serial number is QCS122710AU

CLEI code is CNUPAA8AAA

PS in slot B is ok

Power supply type is "800.10W 110v AC"

Model number is DS-CAC-845W

H/W version is 1.2

Part Number is 341-0052-03

Part Revision is B0

Manufacture Date is Year 12 Week 27

Serial number is QCS122710FE

CLEI code is CNUPAA8AAA

----------------------------------

Chassis has 1 slot for Fan Module

----------------------------------

Fan module is ok

Model number is DS-2SLOT-FAN

H/W version is 4.0

Part Number is 800-24478-02

Part Revision is B0

Manufacture Date is Year 12 Week 33

Serial number is DCH12331948

CLEI code is CNUQABBAAC

-----------------------------------------

Chassis has 1 slot for Interface Module

-----------------------------------------

Interface module is ok

Model number is DS-X9222-MGT

H/W version is 1.0

Part Number is 73-11488-02

Part Revision is A0

Manufacture Date is Year 12 Week 34

Serial number is JAE1234SEB0

CLEI code is 0

SAN2#

-------------------------------------------