Time expected to upgrade CUCM cluster v.8.6.1.a to 8.6.2.a

[r.ferrero]

Hi,

We need to apply an update in our cluster of CUCM from version 8.6.1.a to version 8.6.2.a. Cluster is composed by 1 Publisher and 4 Subscribers deployed along 3 places (Place 1: Publisher + 2 Subscribers; Place 2: Subscriber; Place 3: Subscriber)

I know time expected for the whole process depends on several factors such as amount of data in database, WAN links, and so on, Based in our experience working with older releases and reading Cisco documentation, we have planned 2 hours per server.

Well, customer is encouragingly asking us to be more accurate with time expected to get the upgrade finsih.

We have purposed to upgrade one server each time because of several WAN problems in the past and to prevent problems with DB replications based on big amount of traffic flowing across WAN links. Anyway, I know that Subscribers could be updated simultaneosly when you reach a fixed point in the upgrade log process from one subscriber.

In your opinion, What time is it expected to complete the update process? If we'd decide to update several servers at the same time, How long could save in the whole upgrade process?

Thanks

[Gordon Ross]

The core steps in an upgrade are:

- Install & reboot Pub.

- Install all subs

- Reboot subs.

The pub install & reboot can easilly take two plus hours.

The sub installs usually take me about 1:15.

The sub reboots are the tricky ones to time.

If the ${DEITY}s are kind and you do all the subs at once, then that's about 30 minutes. If you haven't been good, it can take an hour or more for the subs to settle down.

Note, that due to the way the database (apepars to) work, once you've rebooted some subs, wait for them to go state two before rebooting any more. (i.e. don't reboot one sub, wait 5 minutes, then reboot another)

So if you're lucky, you could, in theory, do it in four hours. However, I wouldn't try and rush it, otherwise Mr. Murphy will pay a visit.

(I tend to allow 12 hours to do my system. Occasionally, it's taken me 14 hours)

GTG

[gmgarrian]

One thing I would add is that in terms of down time it should be minimal since you're upgrading the dormant partition and you have dual servers.  I would still do the reboots after hours during a maint window.

[Paul Giralt]

On additional comment I'll add is that you do not have to reboot the pub before you do the subscribers. In previous releases you used to be able to start the subs at a point before the pub was finished. In 8.x releases you need to wait for the pub to be completely finished before it will let you install a sub, however you can install them all at once. This can all be done with the publisher still running the old version. This means you can do:

1. Upgrade Publisher (while still online)

2. Upgrade Subscribers (all at once while still online) - note that DB replication setup does not occur until after the switch version.

3. Once all the subscribers are upgraded, reboot the publisher (you can wait as long as you need before doing this).

4. You might want to increase the repltimeout (utils dbreplication setrepltimeout). The default is 5 minutes (300 seconds). What this value does is tell the publisher how long to wait for additional subscribers before starting replication setup. For example, if you reboot one sub and the pub sees it, it will start the timer. If another sub finishes rebooting after 4 minutes, it will start the timer again, and so on until it has not heard from a new subscriber for 5 minutes. If it takes you longer than 5 minutes to reboot a sub (which is actually quite likely with the amount of time SELinux takes to start up in 8.6) you might want to increase the timeout so you can get all the subscribers into the batch. You don't want to set it too large because you will have to wait that amount of time after the last subscriber contacts the pub before replication setup starts.

5. Reboot all the subs (backups first, then primaries)

6. Monitor replication setup on the publisher with 'utils dbreplication runtimestate'

Gordon's times are pretty accurate from my experience. Publisher is usually around 2 hours for an average sized system (YMMV) and subs are usually around 1:15 - 1:30 but you can do the subscribers all at once.

[Gordon Ross]

3. Once all the subscribers are upgraded, reboot the publisher (you can wait as long as you need before doing this). 

But doesn't the software install take a copy of the active partition DB and put it on the inactive partion ? So any changes you make to the system after this, will be lost after the switch version.

4. You might want to increase the repltimeout (utils dbreplication setrepltimeout). The default is 5 minutes (300 seconds). What this value does is tell the publisher how long to wait for additional subscribers before starting replication setup.

Never knew about that setting. Thanks !

GTG

[Aaron Harrison]

Hi Gordon

Yes - it does the DB migration when you run it. So don't leave it too long, or do a change freeze. You would typically not leave it long, this function allows you to do the upgrade slightly ahead of a maintenance window (as opposed to weeks before).

Paul - I recognise your name I think! Unless I have you mixed up, I still see your Troubleshooting IPT book as one of the best Cisco Press titles I've read. A little old now (CM4) but I learnt a huge amount from it! Thanks :-)

Aaron

[Paul Giralt]

Paul - I recognise your name I think! Unless I have you mixed up, I still see your Troubleshooting IPT book as one of the best Cisco Press titles I've read. A little old now (CM4) but I learnt a huge amount from it! Thanks :-)

Yes, that's me :-) Glad you enjoyed the book. Been working on an updated version for years, but haven't had the time to get it done.

[Aaron Harrison]

Well.. I still recommend the old one to people. It was a million miles away from some of the 're print the manual from CCO' routine that I've got from CP lately!

[Paul Giralt]

But doesn't the software install take a copy of the active partition DB and put it on the inactive partion ? So any changes you make to the system after this, will be lost after the switch version.

Yes - that is true. The install will make a copy of the DB on the publisher and SFTP it to the sub during the install process. After that point, no administrative changes will be reflected in the switch version, however in Unified CM 6.1 and later, all user-facing features are preserved after the switch, so anything like EM logins, speed dial changes, CFwdAll configuration, etc... (anything an end-user can do) will be retained after the switch version.

So just make sure there are no administrative changes made between the time the publisher is upgraded and the time the cluster is switched over.

[Gordon Ross]

One final thing, on a more general matter: Always upgrade the firmware on the phones before doing the system upgrade.

GTG

[Chris Deren]

Gordon,

"Always" is a strong word here :-)

I typically do not do that as with peer firmware sharing feature which I always enable it does not take that much time for the phones to upgrade even at remote locations across small WAN link. So, IMHO this is strictly a matter of preference or if the upgrade window is short and you cannot afford to worry about firmware upgrade during that time.  Just my personal opinion.

Chris