Migrating IP Phones Between Clusters with CUCM 8 and ITL Files

Document

Mar 24, 2011 10:41 AM
Mar 24th, 2011

Purpose

CUCM 8 introduced the new Security By Default feature and the use of ITL (Initial Trust List) files. (More documentation here). With this new feature, care must be taken when moving phones between different CUCM clusters. Without following the proper steps it is possible to encounter a situation where thousands of phones must manually have their ITL files deleted.

Background

Phones supporting the new ITL file download this special file from their CUCM TFTP server. Once an ITL file is installed on a phone, all future configuration files and ITL file updates must be either:

  1. Signed by the CCM+TFTP Server certificate currently installed in the phone's CTL file (If cluster security with CTLs is enabled).
  2. Signed by the CCM+TFTP Server certificate installed in the phone's ITL file.
  3. Signed by a certificate that exists in one of the CUCM Server TVS certificate stores that are listed in the ITL file.

With this new security functionality in mind here are the three problems that we can encounter when moving a phone from one cluster to another cluster.

  1. The ITL file of the new cluster is not signed by the phone's existing ITL CCM+TFTP cert, so the phone will not accept the new ITL file or config files.
  2. The TVS servers listed in the phone's existing ITL may not be reachable when the phones are moved to the new cluster.
  3. Even if the TVS servers are reachable for certificate verification, the old cluster TVS servers may not have the new servers' certificates.

If these three problems are encountered one possible option is to delete the ITL file manually from all phones being moved between clusters. This is not a desirable solution sine it requires massive effort as the number of phones increases.

Symptoms

Any changes that phones receive through TFTP or HTTP of configuration files will not be honored.  Configuration options passed by configuration file partially include:

  • URLs including Authentication URL, Directories URL, Services URL; this includes internal/external directories configuration
  • some locale features
  • callmanager groups for primary and secondary registration

The phone very likely will register, it registers to configured TFTP server by default. The phone likely will not register if the new TFTP server is not running the CallManager service.

When a phone has an incorrect ITL file for the current TFTP server the phone console logs show a message similar to:

1715: ERR 16:59:35.170584 SECD: EROR:verifyFile: sgn verify file failed </usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer not in CTL)

1716: ERR 16:59:35.171327 SECD: EROR:verifyFile: verify FAILED, </usr/ram/SEP00260BD749E9.cnf.xml>

Migration Options

With the three previous problems taken into account we can come up with a plan that allows the migration of phones seamlessly from one cluster to the next without requiring manual phone intervention.

1. Bulk Certificate Export

Note

The Bulk Certificate Export method will only work if both clusters are online with network connectivity while the phones are being migrated.

Another possible option if both the old and new clusters will be online at the same time is to use the Bulk Certificate migration method.

Remember that the IP Phones verify every downloaded file against either the ITL file, or against a TVS server that exists in the ITL file. If the phone needs to move to a new cluster, the ITL file the new cluster presents must be trusted by the old cluster's TVS certificate store.

The Bulk Certificate Export method works in the following way from the OS Adminstration > Security > Bulk Certificate page:

  1. Export certificates from new destination cluster (TFTP only) and original cluster to a central SFTP server.
  2. From original cluster, run Consolidate certificates (TFTP only) on the SFTP server using the Bulk Certificate interface.
  3. On the old origination cluster use the Bulk Certificate function to import the TFTP certificates from the central SFTP server.
  4. Restart TVS services on old origination cluster.
  5. Use DHCP option 150, or some other method, to point the phones to the new destination cluster.
  6. Phones will download the new destination cluster ITL file and attempt to verify it against their existing ITL file.
  7. The cert will not be in the existing ITL file so the phone will ask the old TVS server to verify the signature of the new ITL file. The phone sends a TVS query to the old origination cluster on TCP port 2445 to make this request.
  8. If the certificate export/consolidate/import process worked correctly then TVS returns success, and the phone replaces the in memory ITL file with the newly downloaded ITL file.
  9. The phones can now download and verify the signed configuration files from the new cluster.

2. Rollback Enterprise Parameter

Note

This method is only valid if completed before the phone migration is attempted. This cannot be used once phones are already in the "verify file failed" state.

Phones supporting TVS will potentially lose access to secure URL services such as the Corporate Directory before they are migrated to the new cluster after the "Prepare Cluster for Rollback to pre-8.0" parameter is set to True on the original cluster.  Once migrated to the new cluster, the phones will download their new ITLs and Secure URL operation should go back to normal.

Another option is to make use of the CUCM Enterprise Parameter "Prepare Cluster for Rollback to pre-8.0." Full documentation of the Rollback parameter can be found here. Once this parameter is set to True, the phones will download a special ITL file that contains empty TVS and TFTP certificate sections.

When a phone has an empty ITL file it will accept any unsigned configuration file (for migrations to pre CUCM 8.X clusters), and will also accept any new ITL file (for migrations to different CUCM 8.X clusters).

The empty ITL file can be verified by checking "Settings > Security > Trust List > ITL". Empty entries will appear where the old TVS and TFTP servers used to be.

The phones must have access to the old CUCM servers only as long as it takes them to download the new empty ITL files. Once the phone has an empty ITL file, the old servers can be decommissioned, powered down, thrown into a river, or rebuilt (depending on business requirements).

3. Using Hardware Security tokens (KEY-CCM-ADMIN-K9=)

If hardware security tokens (product number KEY-CCM-ADMIN-K9=) have been used to generate a CTL (Certificate Trust List) on both the old and new clusters, the phones will be able to freely migrate between clusters as long as at least one of the same hardware tokens was used on both the old and new clusters.

When a phone that has a CTL from the old cluster is moved to the new cluster, it will accept the new cluster's CTL since the new CTL contains a security token certificate in common with it's current CTL. Because the CTL also contains the certificate for the CCM+TFTP server, the new cluster's ITL will also be accepted by the phone so there will be no issues in moving the phone between clusters.

For phones that don't utilize security by default (ITLs) such as the 7960s and 7940s, you will need to re-run the CTL client on the original cluster first to add new TFTP entries for the TFTP servers of the new cluster prior to moving the phones to the new cluster.  This is due to those phone models not even reaching out for TFTP files for a server not in the CTL.

This security token method requires this additional hardware and must be configured on the old cluster. Normally security tokens are used to allow for encrypted signaling and media (SRTP) in a cluster and encrypted / authenticated config files. Also, once a cluster has had security enabled with security tokens, disabling security on that cluster requires manually removing the CTL from each phone on that cluster from the phone itself.

4. Manual ITL Delete

If some catastrophic failure happens and the TFTP key/certificate is no longer available from the old cluster (this is maintained in a DRF backup... TAKE A BACKUP), then the only available option to migrate a phone to a new cluster is to manually delete the ITL file from the phones.

This differs for each phone model.  The steps needed for the most common phone models are listed here, but the steps for other models can be found in the Phone Administration Guides.

79XX - Settings > Security > Trust List > ITL File > **# (to unlock the settings) > Erase

89XX/99XX - Settings > Administrator Settings > Reset Settings > Security Settings

Average Rating: 5 (4 ratings)

Comments

gabrielmarino Mon, 02/06/2012 - 09:54

Hi to all ! Thanks for this explanation. Can this happen when I make an update from 8.5 (SU2)  to 8.5 (SU3) ?..

because we already did it for one Cluster and this is happening for us right now.

For update our next Clusters, can we use the method "Rollback Enterprise Parameter"?  I mean, can we do a Rollback from version 8.5 (SU3) to 8.5 (SU2)?

Many thanks

Gabriel

stephenwelsh Mon, 02/06/2012 - 17:36 (reply to gabrielmarino)

Hi Gabriel,

If you have some phones that already have an ITL "issue" (i.e. they do not accept the TFTP configuration or ITL updates, you can check the log entry by browsing to the phones web server). Then the "rollback will not fix the problem", see the quote from the doc above:

This method is only valid if completed before the phone migration is attempted. This cannot be used once phones are already in the "verify file failed" state.

                   

You could use the roll-back as a preventative measure to clear the ITL file from all working phones to ensure they get the ITL file from the updated cluster, however I don't think this would be classed as best practice as it's not the intended purpose for the setting.

I recomend you also review the following document on managing ITL Files:

Thanks

Stephen

johnpgowen Tue, 04/24/2012 - 03:41

Hi,

We are upgrading from 7.1.5 (MCS) to 8.6.2 (UCS), with change of CUCM IP Address, DNS, Hostname etc etc.

Is there a "best practise" supported method for our 9951 migration and reseting the security settings.

Does the Bulk Certificate Export method work for 7.1.5 to 8.6.2 as both clusters will be online in parallel (the document states its for switching between two clusters of V8).

Thanks,

John

stevenluton Mon, 11/18/2013 - 10:46 (reply to johnpgowen)

You shouldn't have to use the bulk certificate method from 7 to 8.  You may want to use the rollback option in case you need to roll back to your version 7 if testing fails on version 8.  If testing on version 8 passes, uncheck the prepare for rollback, reset the required services and shut down you version 7 server.

brmeade Mon, 11/18/2013 - 13:00 (reply to stevenluton)

Steven,

After you export both clusters to the same SFTP server to the same directory, you should get the option to run a Consolidate.  Once it detects the consolidate has been done, it will then give you the import option.

Brian

stevenluton Mon, 11/18/2013 - 13:16 (reply to brmeade)

Thank you Brian.  It does not specifically state those instructions in the directions on Cisco's documents. Do I need to reset the TVS service after it successfully imports?

brmeade Mon, 11/18/2013 - 13:24 (reply to stevenluton)

Steven,

I would restart TVS if possible.  I always do it on my clusters when doing this.  I've never tried it without restarting TVS though so it may work fine.

Thanks,

Brian

stevenluton Mon, 11/18/2013 - 13:44 (reply to brmeade)

That works.  I truly appreciate your input Brian.  Now if we can only get Cisco to fix their documentation.

Thanks,

Steve

stevenluton Mon, 11/18/2013 - 10:40

I am trying to lab out the Bulk Certificate Export situation.  I can export the needed TFTP certificate but I can't import it onto the the current CUCM server.  When I export I get a .pkcs12 file.  Does this need to be changed?  When I attach the SFTP server to the production CUCM I see the file but there isn't any Import button on that page.  Could you please respond.

Thanks,

Steve

Artmeister24 Thu, 04/24/2014 - 12:53

Jason - Thanks for posting such a helpful guide. This will prove to be very helpful for an upcoming migration. One question on the "Rollback Enterprise Paramater" option: assuming that we decomission the servers, and throw them in a river, is it still possible to re-build them after recovering and thoroughly drying? I heard that worked on the MCS servers, but not on the UCS?

Actions

Login or Register to take actions

This Document

Posted March 24, 2011 at 10:41 AM
Stats:
Comments:14 Avg. Rating:5
Views:36099 Contributors:9
Shares:0
Tags: No tags.

Documents Leaderboard