Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 

Communications Manager Security By Default and ITL Operation and Troubleshooting

 

 

Purpose

Communications Manager 8.0 and later introduce the Security By Default (SBD) feature, consisting of Identity Trust List (ITL) files and the Trust Verification Service (TVS). Every Communications Manager (CM) cluster now uses ITL-based security automatically. There is a trade off between security and ease of use / ease of administration that administrators must be aware of before making certain changes to an 8.0 CM cluster.

 

This document is an effort to supplement the official Security By Default documents and provide operational info and troubleshooting tips to help administrators and ease the troubleshooting process.

 

Before we get started it would be good to be familiar with Public Key Infrastructure and Asymmetric Key Cryptography. These concepts are at the core of Security By Default.

 

SBD Overview (Why?)

 

This section is intended to give a quick overview of exactly what Security By Default can provide. For full technical details of each function see the Detail and Troubleshooting section.

 

Security By Default provides these three functions for supported IP Phones:

 

  1. Default authentication of TFTP downloaded files (configuration, locale, ringlist, etc) using a signing key.
  2. Optional encryption of TFTP configuration files using a signing key.
  3. Certificate verification for phone initiated HTTPS connections using a remote certificate trust store on Communications Manager (Trust Verification Service).

 

Let's take a look at each of these functions in an overview now, and dive into the details in the next section.

 

TFTP Download Authentication

 

When a CTL or ITL file is present, the IP Phone requests a signed TFTP configuration file from the CUCM TFTP server. This allows the phone to verify the configuration file came from a trusted source. With CTL / ITL files present on phones, configuration files must be signed by a trusted TFTP server. The file is plain text on the network while being transmitted, but comes with a special verification signature.

 

The phone requests SEP<MAC Address>.cnf.xml.sgn to get the configuration file with the special signature. This configuration file has been signed by the TFTP private key corresponding to CallManager.pem on the OS Administration Certificate Management page.

TFTPAuth-overview.png

 

The signed file has a signature at the top to authenticate the file, but is otherwise in plain text xml. We can see that the signer of the config file is "CN=CUCM8-Publisher.bbbburns.lab" which is in turn signed by "CN=JASBURNS-AD". This means the phone needs to verify the signature of CUCM8-Publisher.bbbburns.lab against the ITL file before this config file will be accepted.

TFTP-config-signed.png

 

Here is a diagram showing how the private key is used along with an MD5 or SHA1 hash function to to create the signed file

 

FileSigning.png

 

Signature verification reverses this process by using the matching public key to decrypt the hash. If the hashes match then we know

 

  1. This file has not been modified in transit
  2. This file comes from the party listed in the signature, since anything decrypted successfully with the public key must have been encrypted with the private key.

 

FileVerify.png

 

TFTP Configuration File Encryption

If optional TFTP configuration encryption is enabled (in the associated Phone Security Profile) then the phone will request an encrypted configuration file. This file is signed with the TFTP private key and encrypted with a symmetric key exchanged between the phone and CM (see the CM Security Guide for full details) so its contents cannot be read with a network sniffer unless the observer has the necessary keys.

 

The phone requests SEP<MAC Address>.cnf.xml.enc.sgn to get the signed an encrypted file.

 

TFTPEnc-overview.png

 

The encrypted config file has the signature at the beginning as well, but there is no plain text data after. Only encrypted data (garbled binary  characters in this text editor). We see the signer is the same as in the previous example, so this signer must be present in the ITL file before the phone will accept the file. Further, the decryption keys must be correct before the phone can read the contents of the file.

TFTP-config-enc.png

 

Trust Verification Service (Remote certificate and signature verification)

IP Phones contain a limited amount of memory and there can also be a large number of phones to manage in a network. Instead of putting a full certificate trust store on each and every IP phone, CM acts as a remote trust store via the Trust Verification Service (TVS). Any time the phone cannot verify a signature or certificate via the CTL or ITL files, it will ask the TVS server for verification. This central trust store is easier to manage than if the trust store was present on all IP Phones.

TVS-overview.png

SBD Detail and Troubleshooting Information (How?)

Now that we've seen what benefits we get once ITL files are present on the phone, let's go over how exactly the process happens.

ITL Files and Certificates Present on CM

First, there are a number of files that must be present on the CM server itself. The most important piece is the TFTP certificate and TFTP private key. We can see the TFTP Certificate under "OS Administration > Security > Certificate Management > CallManager.pem"

 

The CUCM server uses the CallManager.pem certificate's private and public keys for the TFTP service (as well as the CCM service). Here we can see that the CallManager.pem certificate is issued to CUCM8-publisher.bbbburns.lab and signed by JASBURNS-AD. All TFTP configuration files will be signed by the private key below.

 

All phones can use the TFTP public key in the CallManager.pem certificate to decrypt any file encrypted with the TFTP private key, as well as to verify any file signed with the TFTP private key.

CallManager-pem.png

 

In addition to the CallManager.pem certificate private key, the CM server also stores an Identity Trust List file which is presented to phones. We can view the full contents of this ITL file via SSH access to the CM server OS Command Line Interface using "show itl".

 

Let's break down the ITL file piece by piece, because it has a number of important components that the phone uses.

 

The first portion is the signature information. Even the ITL file is a signed file. Here we see it has been signed by the TFTP private key that is associated with the CallManager.pem certificate above.

 

 admin:show itl
 Length of ITL file: 5438
 The ITL File was last modified on Wed Jul 27 10:16:24 EDT 2011

 

         Parse ITL File
        ----------------
Version:        1.2
HeaderLength:   296 (BYTES)
BYTEPOS TAG             LENGTH  VALUE
------- ---             ------  -----
3       SIGNERID        2       110
4       SIGNERNAME      76      CN=CUCM8-Publisher.bbbburns.lab;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
5       SERIALNUMBER    10      21:00:2D:17:00:00:00:00:00:05
6       CANAME          15      CN=JASBURNS-AD
*Signature omitted for brevity*

 

The next sections each contain their purpose inside of a special "Function" parameter. Here is the first function of System Administrator Security Token. This is the signature of the TFTP public key.

 

        ITL Record #:1
                  ----
BYTEPOS TAG             LENGTH  VALUE
------- ---             ------  -----
1       RECORDLENGTH    2       1972
2       DNSNAME         2
3       SUBJECTNAME     76      CN=CUCM8-Publisher.bbbburns.lab;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
4       FUNCTION        2       System Administrator Security Token
5       ISSUERNAME      15      CN=JASBURNS-AD
6       SERIALNUMBER    10      21:00:2D:17:00:00:00:00:00:05
7       PUBLICKEY       140
8       SIGNATURE       256
9       CERTIFICATE     1442    0E 1E 28 0E 5B 5D CC 7A 20 29 61 F5 8A DE 30 40 51 5B C4 89 (SHA1 Hash HEX)
This etoken was used to sign the ITL file.

 

The next function is CCM+TFTP. This is again the TFTP certificate that will serve to authenticate and decrypt downloaded TFTP configuration files. On a "Non-Secure" cluster this function will only list "TFTP". When a cluster is set to "Mixed-Mode" the function will display "CCM+TFTP" as shown below.

 

         ITL Record #:2
                  ----
BYTEPOS TAG             LENGTH  VALUE
------- ---             ------  -----
1       RECORDLENGTH    2       1972
2       DNSNAME         2
3       SUBJECTNAME     76      CN=CUCM8-Publisher.bbbburns.lab;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
4       FUNCTION        2       CCM+TFTP
5       ISSUERNAME      15      CN=JASBURNS-AD
6       SERIALNUMBER    10      21:00:2D:17:00:00:00:00:00:05
7       PUBLICKEY       140
8       SIGNATURE       256
9       CERTIFICATE     1442    0E 1E 28 0E 5B 5D CC 7A 20 29 61 F5 8A DE 30 40 51 5B C4 89 (SHA1 Hash HEX)

 

 

 

The next function is TVS. There will be an entry for the public key of each TVS server the phone can connect to. This allows the phone to establish a secure SSL session to the TVS server.

 

         ITL Record #:3
                  ----
BYTEPOS TAG             LENGTH  VALUE
------- ---             ------  -----
1       RECORDLENGTH    2       743
2       DNSNAME         2
3       SUBJECTNAME     76      CN=CUCM8-Publisher.bbbburns.lab;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
4       FUNCTION        2       TVS
5       ISSUERNAME      76      CN=CUCM8-Publisher.bbbburns.lab;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
6       SERIALNUMBER    8       2E:3E:1A:7B:DA:A6:4D:84
7       PUBLICKEY       270
8       SIGNATURE       256
11      CERTHASH        20      C7 E1 D9 7A CC B0 2B C2 A8 B2 90 FB AA FE 66 5B EC 41 42 5D
12      HASH ALGORITHM  1       SHA-1

 

 

 

The final function included in the ITL file is CAPF (Certificate Authority Proxy Function). This certificate allows the phones to establish a secure connection to the CAPF service on the CM server so the phone can install or update an LSC (Locally Significant Certificate). We'll cover that in another document (Not written yet, but coming soon).

 

         ITL Record #:4
                  ----
BYTEPOS TAG             LENGTH  VALUE
------- ---             ------  -----
1       RECORDLENGTH    2       455
2       DNSNAME         2
3       SUBJECTNAME     61      CN=CAPF-9c4cba7d;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
4       FUNCTION        2       CAPF
5       ISSUERNAME      61      CN=CAPF-9c4cba7d;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
6       SERIALNUMBER    8       0A:DC:6E:77:42:91:4A:53
7       PUBLICKEY       140
8       SIGNATURE       128
11      CERTHASH        20      C7 3D EA 77 94 5E 06 14 D2 90 B1 A1 43 7B 69 84 1D 2D 85 2E
12      HASH ALGORITHM  1       SHA-1
The ITL file was verified successfully.

 

Now that we know the structure of the certificate and ITL file on the CUCM server, let's cover exactly what happens when a phone boots.

 

Phone Downloads ITL and Config File

After the phone boots, obtains an IP address, and the address of a TFTP server, the first files it will ask for are the CTL and the ITL file.

 

Here is the phone requesting the ITL file via a packet capture. If we filter on tftp.opcode == 1 we can see every TFTP Read Request from the phone:

TFTPReadRequests.png

 

Next we see that since the phone received a CTL and ITL file from TFTP successfully, the phone asks for a signed configuration file. We can get the phone console logs showing this behavior from the phone's web interface:

ConsoleLogs.png

 

First the phone requests a CTL file, which succeeds:

837: NOT 09:13:17.561856 SECD: tlRequestFile: Request CTLSEP0011215A1AE3.tlv
846: NOT 09:13:17.670439 TFTP: [27]:Requesting CTLSEP0011215A1AE3.tlv from 14.48.44.80
847: NOT 09:13:17.685264 TFTP: [27]:Finished --> rcvd 4762 bytes

 

Next the phone also requests an ITL file:

868: NOT 09:13:17.860613 TFTP: [28]:Requesting ITLSEP0011215A1AE3.tlv from 14.48.44.80
869: NOT 09:13:17.875059 TFTP: [28]:Finished --> rcvd 5438 bytes

 

Phone Verifies ITL and Config File

After the ITL file is downloaded, it  must be verified. There are a number of states that a phone can be in at  this point - so let's cover them all.

 

  1. The phone has no CTL or ITL file present (or ITL is blank because of "Prepare Cluster for Rollback to Pre 8.0" parameter)
    1. The phone will blindly trust the next CTL or ITL file downloaded and use this signature moving forward.
  2. The phone already has a CTL but no ITL
    1. The phone will only trust an ITL if it can be verified by the CCM+TFTP function in the CTL file.
  3. The phone already has a CTL and an ITL file
    1. The phone will verify that the recently downloaded files match the signature in either the CTL, ITL, or TVS server.

 

Here is a flow chart that describes how the phone verifies signed files and https certificates:

 

PhoneAuthenticationFlowChart.png

 

In this case we see the phone was able to verify the signature in the ITL and CTL files. The phone already had both a CTL and ITL so it just checked against the CTL and ITL and found the correct signature.

877: NOT 09:13:17.925249 SECD: validate_file_envelope: File sign verify SUCCESS; header length <296>

 

Since the phone downloaded a CTL and ITL file, it will from this point on ONLY request signed configuration files. We can see the phone logic determining that the TFTP server is secure (based on the presence of CTL and ITL) and then ask for a signed file:

917: NOT 09:13:18.433411 tftpClient: tftp request rcv'd from /usr/tmp/tftp, srcFile = SEP0011215A1AE3.cnf.xml, dstFile = /usr/ram/SEP0011215A1AE3.cnf.xml max size = 550001
918: NOT 09:13:18.457949 tftpClient: auth server - tftpList[0] = ::ffff:14.48.44.80
919: NOT 09:13:18.458937 tftpClient: look up server - 0
920: NOT 09:13:18.462479 SECD: lookupCTL: TFTP SRVR secure
921: NOT 09:13:18.466658 tftpClient: secVal = 0x9
922: NOT 09:13:18.467762 tftpClient: ::ffff:14.48.44.80 is a secure server
923: NOT 09:13:18.468614 tftpClient: retval = SRVR_SECURE
924: NOT 09:13:18.469485 tftpClient: Secure file requested
925: NOT 09:13:18.471217 tftpClient: authenticated file approved - add .sgn -- SEP0011215A1AE3.cnf.xml.sgn
926: NOT 09:13:18.540562 TFTP: [10]:Requesting SEP0011215A1AE3.cnf.xml.sgn from 14.48.44.80 with size limit of 550001
927: NOT 09:13:18.559326 TFTP: [10]:Finished --> rcvd 7652 bytes 

 

Once the signed configuration file is downloaded the phone must authenticate it against the Function for CCM+TFTP inside the ITL:

937: NOT 09:13:18.656906 SECD: verifyFile: verify SUCCESS </usr/ram/SEP0011215A1AE3.cnf.xml>

 

Phone Contacts Trust Verification Service for Unknown Certificate

The ITL file also provides a TVS function which contains the certificate of the TVS service running on CM server TCP port 2445. TVS runs on all servers in the cluster as a Network Service. The CM TFTP service uses the configured CallManager group to build a list of TVS servers the phone should contact on the phone configuration file.

My lab uses only a single CM server. In a multi-node CM cluster there can be up to three TVS entries for a phone, one for each CM in the CM Group of the phone.

The phone searches in the following order to contact a functional TVS server:

  1. CallManager Group Server 1
  2. CallManager Group Server 2 (if present)
  3. CallManager Group Server 3 (if present)
  4. TFTP Server - Last Resort

 

In this example I press the "Directories" button on the IP Phone. The Directories URL is configured for https, so the phone is presented with the Tomcat web certificate from the Directories server. This Tomcat web certificate (tomcat.pem in OS Administration) is not loaded in the phone, so the phone must contact TVS to authenticate the certificate.

 

Refer to the TVS Overview diagram above for the interaction. Here is the phone console log perspective:

 

First we find the Directory URL

1184: NOT 15:20:55.219275 JVM: Startup Module Loader|cip.dir.TandunDirectories:? - Directory url https://14.48.44.80:8443/ccmcip/xmldirectory.jsp

 

 

Next we realize this will be a SSL/TLS secure http session that requires verification

1205: NOT 15:20:59.404971 SECD: clpSetupSsl: Trying to connect to IPV4, IP: 14.48.44.80, Port : 8443
1206: NOT 15:20:59.406896 SECD: clpSetupSsl: TCP connect() waiting, <14.48.44.80> c:8 s:9 port: 8443
1207: NOT 15:20:59.408136 SECD: clpSetupSsl: TCP connected, <14.48.44.80> c:8 s:9
1208: NOT 15:20:59.409393 SECD: clpSetupSsl: start SSL/TLS handshake, <14.48.44.80> c:8 s:9
1209: NOT 15:20:59.423386 SECD: srvr_cert_vfy: Server Certificate Validation needs to be done 

 

 

The phone will first look to see if the certificate presented by the SSL / TLS server is present in the CTL. Then the phone looks at the Functions in the ITL file to see if it finds a match. The error message below says "HTTPS cert not in CTL", but really means "I couldn't find that cert in the CTL or the ITL.

 

 

1213: NOT 15:20:59.429176 SECD: findByCertAndRoleInTL: Searching TL from CTL file
1214: NOT 15:20:59.430315 SECD: findByCertAndRoleInTL: Searching TL from ITL file
1215: ERR 15:20:59.431314 SECD: EROR:https_cert_vfy: HTTPS cert not in CTL, <14.48.44.80> 

 

Now that we've checked the direct contents of the CTL and ITL file for the certificate, the next thing the phone checks is the TVS cache. This is to cut down on network traffic if the phone has recently asked the TVS server for this very same cert. If the HTTPS cert isn't found in the phone cache, we then make a TCP connection to the TVS server itself.

 

 

1220: NOT 15:20:59.444517 SECD: processTvsClntReq: TVS Certificate Authentication request
1221: NOT 15:20:59.445507 SECD: lookupAuthCertTvsCacheEntry: No matching entry found at cache
1222: NOT 15:20:59.446518 SECD: processTvsClntReq: No server sock exists, must be created
1223: NOT 15:20:59.451378 SECD: secReq_initClient: clnt sock fd 11 bound to </tmp/secClnt_secd>
1224: NOT 15:20:59.457643 SECD: getTvsServerInfo: Phone in IPv4 only mode
1225: NOT 15:20:59.458706 SECD: getTvsServerInfo: Retreiving IPv4 address
1230: NOT 15:20:59.472628 SECD: connectToTvsServer: Successfully started a TLS connection establishment to the TVS server: IP:14.48.44.80, port:2445(default); Waiting for it to get connected.

 

 

Remember that the connection to TVS itself is SSL/TLS (Secure http, or HTTPS), so it is also a certificate that needs to be authenticated against the CTL ot ITL. If everything goes correctly we should find the TVS server's certificate in the TVS function of the ITL file. ITL Record #3 in the example ITL file above.

 

 

1244: NOT 15:20:59.529938 SECD: srvr_cert_vfy: Server Certificate Validation needs to be done
1245: NOT 15:20:59.533412 SECD: findByIssuerAndSerialAndRoleInTL: Searching TL from CTL file
1246: NOT 15:20:59.534936 SECD: findByIssuerAndSerialAndRoleInTL: Searching TL from ITL file
1247: NOT 15:20:59.537359 SECD: verifyCertWithHashFromTL: cert hash and hash in TL MATCH
1248: NOT 15:20:59.538726 SECD: tvs_cert_vfy: TVS cert verified with hash from TL, <14.48.44.80>

 

Success! The phone now has a secure connection to the TVS server. The next step is to ask the TVS server "Hello, do I trust this Directories server certificate?"

 

 

We ask TVS, and get a response of 0 which means success. (No error)

 

1264: NOT 15:20:59.789738 SECD: sendTvsClientReqToSrvr: Authenticate Certificate : request sent to TVS server - waiting for response
1273: NOT 15:20:59.825648 SECD: processTvsSrvrResponse: Authentication Response received, status : 0

 

Since we have a successful response from TVS we save the result for that certificate into cache. This means for the next 86400 seconds if we hit Directories again we won't need to contact the TVS server to verify the cert, we'll access the local cache.

 

1279: NOT 15:20:59.837086 SECD: saveCertToTvsCache: Saving certificate in TVS cache with default time-to-live value: 86400 seconds
1287: ERR 15:20:59.859993 SECD: Authenticated the HTTPS conn via TVS 

 

 

 

And then finally full circle to say that our connection to the Directories server succeeded

 

1302: ERR 15:21:01.959700 JVM: Startup Module Loader|cip.http.ae:? - listener.httpSucceed:  https://14.48.44.80:8443/ccmcip/xmldirectoryinput.jsp?name=SEP0011215A1AE3

 

 

Let's also take a quick look at what happens on the CM server where TVS is running. You can collect TVS logs using RTMT.

 

TVS-trace-config.png

TVS-trace-collection.png

 

The CM TVS logs show that we SSL handshake with the phone, the phone asks TVS about the Tomcat certificate, then TVS responds saying the cert was matched in the TVS certificate store.

 

 

15:21:01.954 |   debug 14.48.44.202: tvsSSLHandShake Session ciphers - AES256-SHA

15:21:01.954 |   debug TLS HS Done for ph_conn .

15:21:02.010 |   debug     MsgType               : TVS_MSG_CERT_VERIFICATION_REQ

15:21:02.011 |   debug tvsGetIssuerNameFromX509 - issuerName : CN=CUCM8-Publisher.bbbburns.lab;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US and Length: 75

 

15:21:02.011 |   debug CertificateDBCache::getCertificateInformation - Certificate compare return =0

15:21:02.011 |   debug CertificateDBCache::getCertificateInformation -Certificate found and equal

15:21:02.011 |   debug     MsgType               : TVS_MSG_CERT_VERIFICATION_RES

 

The TVS certificate store is a list of all certificates contained on the OS Administration > Certificate Management web page.

 

Manually Verifying Phone ITL matches CM ITL (or "Don't delete your ITL!")

One common misconception seen while troubleshooting is the urge to delete the ITL file in the hopes that it will resolve a file verification problem. Sometimes ITL file deletion is required, but there may be a better way.

 

The ITL file only ever needs to be deleted when ALL of the following conditions are met.

 

  1. The signature of the ITL file on the phone does not match the signature of the ITL file on the CM TFTP server.
  2. The TVS signature in the ITL file does not match the certificate presented by TVS.
  3. The phone shows "Verification Failed" when attempting to download the ITL file or config files.
  4. No backup exists of the old TFTP private key.

 

Let's go over how to check the first two of these.

 

First, we can compare the checksum of the ITL file present on CUCM with the checksum ITL file of the phone. There currently isn't a way to look at the md5sum of the ITL file on CUCM from CUCM itself until running a version with the fix for this defect CSCto60209.

 

In the interim run the following with your favorite GUI or CLI programs:

 

 

jasburns@jasburns-gentoo /data/trace/jasburns/certs/SBD $ tftp 14.48.44.80

tftp> get ITLSEP0011215A1AE3.tlv

Received 5438 bytes in 0.0 seconds

tftp> quit

jasburns@jasburns-gentoo /data/trace/jasburns/certs/SBD $ md5sum ITLSEP0011215A1AE3.tlv

b61910bb01d8d3a1c1b36526cc9f2ddc  ITLSEP0011215A1AE3.tlv

 

So we see the md5sum of the ITL file in CUCM is b61910bb01d8d3a1c1b36526cc9f2ddc

 

Now we can look at the phone itself to see what the hash of the ITL file loaded there is

 

Settings > Security Configuration > Trust List

ITL-md5sum-phone.bmp

 

We see here that the md5sums match! This means the ITL file on the phone matches the file on the CUCM, so it doesn't need to be deleted.

 

If it DOES match then we need to move on to the next operation of checking whether or not the TVS certificate in the ITL matches the cert presented by TVS. This is a bit more involved.

 

First we look at the packet capture of the phone connecting to the TVS server on TCP port 2445.

 

Right click on any packet in this stream in Wireshark and click "Decode As" and select "SSL". Find the Server Certificate that looks like the following:

 

TVS-certificate-capture.png

 

Then we look at the TVS certificate contained within the ITL file above. We should see an entry with the serial number 2E3E1A7BDAA64D84.

 

admin:show itl
        ITL Record #:3
                  ----
BYTEPOS TAG             LENGTH  VALUE
------- ---             ------  -----
1       RECORDLENGTH    2       743
2       DNSNAME         2
3       SUBJECTNAME     76      CN=CUCM8-Publisher.bbbburns.lab;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
4       FUNCTION        2       TVS
5       ISSUERNAME      76      CN=CUCM8-Publisher.bbbburns.lab;OU=TAC;O=Cisco;L=RTP;ST=North Carolina;C=US
6       SERIALNUMBER    8       2E:3E:1A:7B:DA:A6:4D:84

 

 

 

Success, the TVS.pem inside of the ITL file matches the TVS certificate presented on the network. We don't need to delete the ITL, and TVS is presenting the correct certificate.

 

If file authentication is still failing, check the rest of the flow chart above.

 

Restrictions and Interactions

Regenerating Certificates / Rebuilding a Cluster / Certificate Expiry

The most important certificate is now the CallManager.pem certificate. This certificate's private key is used to sign all TFTP configuration files, including the ITL file.

 

If the CallManager.pem file is regenerated, a new CCM+TFTP certificate will be generated with a new private key. Additionally the ITL file will now be signed by this new CCM+TFTP key.

 

After regenerating CallManager.pem and restarting the TVS and TFTP service the following will happen when a phone boots.

 

  1. The phone will attempt to download the new ITL file signed by the new CCM+TFTP from the TFTP server.
  2. The phone only has the old ITL file at this point, and the new keys are not in the ITL file present on the phone.
  3. Since the phone could not find the new CCM+TFTP signature in the old ITL, it will attempt to contact the TVS service.
    1. This part is extremely important. The TVS certificate from the old ITL file must still match. If both the CallManager.pem and TVS.pem are regenerated at the same exact time, then phones will not be able to download any new files without deleting the ITL from the phone manually.
  4. When the phone contacts TVS, the CM server running TVS will have the new CallManager.pem certificate in the OS Certificate Store.
  5. The TVS server will return success and the phone will load the new ITL file into memory.
  6. The phone will now attempt to download a configuration file, which has been signed by the new CallManager.pem key.
  7. Since the new ITL has been loaded, the newly signed config file will be successfully verified by the ITL in memory.

 

Key Points:

  • Never regenerate both the CallManager.pem and TVS.pem certificates at the same time.
  • If either TVS.pem or CallManager.pem is regenerated, TVS and TFTP should be restarted and phones reset to get the new ITL files. Newer versions of CUCM will handle this phone reset automatically and warn the user at certificate regeneration time.
  • If more than one TVS server exists (more than one server in the CallManager Group), the additional servers can authenticate the new CallManager.pem certificate.                           

Moving Phones Between Clusters

When moving phones from one cluster to another with ITLs in place, the ITL and TFTP Private Key must be taken into account. Any new configuration file presented to the phone MUST match a signature in CTL, ITL, or a signature in the phone's existing TVS service.

 

The document below explains how to make sure the new cluster's ITL file and config files can be trusted by the existing ITL file on the phone.

 

https://supportforums.cisco.com/docs/DOC-15799

Backup And Restore

The CallManager.pem certificate and private key are backed up via DRS.  If a TFTP Server is rebuilt it MUST be restored from backup so the private key can be restored. Without this CallManager.pem private key on the server, phones with existing ITLs using the old key will not trust signed config files.

 

If a cluster is rebuilt and not restored from backup, it will be exactly like the "Moving Phones Between Clusters" document above. This is because a cluster with a new key is a different cluster as far as the phones are concerned.

 

There is one very serious defect associated with backup and restore. If a cluster is susceptible to defect CSCtn50405, the DRS backups will not contain the CallManager.pem certificate. This will cause any server restored from this backup to generate corrupt ITL files until a new CallManager.pem is generated. If there are no other working TFTP servers that did not go through the backup and restore operation, then this may mean all ITL files need to be deleted from the phones.

 

To verify if your CallManager.pem file needs to be regenerated please run

 

show itl

 

followed by

 

run sql select c.subjectname, c.serialnumber, c.ipv4address, t.name from  certificae as c, certificatetrustrolemap as r, typetrustrole as t where  c.pkid = r.fkcertificate and t.enum = r.tktrustrole 

 

In the ITL output the key errors to look for are:

 

This etoken was not used to sign the ITL file.

 

and

 

Verification of the ITL file failed.
Error parsing the ITL file!!

 

In the SQL query above, we're looking for the certificates that have a role of "Authentication and Authorization". The CallManager.pem certificate in the database query above that has the role of Authentication and Authorization should ALSO be present in the OS Administration Certificate Management web page. If the defect above is encountered, there will be a mismatch between the CallManager.pem certificates in the query and in the OS web page.

Changing Host Names or Domain Names

Changing the hostname or domain name of a CM server regenerates all certificates at once on that server. In the certificate regeneration section above we learned that regenerating both TVS.pem and CallManager.pem is a "bad thing".

 

There are a few scenarios where changing a hostname will fail, and a few where it will work without problems. Let's cover all of them and link them back to the what we already know about TVS and ITL from above:

 

  1. Single Node Cluster using only ITL (use caution, this will break without preparation)
    1. With a Business Edition server or publisher only deployment, both the CallManager.pem and TVS.pem are regenerated at the same time during a hostname change.
    2. If the hostname is changed on a single node cluster without first using the Rollback Enterprise parameter covered here, the phones will not be able to verify the new ITL file or config files against their existing ITL file. Additionally, they will not be able to connect to TVS because the TVS cert is also no longer trusted.
    3. The behavior displayed will be that the phones will display an error about "Trust List Verification Failed", no new config changes will take effect, and secure service URLs will fail.
    4. The only solution if the precaution in step 2 isn't first taken is to manually delete the ITL from every phone.
  2. Single Node Cluster using both CTL and ITL (this can be temporarily broken, but easily fixed)
    1. After running through the rename of servers, re-run the CTL client. This will place the new CallManager.pem certificate in the CTL file that the phone downloads.
    2. New config files (including new ITL files) can be trusted based on the CCM+TFTP function in the CTL file.
    3. This works because the updated CTL file is trusted based on a USB eToken private key that remains the same no matter what.
  3. Multi Node Cluster using only ITL (this generally works, but can be permanently broken if done hastily)
    1. Because a multinode cluster has multiple TVS servers, any single server can have its certificates regenerated without a problem. When the phone is presented with this new unfamiliar signature it will ask another of the TVS servers which can verify the new server certificate.
    2. There are two main problems that can cause this to fail
      1. If all servers are renamed and rebooted at the same time, none of the TVS servers will be reachable with known certificates when the servers and phones come back up.
      2. If a phone has only a single server in the CallManager Group, the additional TVS servers make no difference. See the "Single Node Cluster" scenario to resolve this, or add another server to the phone's CallManager Group.
  4. Multi Node Cluster using both CTL and ITL (this cannot be permanently broken)
    1. After running through the renames the TVS service will authenticate the new certificates.
    2. Even if all TVS servers were unavailable for some reason, the CTL client can still be used to update the phones with the new CallManager.pem CCM+TFTP certificates.

Centralized TFTP

When a phone with an ITL boots it requests the file "CTLSEP<MAC Address>.tlv" and "ITLSEP<MAC Address>.tlv as well as SEP<MAC Address>.cnf.xml.sgn.

 

If the phone cannot find these files it requests ITLFile.tlv and CTLFile.tlv, which a centralized TFTP server will provide to any phone that requests it.

 

With centralized TFTP there is a single TFTP cluster that points to a number of other sub clusters. Often this is done because phones on multiple CUCM clusters share the same DHCP scope, therefore must have the same DHCP Option 150 TFTP server. All IP Phones point to the central TFTP cluster, even if they register to other clusters. This central TFTP server queries the remote TFTP servers whenever it receives a request for a file it cannot find.

 

Because of this operation, centralized TFTP will only work in an ITL homogeneous environment. All servers must be running CUCM 8.x or above, or all servers must be running versions prior to 8.x.

 

If an ITLFile.tlv is presented from the Centralized TFTP server, phones will not trust any files from the remote TFTP server as the signatures will not match. This happens in a heterogenous mix. In a homogeneous mix the phone will request ITLSEP<MAC>.tlv which will be pulled from the correct remote cluster.

 

In a heterogeneous environment with a mix of pre-8.x and 8.x clusters, the "Prepare Cluster for Rollback to Pre 8.0" must be enabled on the 8.x cluster as per defect CSCto87262 and the "Secured Phone URL Parameters" configured with http instead of https. This effectively disables the ITL functions on the phone.

 

Frequently Asked Questions

Can I turn off Security By Default?

Only if SBD and ITL is currently working.

 

Security By Default can be temporarily disabled on phones by using the "Prepare Cluster for Rollback to pre 8.0" Enterprise Parameter and

configuring the "Secured Phone URL Parameters" with http instead of https. Setting the Rollback parameter creates a signed ITL file with blank function entries. The "empty" ITL file is still signed, so the cluster must be in a fully working security state before this parameter can be turned on.

 

After this parameter is enabled and the new ITL file with blank entries downloaded and verified, the phones will accept any configuration file, no matter who has signed it.

 

It is not recommended to leave the cluster in this state, because none of the 3 functions above (authenticated config files, encrypted config files, and https URLs) will be available.

 

Can I easily delete the ITL file from all phones once the CallManager.pem is lost?

There is currently no method to delete all ITLs from a phone remotely provided by Cisco. That's why the procedures and interactions above are so important to take into account.

 

There is a currently unresolved enhancement defect CSCto47052 requesting this functionality, but it has not yet been implemented.

 

In the interim period a new feature has been added via defect CSCts01319 that may allow Cisco TAC to revert to the previously trusted ITL if it is still available on the server. This will only work in certain instances where the cluster is on a version with this defect fix, and where the previous ITL exists in a backup stored in a special location on the server. View the defect to see if your version has the fix. Contact Cisco TAC to run through the potential recovery procedure explained in the defect.

 

If the previous procedure is not available, the phone buttons must be pushed manually on the phone to delete the ITL file. This is the trade off that is made between security and ease of administration. In order for the ITL file to be truly secure it must not be easily removed remotely.

 

Even with scripted button presses using SOAP XML objects, the ITL could not be remotely removed, because at that point TVS access (and thus Secure Authentication URL access to validate incoming SOAP XML button push objects) will be non functional. If the authentication URL was not configured as secure, it may be possible to script the key presses to delete an ITL, but this script is not available from Cisco.

 

Other methods may be available from a third party to script remote key presses without using the Authentication URL, but these applications are not provided by Cisco.

 

The most frequently used method to delete the ITL is an email broadcast to all phone users instructing them of the key sequence. If settings access is set to "Restricted" or "Disabled" then the phone will need to be factory reset, as users will not have access to the "Settings" menu of the phone.

Comments

Great work Jason.  A++

New Member

Thanks for the detailed document, however I have to disagree with the point at the end relating to the remote deletion of ITL files, this is possible, even when the Secure Authentication URL does not get validated by the TVS. We recently used PhoneView from unifiedfx.com, have a look at the video on how simple this is: http://www.youtube.com/watch?v=f7oPuj1szNg

Jamie

Silver

Thanks for the comment. I've updated the final "ITL Deletion" section accordingly.

Cisco Employee

Fantastic document.

New Member

Hi,

We are using this version of CUCM and are seeing that it does not have the ITL or TVS.pem certificate.

Small question, do you know if this procedure can be implemented with CUCM 6.1.5?

Thanks & Regards

Silver

Luis,

This feature was introduced in 8.0. It did not exist before 8.0, so you will not be able to use it in 6.x or 7.x.

-Jason

New Member

Jason,

I see and thanks for letting me know. Right we have found out an certificate has expired already and need to change since its affecting the phones clock, ring tones, directory and some with expansion module. Do you know of a process of how to regenarate the certificate in the CUCM 6.x to bring the phones back to normal? Also, do you know if the app load id jar60sccp.9-2-3TH1 is compatible with CUCM 6.x?

Silver

Luis,

It sounds like you might be using CTL security if you are talking about certificates. I wrote another document for that:

https://supportforums.cisco.com/docs/DOC-18834

The rest of your questions might be better answered by posting onto the IP Telephony forums.

Luis,

If you need to remotely remove the CTL (or ITL files) you can use PhoneView from Unified FX (http://www.unifiedfx.com)

Note: You can request a trial license for PhoneView here: http://www.unifiedfx.com/phoneview/trial

The following video shows you how to manage ITL Files, the processs is the same for CTL Files:

http://www.youtube.com/watch?v=pz6FKIjsQv4

Thanks

Stephen

New Member

Jfawcett,

I'm not able to create an account in UnifiedFX to download this software. Can you please send me the demo version to my email ID riyas328@yahoo.co.in.

Thanks in Advance

Riyas

Cisco Employee

I found out that the sentence “In a multi-node CM cluster there can be up to three TVS entries for a phone, one for each CM in the CM Group of the phone.” is incorrect. I have an ITL file with 10 entries here for example.

Silver

Cyprien,

Each node will have an entry in the ITL file, yes that's true. The phone will only use those entires that exist in the phone configuration file though. Remember that the phone config file keeps a list of TVS servers that is an exact copy of the CallManager Group. This means a max of three entries.

There is a special case though where the phone will use the 3 entries from the CallManager group first. As a last resort the phone attempts to use the TFTP server for TVS. If the TFTP server was not in the CM Group then this gives a maximum of 4 servers that any given phone could try for TVS.

-Jason

Cisco Employee

Thanks for the clarification Jason!

Cisco Employee

great work!!

New Member

Thanks very much, very useful guide!

Only a little note: when you say, about centralized TFTP,

"In a heterogeneous environment with a mix of pre-8.x and 8.x clusters,  the "Prepare Cluster for Rollback to Pre 8.0" must be enabled on the 8.x  cluster as per defect CSCto87262"

I think that that enterprise parameter should be enabled only on centralized TFTP cluster, and not on leaf 8.x cluster. Because

- on leaf 8.x cluster, there will be ITLSEPxxx file, so phone will download that file

- on leaf 7.x cluster, the phone will download the ITLfile.tlv from centralized cluster. So that one needs to be empty, not the leaf cluster one.

Is my understanding correct?

Andrea

New Member

You did a great job!

There is one thing left i'm wondering about:

What are the limits to SbD when it comes down to authentication?

Does SbD provides full authentication for a voice communication?

Thanks in advance for your help,

Pier

Silver

SBD only provides the three functions mentioned above:

Security By Default provides these three functions for supported IP Phones:

  1. Default authentication of TFTP downloaded files (configuration, locale, ringlist, etc) using a signing key.
  2. Optional encryption of TFTP configuration files using a signing key.
  3. Certificate  verification for phone initiated HTTPS connections using a remote  certificate trust store on Communications Manager (Trust Verification  Service).

For voice payload authentication and encryption you need to use the CTL and USB eToken based security that I describe in my other document.

https://supportforums.cisco.com/docs/DOC-18834

Hi Jason

You said "TVS runs on all servers where the CallManager service is activated". I've TVS running on all the servers in my cluster despite the Publisher and 2 Subscribers TFTP dedicated have no Call Manager service running.

For information, I'm running CUCM 8.6.2-23900.

New Member

Jason,

 

Great doc.  Is there any new doc for CUCM 10.5?

 

thanks

Cisco Employee

Here is a doc with some information about CUCM 10.0(1) and ITLs if you haven't already found it, http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/117598-technote-itl-00.html.

Silver

Thanks for the keen observation. You're right that the TVS service does get activated on every installed node. However, this service will only be accessed on the nodes that are in the CUCM CallManager Group of the phone (and in some cases TFTP).

 

The phone logic on searching for a TVS server works like this:

1. First CUCM Server

2. Second CUCM Server (if present)

3. Third CUCM Server (if present)

4. TFTP Server - last resort

 

I'll reword the document accordingly.

New Member

Late to the party, but glad I came across it. A lot of hard work put into this doc, awesome explanation...

Cisco Employee

Very well explained !!!

Good Job

Cleared so many concepts.

 

Thank you Jason

jip Cisco Employee
Cisco Employee

Jason: I am working on a project migrating 2 CUCM Clusters of 8.6(2a) on MCS to UCS and upgrading them to 9.1.2 at the same time.

Our plan is :

Step 1: In the existing 8.6.2a cluster, Disabled the Security By Default by using the "Prepare Cluster for Rollback to pre 8.0" Enterprise Parameter.

Step2: Reboot the whole cluster and make sure all phones are registered.

Step3: Run Backup of the database

Step4: Create a new cluster of 8.6.2a with the backup database in UCS

Step5: Upgrade to 9.1.2 and do Backup

Step6: Install brand new 9.1.2SU4 with the Backup Database in Step5
Step7 : Change IP Addresses
Step8 : Then migrate all phones to the new cluster in UCS (All phones should still contain the Empty ITL)

Questions: will this plan work and if yes, is there any catches that I should be aware of?

Thanks

Jip,

I have done this using the parameter mentioned and worked like a charm, just as you describe it.

You can also make use of Prime Collaboration Deployment to migrate your CUCM servers, it is a pretty good tool.

Cheers

jip Cisco Employee
Cisco Employee

Great! Did you leave the Enterprise Parameter "Prepare Cluster for Rollback to pre 8.0" Enterprise Parameter" to True to disable Security by Default after migration?

Or Set the parameter back to False and reboot all phones to get the new ITL from the new cluster?

Cheers

There is no real need to set it back to False, but I would =) . Also, some applications such as Extension Mobility may not work unless you set it back to False.

jip Cisco Employee
Cisco Employee

I will check on the EM. So if I want to re-enable SBD in the new cluster, then I would set it back first in the new cluster before migrating the phones over from old cluster 8.6.2a to new cluster 9.1.2.

Then all phones will be registered with new ITL after migration. I do not need to setup another maintenance window to re-enable SBD and reboot the whole new cluster again.

Will it cause any issue?

Make Sense?

No, make sure all phones are registered to the new cluster before changing the parameter back to False. That is, you just need to follow the exact steps you mentioned in your very first comment =)

And yes, you will need to reboot all phones so they can get their ITL files from the new cluster afterwards. 

If you are still unsure if that will do, or prevent the phones registering to the new cluster, you can always go for the bulk certificate export approach: https://supportforums.cisco.com/document/60716/migrating-ip-phones-between-clusters-cucm-8-and-itl-files

And that, imho, is safer ;)

New Member

Thanks for Doc!

1) ITL - is it "Identity Trust List" or "Initial Trust List" ?

I see both terms in:
Cisco Unified Communications Manager Security Guide, Release 8.5(1)
http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/security/8_5_1/secugd/sec-851-cm/secusbd.html#wp1091990

And in:
Cisco Unified Communications Manager Security Guide, Release 10.0(1)
http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/security/10_0_1/secugd/CUCM_BK_C68276B4_00_cucm-security-guide-100/CUCM_BK_C68276B4_00_cucm-security-guide-100_chapter_011.html

And in:
Security Guide for Cisco Unified Communications Manager, Release 11.0(1)
http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/security/11_0_1/secugd/CUCM_BK_C1A78C1D_00_cucm-security-guide-1101/CUCM_BK_C1A78C1D_00_cucm-security-guide-1101_chapter_011.html#CUCM_RF_D957C6DB_00


But only "Initial Trust List" in:
Cisco Unified Communications Manager Security Guide, Release 9.1(1)
http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/security/9_1_1/secugd/CUCM_BK_C0395F44_00_cucm-security-guide-91/CUCM_BK_C0395F44_00_cucm-security-guide-91_chapter_011.html#CUCM_RF_I189964E_00

And in:
Cisco Unified Communications Manager Security Guide, Release 9.0(1)
http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/security/9_0_1/secugd/CUCM_BK_CCB00C40_00_cucm-security-guide-90/CUCM_BK_CCB00C40_00_cucm-security-guide_chapter_011.html


2) "The mirror" of your Document has very low quality pics
http://www.cisco.com/c/en/us/support/docs/voice-unified-communications/unified-communications-manager-callmanager/116232-technote-sbd-00.html

Cisco Employee

Hi Jason,

Brilliant work!!!

Just want to clarify few things:

Suppose all of my certs expire on the same day, what should be the approach to regenerate the certs?

I have gone through guides, it says that we need to regenerate the TFTP cert(s) in the last. Now, let say if we regenerate all the certs including TVS( excluding TFTP ). Once the certs are generated, will that update the ITL file automatically, i believe NO. 

Even we reset the phones at this moment, will the new ITL contain the new regenerated certs?

If yes, then how the phone will trust this new ITL because the TFTP server signature is still old, is this correct?

After it saves the new ITL files, it has all new certs except TFTP.

And eventually when we regenerate the TFTP ( after all the certs are regenerated ), it will create a new private-public key pair. Now, do we reset phone at this moment to update the new ITL again?

If we reset the phone, the new ITL has different signature because the TFTP key is changed. So, how the phone will trust the new ITL?

Will it contact the TVS in this case and will TVS authenticate this new ITL file?

Thanks in Advance!!

~Amit

Cisco Employee

Suppose all of my certs expire on the same day, what should be the approach to regenerate the certs?

The key to regenerating certs and not screwing up your ITL is to do one of your CallManager.pem or TVS.pem certs at a time and make sure that the phones get the updated ITL after each cert is changed. 

Which cert you do first shouldn't matter as long as you are starting with the phones having a current ITL, and as long as the phone has a current TVS cert in its ITL it'll update any ITL signed by a cert that TVS can validate.

Note that uploading or regenerating either cert will reset all your phones automatically in an attempt to let them to get an updated ITL. You don't have a choice in this so don't go doing cert operations in the middle of the day or you risk angering your users.

Say you don't know the state of the ITLs on all of your phones. How can you test this? A quick way to do it is to change your CM group around to move all your phones from a primary sub to a backup and reset all your phones (leave the primary in the CM group though). Since the list of servers to register to is contained in the config file, which is always signed and validated by the ITL, any phone that registers back to the former primary didn't get an updated config file and should be investigated. 

Cisco Employee

Hi Phillip,

I tried to understand your notes, got most part of it. But, here are the confusions again.

For eg. If i regenerate TVS cert first, the ITL will change. Phones will reset by itself. Now, when the phone get the new updated ITL, will that accept it because it is signed by the same TFTP, it is correct ? or it will be a different behavior.

And let's say if the above is correct, and when i regenerate the TFTP certs, it will change the ITL again, phones will reset again and now the TFTP signature is different, so will it contact the TVS for the ITL authentication?

~Amit

Cisco Employee

You've got it Amit.  That is exactly how the process is supposed to work. Things mostly go bad when the phone doesn't trust the updated ITL so it can't pick up the new TVS certificate. 

When you have multiple nodes and your TFTP server isn't running CCM this gets more reliable because you tend to update certificates on one node at a time, so the TVS servers don't change at the same time the TFTP server's CallManager.pem cert does.

Cisco Employee

Excellent !!

Thanks Phillip, these doubts have been pending for long and i could not found any document that explains it. I appreciate all your help.

~Amit

New Member

Very good doc. I have one question, a Cisco TAC SE told us we have also to use the CTL Client to regenerate the CTL File after we regenerate the callmanager cert and TVS cert. Is that correct? We regenerate not the CAPF Cert.

We run CUCM 9.1

Thanks in Advance!!

Jörg

Cisco Employee

You need to manually regenerate the CTL file any time a certificate contained in the CTL changes. 

The CTL does not contain the TVS cert so in your case that would not require a CTL update. The CallManager.pem certificate would require a CTL update.


Make sure you reset all of your phones between uploading the new CallManager and TVS certificates. If you do both at the same time you risk breaking the trust chain for your ITL files.

77671
Views
75
Helpful
37
Comments