cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
11023
Views
5
Helpful
0
Comments
Shawn Ford
Community Member

 

Troubleshooting Overview

This section provides the necessary background information and available resources to troubleshoot the Cisco Unified CallManager.

The section covers following topics:

 •  Cisco Unified CallManager Serviceability 

 

•  Cisco Unified Communications Operating System Administration

 

• General Model of Problem Solving

 

• Network Failure Preparation

 

Cisco Unified CallManager Serviceability

Cisco Unified CallManager Serviceability, a web-based troubleshooting tool for Cisco Unified CallManager, provides the following functionality to assist administrators troubleshoot system problems:

•Saves Cisco Unified CallManager services alarms and events for troubleshooting and provides alarm message definitions.

 

•Saves Cisco Unified CallManager services trace information to various log files for troubleshooting. Administrators can configure, collect, and view trace information.

 

•Monitors real-time behavior of the components in a Cisco Unified CallManager cluster through the real-time monitoring tool (RTMT).

 

•Generates reports for Quality of Service, traffic, and billing information through Cisco Unified CallManager CDR Analysis and Reporting (CAR).

 

•Provides feature services that you can activate, deactivate, and view through the Service Activation window.

 

•Provides an interface for starting and stopping feature and network services.

 

•Archives reports that are associated with Cisco Unified CallManager Serviceability tools.

 

•Allows Cisco Unified CallManager to work as a managed device for SNMP remote management and troubleshooting.

 

•Monitors the disk usage of the log partition on a server (or all servers in the cluster).

Access Cisco Unified CallManager Serviceability from the Cisco Unified CallManager Administration window by choosing Cisco Unified CallManager Serviceability from the Navigation drop-down list box. Installing the Cisco Unified CallManager software automatically installs Cisco Unified CallManager Serviceability and makes it available.

 

Cisco Unified Communications Operating System Administration

Cisco Unified Communications Operating System Administration allows you to perform the following tasks to configure and manage the Cisco Unified Communications Operating System:

•Check software and hardware status.

 

•Check and update IP addresses.

 

•Ping other network devices.

 

•Manage Network Time Protocol servers.

 

•Upgrade system software and options.

 

•Restart the system.

 

General Model of Problem Solving

When troubleshooting a telephony or IP network environment, define the specific symptoms, identify all potential problems that could be causing the symptoms, and then systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear.

The following steps provide guidelines to use in the problem-solving process.

Procedure


Step 1 Analyze the network problem and create a clear problem statement. Define symptoms and potential causes.

 

Step 2 Gather the facts that you need to help isolate possible causes.

 

Step 3 Consider possible causes based on the facts that you gathered.

 

Step 4 Create an action plan based on those causes. Begin with the most likely problem and devise a plan in which you manipulate only one variable.

 

Step 5 Implement the action plan; perform each step carefully while testing to see whether the symptom disappears.

 

Step 6 Analyze the results to determine whether the problem has been resolved. If the problem was resolved, consider the process complete.

 

Step 7 If the problem has not been resolved, create an action plan based on the next most probable cause on your list. Return to Step 4 and repeat the process until the problem is solved.

NOTE: Make sure that you undo anything that you changed while implementing your action plan. Remember that you want to change only one variable at a time. NOTE: If you exhaust all the common causes and actions (either those outlined in this document or others that you have identified in your environment), contact Cisco TAC.

 

Network Failure Preparation

You can always recover more easily from a network failure if you are prepared ahead of time. To determine if you are prepared for a network failure, answer the following questions:

•Do you have an accurate physical and logical map of your internetwork that outlines the physical location of all of the devices on the network and how they are connected as well as a logical map of network addresses, network numbers, and subnetworks?

 

•Do you have a list of all network protocols that are implemented in your network for each of the protocols implemented and a list of the network numbers, subnetworks, zones, and areas that are associated with them?

 

•Do you know which protocols are being routed and the correct, up-to-date configuration information for each protocol?

 

•Do you know which protocols are being bridged? Are any filters configured in any of these bridges, and do you have a copy of these configurations? Is this applicable to Cisco Unified CallManager?

 

•Do you know all the points of contact to external networks, including any connections to the Internet? For each external network connection, do you know what routing protocol is being used?

 

•Has your organization documented normal network behavior and performance, so you can compare current problems with a baseline?

If you can answer yes to these questions, faster recovery from a failure results.

 

Troubleshooting Tools

This section addresses the tools and utilities that you use to configure, monitor, and troubleshoot Cisco Unified CallManager and provides general guidelines for collecting information to avoid repetitive testing and recollection of identical data.

NOTE: To access some of the URL sites that are listed in this document, you must be a registered user, and you must be logged in.

This section contains the following topics:

•Cisco Unified CallManager Serviceability Troubleshooting Tools

 

•  Command Line Interface

 

• CiscoWorks2000

 

•  Sniffer Traces

 

•  Debugs

 

•  Cisco Secure Telnet

 

•  Packet Capture

 

•  Troubleshooting Perfmon Data Logging

 

•  Common Troubleshooting Tasks, Tools, and Commands

 

•  Troubleshooting Tips

 

•  Verify Cisco Unified CallManager Services Are Running

 

 

Cisco Unified CallManager Serviceability Troubleshooting Tools

Refer to the Cisco Unified CallManager Serviceability Administration Guide and the Cisco Unified CallManager Serviceability System Guide for detailed information of the following different types of tools that Cisco Unified CallManager Serviceability provides to monitor and analyze the various Cisco Unified CallManager systems.

 

 

 

Command Line Interface

Use the command line interface (CLI) to access the Cisco Unified CallManager system for basic maintenance and failure recovery. Obtain access to the system by either a hard-wired terminal (a system monitor and keyboard) or by performing a SSH session.

 

The account name and password get created at install time. You can change the password after install, but you never can change the account name.

 

A command represents a text instruction that caused the system to perform some function. Commands may be stand alone, or they can have mandatory or optional arguments or options.

 

A level comprises a collection of commands; for example, show designates a level, whereas show status specifies a command. Each level and command also includes an associated privilege level. You can execute a command only if you have sufficient privilege level.

 

For complete information on the Cisco Unified CallManager CLI command set, see the Cisco Unified CallManager Administration Guide, Release 5.1.

 

CiscoWorks2000

CiscoWorks2000 serves as the network management system of choice for all Cisco devices including Cisco Unified CallManager. Because CiscoWorks2000 is not bundled with Cisco Unified CallManager, you must purchase it separately. Use the following tools with CiscoWorks2000 for remote serviceability:

 

• System Log Management

• Cisco Discovery Protocol Support

• Simple Network Management Protocol Support

 

System Log Management

Although it can be adapted to other network management systems, Cisco Syslog Analysis, which is packaged with CiscoWorks2000 Resource Manager Essentials, provides the best method to manage Syslog messages from Cisco devices.

 

Cisco Syslog Analyzer serves as the component of Cisco Syslog Analysis that provides common storage and analysis of the system log for multiple applications. The other major component, Syslog Analyzer Collector, gathers log messages from Cisco Unified CallManager servers.

 

These two Cisco applications work together to provide a centralized system logging service for Cisco Unified Communications Solutions.

 

Cisco Discovery Protocol Support

The Cisco Discovery Protocol Support enables discovery of Cisco Unified CallManager servers and management of those servers by CiscoWorks2000.

 

Simple Network Management Protocol Support

Network management systems (NMS) use SNMP, an industry-standard interface, to exchange management information between network devices. A part of the TCP/IP protocol suite, SNMP enables administrators to remotely manage network performance, find and solve network problems, and plan for network growth.

An SNMP-managed network comprises three key components: managed devices, agents, and network management systems.

•A managed device designates a network node that contains an SNMP agent and resides on a managed network. Managed devices collect and store management information and make it available by using SNMP.

 

•An agent, as network management software, resides on a managed device. An agent contains local knowledge of management information and translates it into a form that is compatible with SNMP.

 

•A network management system comprises an SNMP management application together with the computer on which it runs. An NMS executes applications that monitor and control managed devices. An NMS provides the bulk of the processing and memory resources that are required for network management. The following NMSs share compatibility with Cisco Unified CallManager:

 

– CiscoWorks2000

 

– HP OpenView

 

– Third-party applications that support SNMP and Cisco Unified CallManager SNMP interfaces

 

Sniffer Traces

Typically, you collect sniffer traces by connecting a laptop or other sniffer-equipped device on a Catalyst port that is configured to span the VLAN or port(s) (CatOS, Cat6K-IOS, XL-IOS) that contains the trouble information. If no free port is available, connect the sniffer-equipped device on a hub that is inserted between the switch and the device.

TIP: To help facilitate reading and interpreting of the traces by the TAC engineer, Cisco recommends using Sniffer Pro software because it is widely used within the TAC.

Have available the IP/MAC addresses of all equipment that is involved, such as IP phones, gateways, Cisco Unified CallManagers, and so on.

 

Debugs

The output from debug privileged EXEC commands provides diagnostic information about a variety of internetworking event that relate to protocol status and network activity in general.

 

Set up your terminal emulator software (such as HyperTerminal), so it can capture the debug output to a file. In HyperTerminal, click Transfer; then, click Capture Text and choose the appropriate options.

 

Before running any IOS voice gateway debugs, make sure that service timestamps debug datetime msec is globally configured on the gateway.

NOTE: Avoid collecting debugs in a live environment during operation hours.

Preferably, collect debugs during non-working hours. If you must collect debugs in a live environment, configure no logging console and logging buffered. To collect the debugs, use show log.

 

Because some debugs can be lengthy, collect them directly on the console port (default logging console) or on the buffer (logging buffer). Collecting debugs over a Telnet session may impact the device performance, and the result could be incomplete debugs, which requires that you re-collect them.

 

To stop a debug, use the no debug all or undebug all commands. Verify that the debugs have been turned off by using the command show debug.

 

Cisco Secure Telnet

Cisco Secure Telnet allows Cisco Service Engineers (CSE) transparent firewall access to the Cisco Unified CallManager node on your site. Using strong encryption, Cisco Secure Telnet enables a special Telnet client from Cisco Systems to connect to a Telnet daemon behind your firewall. This secure connection allows remote monitoring and troubleshooting of your Cisco Unified CallManager nodes, without requiring firewall modifications.

NOTE: Cisco provides this service only with your permission. You must ensure that a network administrator is available at your site to help initiate the process.

 

Packet Capture

This section contains information on the following topics:

 

• Packet Capturing Overview

• Configuration Checklist for Packet Capturing

• Adding an End User to the Standard Packet Sniffer Users Group

• Configuring Packet-Capturing Service Parameters

• Configuring Packet Capturing in the Phone Configuration Window

• Configuring Packet Capturing in Gateway and Trunk Configuration Windows

• Packet-Capturing Configuration Settings

• Analyzing Captured Packets

 

 

Packet Capturing Overview

Because third-party troubleshooting tools that sniff media and TCP packets do not work after you enable encryption, you must use Cisco Unified CallManager Administration to perform the following tasks if a problem occurs:

 

•Analyze packets for messages that are exchanged between Cisco Unified CallManager and the device (Cisco Unified IP Phone, Cisco Unified SIP IP Phone, Cisco IOS MGCP gateway, H.323 gateway, H.323/H.245/H.225 trunk, or SIP trunk).

 

•Capture the Secure Real Time Protocol (SRTP) packets between the devices.

 

•Extract the media encryption key material from messages and decrypt the media between the devices.

TIP: Performing this task for several devices at the same time may cause high CPU usage and call-processing interruptions. Cisco strongly recommends that you perform this task when you can minimize call-processing interruptions.

For more information, see the Cisco Unified CallManager Security Guide.

 

Configuration Checklist for Packet Capturing

Extracting and analyzing pertinent data includes performing the following tasks in Table 2-2:

Table 2-2 Configuration Checklist for Packet Capturing

 

Packet-Capturing Configuration Settings

Use Table 2-3, which describes the Packet Capture Mode and Packet Capture Duration settings, with the following sections:

• Configuring Packet Capturing in the Phone Configuration Window

 

• Configuring Packet Capturing in Gateway and Trunk Configuration Windows

Table 2-3 Packet-Capturing Configuration Settings

 

Adding an End User to the Standard Packet Sniffer Users Group

End users that belong to the Standard Packet Sniffer Users group can configure the Packet Capture Mode and Packet Capture Duration settings for devices that support packet capturing. If the user does not exist in the Standard Packet Sniffer Users group, the user cannot initiate packet capturing.

 

The following procedure, which describes how to add an end user to the Standard Packet Sniffer Users group, assumes that you configured the end user in Cisco Unified CallManager Administration, as described in the Cisco Unified CallManager Administration Guide.

 

Procedure


Step 1 Find the user group, as described in the Cisco Unified CallManager Administration Guide.

 

Step 2 After the Find/List window displays, click the Standard Packet Sniffer Users link.

 

Step 3 Click the Add Users to Group button.

 

Step 4 Add the end user, as described in the Cisco Unified CallManager Administration Guide.

 

Step 5 After you add the user, click Save.

 


 

Configuring Packet-Capturing Service Parameters

To configure parameters for packet capturing, perform the following procedure:


Procedure

Step 1 In Cisco Unified CallManager Administration, choose System > Service Parameters.

 

Step 2 From the Server drop-down list box, choose an Active server where you activated the Cisco Unified CallManager service.

 

Step 3 From the Service drop-down list box, choose the Cisco CallManager (Active) service.

 

Step 4 Scroll to the TLS Packet Capturing Configuration pane and configure the packet capturing settings.

TIP: For information on the service parameters, click the name of the parameter or the question mark that displays in the window. NOTE: For packet capturing to occur, you must set the Packet Capture Enable service parameter to True.

Step 5 For the changes to take effect, click Save.

 

Step 6 To continue packet-capturing configuration, see one of the following sections:

 

• Configuring Packet Capturing in the Phone Configuration Window

 

• Configuring Packet Capturing in Gateway and Trunk Configuration Windows

 


 

Configuring Packet Capturing in the Phone Configuration Window

After you enable packet capturing in the Service Parameter window, you can configure packet capturing on a per-device basis in the Phone Configuration window of Cisco Unified CallManager Administration.

You enable or disable packet capturing on a per-phone basis. The default setting for packet capturing equals None.

TIP: Cisco strongly recommends that you do not enable packet capturing for many phones at the same time because this task may cause high CPU usage in your network.

If you do not want to capture packets or if you completed the task, set the Packet Capture Enable service parameter to False.

To configure packet capturing for phones, perform the following procedure:


Procedure

Step 1 Before you configure the packet-capturing settings, see the  Configuration Checklist for Packet Capturing  section.

 

Step 2 Find the SIP or SCCP phone, as described in the Cisco Unified CallManager Administration Guide.

 

Step 3 After the Phone Configuration window displays, configure the troubleshooting settings, as described in Table 2-3.

 

Step 4 After you complete the configuration, click Save.

 

Step 5 In the Reset dialog box, click OK.

TIP: Although Cisco Unified CallManager Administration prompts you to reset the device, you do not need to reset the device to capture packets.

Additional Steps

- Capture SRTP packets by using a sniffer trace between the affected devices.

 

- After you capture the packets, set the Packet Capture Enable service parameter to False.

 

- See the  Analyzing Captured Packets section.

 


 

Configuring Packet Capturing in Gateway and Trunk Configuration Windows

The following gateways and trunks support packet capturing in Cisco Unified CallManager Administration:

•Cisco IOS MGCP gateways

 

•H.323 gateways

 

•H.323/H.245/H.225 trunks

 

•SIP trunks

TIP: Cisco strongly recommends that you do not enable packet capturing for many devices at the same time because this task may cause high CPU usage in your network.

If you do not want to capture packets or if you completed the task, set the Packet Capture Enable service parameter to False.

To configure packet-capturing settings in the Gateway or Trunk Configuration window, perform the following procedure:

 


Procedure

Step 1 Before you configure the packet-capturing settings, see the "Configuration Checklist for Packet Capturing" section.

 

Step 2 Perform one of the following tasks:

 

•Find the Cisco IOS MGCP gateway, as described in the Cisco Unified CallManager Administration Guide.

 

•Find the H.323 gateway, as described in the Cisco Unified CallManager Administration Guide.

 

•Find the H.323/H.245/H.225 trunk, as described in the Cisco Unified CallManager Administration Guide.

 

•Find the SIP trunk, as described in the Cisco Unified CallManager Administration Guide.

 

Step 3 After the configuration window displays, locate the Packet Capture Mode and Packet Capture Duration settings.

TIP: If you located a Cisco IOS MGCP gateway, ensure that you configured the ports for the Cisco IOS MGCP gateway, as described in the Cisco Unified CallManager Administration Guide. The packet-capturing settings for the Cisco IOS MGCP gateway display in the Gateway Configuration window for endpoint identifiers. To access this window, click the endpoint identifier for the voice interface card.

Step 4 Configure the troubleshooting settings, as described in Table 2-3.

 

Step 5 After you configure the packet-capturing settings, click Save.

 

Step 6 In the Reset dialog box, click OK.

TIP: Although Cisco Unified CallManager Administration prompts you to reset the device, you do not need to reset the device to capture packets.

Additional Steps

- Capture SRTP packets by using a sniffer trace between the affected devices.

 

- After you capture the packets, set the Packet Capture Enable service parameter to False.

 

- See the  Analyzing Captured Packets section.

 


 

Packet-Capturing Configuration Settings

Use Table 2-3, which describes the Packet Capture Mode and Packet Capture Duration settings, with the following sections:

• Configuring Packet Capturing in the Phone Configuration Window

• Configuring Packet Capturing in Gateway and Trunk Configuration Windows


Table 2-3 Packet-Capturing Configuration Settings

 

Analyzing Captured Packets

Cisco Technical Assistance Center (TAC) analyzes the packets by using a debugging tool. Before you contact TAC, capture SRTP packets by using a sniffer trace between the affected devices. Contact TAC directly after you gather the following information:

•Packet Capture File— Insert non-formatted text here https://<IP address or server name>/pktCap/pktCap.jsp?file=mm-dd-yyyy.pkt, where you browse into the server and locate the packet-capture file by month, date, and year (mm-dd-yyyy)

•Key for the file— Insert non-formatted text here https://<IP address or server name>/pktCap/pktCap.jsp?key=mm-dd-yyyy.pkt, where you browse into the server and locate the key by month, date, and year (mm-dd-yyyy)

•User name and password of end user that belongs to the Standard Packet Sniffer Users group

For more information, see the Cisco Unified CallManager Security Guide.

 

Troubleshooting Perfmon Data Logging

CAUTION: ENABLING THE TROUBLEHSOOTING PERFMON DATA LOGGING FEATURE IMPACTS SYSTEMS PERFORMANCE ON THE SELECTED NODE.
DO NOT ENABLE THIS PARAMETER UNLESS CISCO TECHNICAL CENTER DIRECTS YOU TO DO SO.
.

The troubleshooting perfmon data logging feature assists Cisco TAC in identifying system problems. When you enable troubleshooting perfmon data logging, you initiate the collection of a set of Cisco Unified CallManager and operating system performance statistics on the selected node. The statistics that are collected include comprehensive information that can be used for system diagnosis and information from a set of counters that is not a part of the current set of preconfigured counters in the Real-Time Monitoring Tool.

Because an extensive amount of information is collected in a short time, Cisco recommends that you do not enable the troubleshooting perfmon data logging for any extended time and that you enable the Log Partitioning Monitor to monitor disk usage while troubleshooting perfmon data logging is enabled.

When you enable the troubleshooting perfmon data logging feature on a system with no active phone calls and you use the default setting for the troubleshooting perfmon data-logging parameters, Cisco estimates that the system experiences a less than 5-percent increase in CPU utilization and an insignificant increase in the amount of memory that is being used, and it writes approximately 50 MB of information to the log files daily.

You can perform the following administrative tasks with the troubleshooting perfmon data-logging feature:

•Enable and disable the trace filter for Troubleshooting perfmon data logging.

 

•Monitor a set of predefined System and Cisco Unified CallManager performance objects and counters on each server.

 

•Log the monitored performance data in CSV file format on the local server in the active log partition in the var/log/active/cm/log/ris/csv directory. The log file uses the following naming convention: PerfMon_<node>_<month>_<day>_<year>_<hour>_<minute>.csv; for example, PerfMon_172.19.240.80_06_15_2005_11_25.csv.

 

•Specify the polling rate. This rate specifies the rate at which performance data gets gathered and logged. You can configure the polling rate down to 5 seconds. Default polling rate equals 15 seconds.

 

•Specify the maximum number of log files that will be stored on disk. Log files exceeding this limit get purged automatically by removal of the oldest log file. The default specifies 50 files.

 

•Specify the rollover criteria of the log file based on the maximum size of the file in megabytes. The default value specifies 2 MB.

 

•Collect the SOAP log file by using the Trace & Log Central feature of the Real-Time Monitoring Tool or Command Line Interface.

 

•Collect the Cisco RIS Data Collector PerfMonLog log file by using the Trace & Log Central feature of the Real-Time Monitoring Tool or Command Line Interface.

 

•View the log file in graphical format by using the Microsoft Windows performance tool as described in "Viewing the Perfmon Log Files with the Microsoft Performance Tool" section or by using the Real-Time Monitoring Tool as described in the Cisco Unified CallManager Serviceability Administration Guide.

The troubleshooting perfmon data-logging feature collects information from the following counters within the following perfmon objects. Refer to the "Performance Objects and Counters" chapter in Cisco Unified CallManager Serviceability System Guide for a description on the counters:

•Cisco CallManager Object:

 

–CallManagerHeartBeat

 

–CallsActive

 

–CallsAttempted

 

–CallsCompleted

 

–InitializationState

 

–RegisteredHardwarePhones

 

–RegisteredMGCPGateway

 

•Cisco CallManager System Performance Object:

 

–AverageExpectedDelay

 

–CallsRejectedDueToThrottling

 

–QueueSignalsPresent 1-High

 

–QueueSignalsPresent 2-Normal

 

–QueueSignalsPresent 3-Low

 

–QueueSignalsPresent 4-Lowest

 

–QueueSignalsProcessed 1-High

 

–QueueSignalsProcessed 2-Normal

 

–QueueSignalsProcessed 3-Low

 

–QueueSignalsProcessed 4-Lowest

 

–QueueSignalsProcessed Total

 

–TotalCodeYellowEntry

 

•Cisco TFTP

 

–BuildAbortCount

 

–BuildCount

 

–BuildDeviceCount

 

–BuildDialruleCount

 

–BuildDuration

 

–BuildSignCount

 

–BuildSoftkeyCount

 

–BuildUnitCount

 

–ChangeNotifications

 

–DeviceChangeNotifications

 

–DialruleChangeNotifications

 

–EncryptCount

 

–GKFoundCount

 

–GKNotFoundCount

 

–HeartBeat

 

–HttpConnectRequests

 

–HttpRequests

 

–HttpRequestsAborted

 

–HttpRequestsNotFound

 

–HttpRequestsOverflow

 

–HttpRequestsProcessed

 

–HttpServedFromDisk

 

–LDFoundCount

 

–LDNotFoundCount

 

–MaxServingCount

 

–Requests

 

–RequestsAborted

 

–RequestsInProgress

 

–RequestsNotFound

 

–RequestsOverflow

 

–RequestsProcessed

 

–SegmentsAcknowledged

 

–SegmentsFromDisk

 

–SegmentsSent

 

–SEPFoundCount

 

–SEPNotFoundCount

 

–SIPFoundCount

 

–SIPNotFoundCount

 

–SoftkeyChangeNotifications

 

–UnitChangeNotifications

 

•Process Object:

 

–PID

 

–STime

 

–% CPU Time

 

–Page Fault Count

 

–Process Status

 

–VmData

 

–VmRSS

 

–VmSize

 

–Thread Count

 

•Memory Object:

 

–Used Kbytes

 

–Free Kbytes

 

–Total Kbytes

 

–Shared Kbytes

 

–Buffers Kbytes

 

–Cached Kbytes

 

–Free Swap Kbytes

 

–Total Swap Kbytes

 

–Used Swap Kbytes

 

–Pages Input

 

–Pages Output

 

–Pages

 

–Used VM Kbytes

 

–Total VM Kbytes

 

–% Page Usage

 

–% VM Used

 

–% Mem Used

 

•Processor Object:

 

–Irq Percentage

 

–Softirq Percentage

 

–IOwait Percentage

 

–User Percentage

 

–Nice Percentage

 

–System Percentage

 

–Idle Percentage

 

–%CPU Time

 

•Thread Object—Troubleshooting Perfmon Data Logger only logs CCM threads:

 

–%CPU Time

 

•Partition Object:

 

–Used Mbytes

 

–Total Mbytes

 

–%Used

 

–Await Read Time

 

–Await Write Time

 

–Await Time

 

–% CPU Time

 

–Read Bytes Per Sec

 

–Write Bytes Per Sec

 

–Queue Length

 

•IP Object:

 

–In Receives

 

–InHdrErrors

 

–In UnknownProtos

 

–In Discards

 

–In Delivers

 

–Out Requests

 

–Out Discards

 

–Reasm Reqds

 

–Reasm Oks

 

–Reasm Fails

 

–Frag OKs

 

–Frag Fails

 

–Frag Creates

 

–InOut Requests

 

•TCP Object:

 

–Active Opens

 

–Passive Opens

 

–Attempt Fails

 

–Estab Resets

 

–Curr Estab

 

–In Segs

 

–Out Segs

 

–Retrans Segs

 

–InOut Segs

 

•Network Interface Object:

 

–Rx Bytes

 

–Rx Packets

 

–Rx Errors

 

–Rx Dropped

 

–Rx Multicast

 

–Tx Bytes

 

–Tx Packets

 

–Tx Errors

 

–Tx Dropped

 

–Total Bytes

 

–Total Packets

 

–Tx QueueLen

 

•System Object:

 

–Allocated FDs

 

–Freed FDs

 

–Being Used FDs

 

–Max FDs

 

–Total Processes

 

–Total Threads

 

–Total CPU Time

The following procedure provides the steps for using the troubleshooting perfmon data- logging feature.


Procedure

Step 1 Configure the Troubleshooting Perfmon Data Logging parameters in the Cisco RIS Data Collector service.

 

See the "Configuring Troubleshooting Perfmon Data Logging" section

 

Step 2 Verify that log partition monitoring is enabled.

 

See the Cisco Unified CallManager Administration Guide.

 

Step 3 Collect the log files for the Cisco RIS Data Collector service on the server that has troubleshooting perfmon data logging enabled

 

•If you want to download the log files by using RTMT, refer to Cisco Unified CallManager Serviceability Administration Guide.

 

•If you want to download the log files by using the CLI, refer to Cisco Unified Communications Operating System Administration Guide.

 

Step 4 View the log file in graphical format by using the Microsoft Windows performance tool as described in "Viewing the Perfmon Log Files with the Microsoft Performance Tool" section or by using the Real-Time Monitoring Tool as described in the Cisco Unified CallManager Serviceability Administration Guide.

 

Step 5 When you have collected all the necessary files, disable troubleshooting perfmon data logging by setting the Enable Logging parameter to False.

 


 

Configuring Troubleshooting Perfmon Data Logging

The following procedure describes how to configure the troubleshooting perfmon data-logging feature.


Procedure

Step 1 In Cisco Unified CallManager Administration, choose System > Service Parameters.

 

The Service Parameter Configuration window displays.

 

Step 2 From the Server drop-down list box, choose the server.

 

Step 3 From the Service drop-down list box, choose Cisco RIS Data Collector.

 

Step 4 Enter the appropriate settings as described in Table 2-4.

 

Step 5 Click Save.

 


 

Troubleshooting Perfmon Data-Logging Configuration Settings

Table 2-4 describes the available settings to enable and disable troubleshooting perfmon data logging.


Table 2-4 Troubleshooting Perfmon Data-Logging Parameters

 

 

Viewing the Perfmon Log Files with the Microsoft Performance Tool

To view the log files by using the Microsoft Performance tool, follow these steps:


Procedure

Step 1 Choose Start > Settings > Control Panel > Administrative Tools > Performance.

 

Step 2 In the application window, click the right mouse button and choose Properties.

 

Step 3 Click the Source tab in the System Monitor Properties dialog box.

 

Step 4 Browse to the directory where you downloaded the perfmon log file and choose the perfmon csv file. The log file includes the following naming convention: PerfMon_<node>_<month>_<day>_<year>_<hour>_<minute>.csv; for example, PerfMon_172.19.240.80_06_15_2005_11_25.csv.

 

Step 5 Click Apply.

 

Step 6 Click the Time Range button. To specify the time range in the perfmon log file that you want to view, drag the bar to the appropriate starting and ending times.

 

Step 7 To open the Add Counters dialog box, click the Data tab and click Add.

 

Step 8 From the Performance Object drop-down box, choose the perfmon object. If an object has multiple instances, you may choose All instances or select only the instances that you are interested in viewing.

 

Step 9 You can choose All Counters or select only the counters that you are interested in viewing.

 

Step 10 To add the selected counters, click Add

 

Step 11 When you finish selecting counters, click Close.

 


 

Common Troubleshooting Tasks, Tools, and Commands

This section provides a quick reference for commands and utilities to help you troubleshoot a Cisco Unified CallManager server with root access disabled. Table 2-5 provides a summary of the CLI commands and GUI selections that you can use to gather information troubleshoot various system problems.


Table 2-5 Summary of CLI Commands and GUI Selections

    • INSERT TABLE

Table 2-6 provides a list of common problems and tools to use to troubleshoot them.


Table 2-6 Troubleshooting Common Problems with CLI Commands and GUI Selections

    • INSERT TABLE

 

Troubleshooting Tips

The following tips may help you when you are troubleshooting the Cisco Unified CallManager.

TIP: Check the release notes for Cisco Unified CallManager for known problems. The release notes provide descriptions and workaround solutions for known problems.
TIP: Know where your devices are registered.

Each Cisco Unified CallManager log traces files locally. If a phone or gateway is registered to a particular Cisco Unified CallManager, the call processing gets done on that Cisco Unified CallManager if the call is initiated there. You will need to capture traces on that Cisco Unified CallManager to debug a problem.

A common mistake involves having devices that are registered on a subscriber server but are capturing traces on the publisher server. These trace files will be nearly empty (and definitely will not have the call in them).

Another common problem involves having Device 1 registered to CM1 and Device 2 registered to CM2. If Device 1 calls Device 2, the call trace occurs in CM1, and, if Device 2 calls Device 1, the trace occurs in CM2. If you are troubleshooting a two-way calling issue, you need both traces from both Cisco Unified CallManagers to obtain all the information that is needed to troubleshoot.

TIP: Know the approximate time of the problem.

Multiple calls may have occurred, so knowing the approximate time of the call helps TAC quickly locate the trouble.

You can obtain phone statistics on a Cisco Unified IP Phone 79xx by pressing the i or ? button twice during an active call.

When you are running a test to reproduce the issue and produce information, know the following data that is crucial to understanding the issue:

•Calling number/called number

 

•Any other number that is involved in the specific scenario

 

•Time of the call

NOTE: Remember that time synchronization of all equipment is important for troubleshooting.

If you are reproducing a problem, make sure to choose the file for the timeframe by looking at the modification date and the time stamps in the file. The best way to collect the right trace means that you reproduce a problem and then quickly locate the most recent file and copy it from the Cisco Unified CallManager server.

TIP: Save the log files to prevent them from being overwritten.

Files will get overwritten after some time. The only way to know which file is being logged to is to choose View >Refresh on the menu bar and look at the dates and times on the files.

 

Verify Cisco Unified CallManager Services Are Running

Use the following procedure to verify which Cisco CallManager services are active on a server.


Procedure

Step 1 From Cisco Unified CallManager Administration, choose Navigation > Cisco Unified CallManager Serviceability.

 

Step 2 Choose Tools > Service Activation.

 

Step 3 From the Servers column, choose the desired server.

 

The server that you choose displays next to the Current Server title, and a series of boxes with configured services displays.

 

Activation Status column displays either Activated or Deactivated in the Cisco CallManager line.

 

If the Activated status displays, the specified Cisco CallManager service remains active on the chosen server.

 

If the Deactivated status displays, continue with the following steps.

 

Step 4 Check the check box for the desired Cisco CallManager service.

 

Step 5 Click the Update button.

 

The Activation Status column displays Activated in the specified Cisco CallManager service line.

 

The specified service now shows active for the chosen server.

Perform the following procedure if the Cisco CallManager service has been in activated and you want to verify if the service is currently running.


Procedure

Step 1 From Cisco Unified CallManager Administration, choose Navigation > Cisco Unified CallManager Serviceability.

 

The Cisco Unified CallManager Serviceability window displays.

 

Step 2 Choose Tools > Control Center - Feature Services.

 

Step 3 From the Servers column, choose the server.

 

The server that you chose displays next to the Current Server title, and a box with configured services displays.

 

The Status column displays which services are running for the chosen server.

 

Cisco Unified CallManager System Issues

This section covers solutions for the following most common issues that relates to a Cisco Unified CallManager system.

 

• Cisco Unified CallManager System Not Responding

• Replication Fails Between the Publisher and the Subscriber

• Slow Server Response

• JTAPI Subsystem Startup Problems

• Security Issues

 

 

Cisco Unified CallManager System Not Responding

This section covers the following issues for a Cisco Unified CallManager system that is not responding:

 

• Cisco Unified CallManager System Stops Responding

• Cisco Unified CallManager Administration Does Not Display

• Error When Attempting to Access Cisco Unified CallManager Administration

• Error When Attempting to Access Cisco Unified CallManager Administration on a Subsequent Node

• You Are Not Authorized to View

• Problems Displaying or Adding Users with Cisco Unified CallManager

• Name to Address Resolution Failing

• Port 80 Blocked Between Your Browser and the Cisco Unified CallManager Server

• Improper Network Setting Exists in the Remote Machine

• Slow Server Response

 

 

Cisco Unified CallManager System Stops Responding

Symptom The Cisco Unified CallManager system does not respond.

 

When the Cisco CallManager service crashes, the following message displays in the System Event log:

The Cisco CallManager service terminated unexpectedly.  It has done this 1 time. The following corrective action  will be taken in 60000 ms. Restart the service. Other messages you may see in the event of a crash follow:
Timeout 3000 milliseconds waiting for Cisco CallManager service to connect. The Cisco CallManager failed to start due to the following error: The service did not respond to the start or control request in a timely fashion. At this time, when devices such as the Cisco Unified IP Phones and gateways unregister from the Cisco Unified CallManager, users receive delayed dial tone, and/or the Cisco Unified CallManager server freezes due to high CPU usage. For event log messages that are not included here, view the Cisco Unified CallManager Event Logs.

Possible Cause The Cisco CallManager service can crash because the service does not have enough resources such as CPU or memory to function. Generally, the CPU utilization in the server is 100 percent at that time.

 

Recommended Action Depending on what type of crash you experience, you will need to gather different data that will help determine the root cause of the crash.

Use the following procedure if a lack of resources crash occurs.


Procedure

Step 1 Collect Cisco CallManager traces 15 minutes before and after the crash.

 

Step 2 Collect SDL traces 15 minutes before and after the crash.

 

Step 3 Collect perfmon traces if available.

 

Step 4 If the traces are not available, start collecting the perfmon traces and track memory and CPU usage for each process that is running on the server. These will help in the event of another lack of resources crash.

 


 

Cisco Unified CallManager Administration Does Not Display

Symptom Cisco Unified CallManager Administration does not display.

 

Possible Cause The Cisco CallManager service stopped.

 

Recommended Action Verify that the Cisco CallManager service is active and running on the server, as described in "Verify Cisco Unified CallManager Services Are Running" section on page 2-21 or in the Cisco Unified CallManager Serviceability Administration Guide.

 

Error When Attempting to Access Cisco Unified CallManager Administration

Symptom

One of the following messages displays when you are trying to access Cisco Unified CallManager Administration.

•Internet Explorer—The page cannot be displayed.

•Netscape—Warning box displays: There was no response. The server could be down or is not responding.

 

Possible Cause

The services did not start automatically as expected. One of the services stopping represents the most frequent reason for Cisco Unified CallManager Administration not displaying.

 

Recommended Action

Try starting the other services.

 

 

Error When Attempting to Access Cisco Unified CallManager Administration on a Subsequent Node

Symptom One of the following error messages displays when you are trying to access the Cisco Unified CallManager Administration.

 

Possible Cause If the IP address of the first Cisco Unified CallManager node gets changed while a subsequent node is offline, you may not be able to log in to Cisco Unified CallManager Administration on the subsequent node.

 

Recommended Action If this occurs, follow the procedure for changing the IP address on a subsequent Cisco Unified CallManager node in the Cisco Unified Communications Operating System Administration Guide.

 

 

You Are Not Authorized to View

Symptom When accessing the Cisco Unified CallManager Administration, one of the following messages displays.

 

•You Are Not Authorized to View This Page

 

•You do not have permission to view this directory or page using the credentials you supplied.

 

•Server Application Error. The server has encountered an error while loading an application during the processing of your request. Please refer to the event log for more detailed information. Please contact the server administrator for assistance.

 

•Error: Access is Denied.

 

Possible Cause Unknown

 

Recommended Action Contact TAC for further assistance.

 

 

Problems Displaying or Adding Users with Cisco Unified CallManager

Symptom You cannot add a user or conduct a search in Cisco Unified CallManager Administration.

 

Possible Cause You may encounter the following problems if you are working with Cisco Unified CallManager that is installed on a server that has a special character (such as an underscore) in its hostname or Microsoft Internet Explorer 5.5 with SP2 and a Q313675 patch or above.

•When you conduct a basic search and hit submit, the same page redisplays.

•When you try to insert a new user, the following message displays.

The following error occurred while trying to execute the command. Sorry, your session object has timed out. Click here to Begin a New Search

 

Recommended Action You may not be able to add a user or do a search on Cisco Unified CallManager Administration, if your Cisco Unified CallManager hostname contains any special characters such as underscore or period (for example, Call_Manager). Domain Name System (DNS)-supported characters include all letters (A-Z, a-z), numbers (0-9), and hyphen (-); any special characters are not allowed. If the Q313675 patch is installed on your browser, make sure that the URL does not contain any non-DNS supported characters.

For more information about the Q313675 patch, refer to MS01-058: File Vulnerability Patch for Internet Explorer 5.5 and Internet Explorer 6.

 

To resolve this problem, you have the following options:

•Access Cisco Unified CallManager Administration by using the IP address of the server.

•Do not use non-DNS characters in the Server Name.

•Use the localhost or IP address in the URL.

 

 

Name to Address Resolution Failing

Symptom One of the following messages displays when you try to access the following URL:

 

http://your-cm-server-name/ccmadmin

 

•Internet Explorer—This page cannot be displayed

 

•Netscape—Not Found. The requested URL /ccmadmin was not found on this server.

If you try to access the same URL by using the Cisco CallManager IP address (http://10.48.23.2/ccmadmin) instead of the name, the window displays.

Possible Cause The name that you entered as "your-cm-server-name" maps to the wrong IP address in DNS or hosts file.

 

Recommended Action If you have configured the use of DNS, check in the DNS to see whether the entry for the your-cm-server-name has the correct IP address of the Cisco Unified CallManager server. If it is not correct, change it.

If you are not using DNS, your local machine will check in the "hosts" file to see whether an entry exists for the your-cm-server-name and an IP address that is associated to it. Open the file and add the Cisco Unified CallManager server name and the IP address. You can find the "hosts" file at C:\WINNT\system32\drivers\etc\hosts.

 

 

Port 80 Blocked Between Your Browser and the Cisco Unified CallManager Server

Symptom One of the following messages displays when a firewall blocks the port that is used by the web server or the http traffic:

 

•Internet Explorer—This page cannot be displayed

 

•Netscape—There was no response. The server could be down or is not responding

 

Possible Cause For security reasons, the system blocked the http access from your local network to the server network.

 

Recommended Action

 

1. Verify whether other types of traffic to the Cisco Unified CallManager server, such as ping or Telnet, are allowed. If any are successful, it will show that http access to the Cisco Unified CallManager web server has been blocked from your remote network.

 

2. Check the security policies with your network administrator.

 

3. Try again from the same network where the server is located.

 

 

Improper Network Setting Exists in the Remote Machine

Symptom No connectivity exists, or no connectivity exists to other devices in the same network as the Cisco Unified CallManager.

 

When you attempt the same action from other remote machines, Cisco Unified CallManager Administration displays.

 

Possible Cause Improper network configuration settings on a station or on the default gateway can cause a web page not to display because partial or no connectivity to that network exists.

 

Recommended Action

 

1. Try pinging the IP address of the Cisco Unified CallManager server and other devices to confirm that you cannot connect.

 

2. If the connectivity to any other device out of your local network is failing, check the network setting on your station, as well as the cable and connector integrity. Refer to the appropriate hardware documentation for detailed information.

 

If you are using TCP-IP over a LAN to connect, continue with the following steps to verify the network settings on the remote station.

 

3. Choose Start > Setting > Network and Dial-up connections.

 

4. Choose Local Area Connection, then Properties.

 

The list of communication protocols displays as checked.

 

5. Choose Internet Protocol (TCP-IP) and click Properties again.

 

6. Depending on your network, choose either Obtain an ip address automatically or set manually your address, mask and default Gateway.

 

The possibility exists that a browser-specific setting could be improperly configured.

 

7. Choose the Internet Explorer browser Tools > Internet Options.

 

8. Choose the Connections tab and then verify the LAN settings or the dial-up settings.

 

By default, the LAN settings and the dial-up settings do not get configured. The generic network setting from Windows gets used.

 

9. If the connectivity is failing only to the Cisco Unified CallManager network, a routing issue probably exists in the network. Contact the network administrator to verify the routing that is configured in your default gateway.

NOTE: If you cannot browse from the remote server after following this procedure, contact TAC to have the issue investigated in more detail.

 

 

Replication Fails Between the Publisher and the Subscriber

Replicating the database is a core function of Cisco Communications Manager clusters. The server with the master copy of the database is called the publisher, while the servers replicating the database are called subscribers.

Symptom Changes made on the publisher are not reflected on phones that are registered with the subscriber.

 

Possible Cause Replication fails between the publisher and subscriber.

 

Recommended Action Complete the following steps to reestablish the relationship between the two systems.

 

1. Verify the Replication.

 

a. Open Cisco Unified Communications Manager Real-Time Monitoring Too (RTMT).

 

b. Choose System > Performance > Open Peformance Monitoring.

 

c. Double-click the publisher node to expand the performance monitors.

 

d. Double-click Replication Counters.

 

e. Double-click Number of Replicates Created.

 

f. Choose ReplicateCount from the Object Instances dialog box and click Add.

 

g. Double-click Replication Status.

 

h. Choose ReplicateCount from the Object Instances dialog box and click Add.

NOTE: Right click the counter name and choose Counter Description to view the definition of the counter.

2. Check State of Replication via CLI.

 

a. Access the platform CLI and use the following command to check repliaction:

 

utils dbreplication status file view activelog <filename_output_above>

 

b. Review the Summary information and counts for each node to verify replication.

 

3. To repair replication, use the following procedure:

 

a. Access the platform CLI.

 

b. Repair replication by using the following command:

utils dbreplication repair usage:utils dbreplicatoin repair [nodename]|all

 

Slow Server Response

This section addresses a problem that relates to a slow response from the server due to mismatched duplex port settings.

Symptom Slow response from the server occurs.

 

Possible Cause

 

Slow response could result if the duplex setting of the switch does not match the duplex port setting on the Cisco Unified CallManager server.

 

Recommended Action

 

1. For optimal performance, set both switch and server to 100/Full.

 

Cisco does not recommend using the Auto setting on either the switch or the server.

 

2. You must restart the Cisco Unified CallManager server for this change to take effect.

 

 

JTAPI Subsystem Startup Problems

The JTAPI (Java Telephony API) subsystem represents a very important component of the Cisco Customer Response Solutions (CRS) platform. JTAPI communicates with the Cisco Unified CallManager and has responsibility for telephony call control. The CRS platform hosts telephony applications, such as Cisco Unified AutoAttendant, Cisco IP ICD, and Cisco Unified IP-IVR. Although this section is not specific to any of these applications, keep in mind that the JTAPI subsystem is an underlying component that all of them use.

Before starting the troubleshooting process, ensure that the software versions that you are using are compatible. To verify compatibility, read the Cisco Unified CallManager Release Notes for the version of Cisco Unified CallManager that you are using.

To check the version of CRS, log in to the AppAdmin page by typing http://servername/appadmin, where servername is the name of the server on which CRS is installed. The current version is located in the lower-right corner of the main menu.

 

 

JTAPI Subsystem is OUT_OF_SERVICE

Symptom

The JTAPI subsystem does not start.

Possible Cause

One of the following exceptions displays in the trace file:

 

• MIVR-SS_TEL-4-ModuleRunTimeFailure

• MIVR-SS_TEL-1-ModuleRunTimeFailure

 

MIVR-SS_TEL-4-ModuleRunTimeFailure Search for the MIVR-SS_TEL-1-ModuleRunTimeFailure string in the trace file. At the end of the line, an exception reason appears.

 

The following list gives the most common errors:

 

• Unable to create provider—bad login or password

• Unable to create provider—Connection refused

• Unable to create provider—login=

• Unable to create provider—hostname

• Unable to create provider—Operation timed out

• Unable to create provider—null

Unable to create provider—bad login or password

 

Possible Cause

Administrator entered an incorrect user name or password in the JTAPI configuration.

 

Full Text of Error Message

 

%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem,Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- bad login or password. %MIVR-SS_TEL-7- EXCEPTION:com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- bad login or password. Recommended Action

Verify that the user name and password are correct. Try logging into the Unified CMuser page (http://servername/ccmuser) on the Cisco Unified CallManager to ensure that the Cisco Unified CallManager cannot authenticate correctly.

 

Unable to create provider—Connection refused

 

Possible Cause

The Cisco Unified CallManager refused the JTAPI connection to the Cisco Unified CallManager.

 

Full Text of Error Message

 

%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- Connection refused %MIVR-SS_TEL-7-EXCEPTION:com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- Connection refused Recommended Action

Verify that the CTI Manager service is running in the Cisco Unified CallManager Control Center.

Unable to create provider—login=  Possible Cause

Nothing has been configured in the JTAPI configuration window.

Full Text of Error Message

%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- login= %MIVR-SS_TEL-7-EXCEPTION:com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- login= Recommended Action

Configure a JTAPI provider in the JTAPI configuration window on the CRS server.

 

Unable to create provider—hostname

 

Possible Cause

The CRS engine cannot resolve the host name of the Cisco Unified CallManager.

 

Full Text of Error Message

%M%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- dgrant-mcs7835.cisco.com %MIVR-SS_TEL-7-EXCEPTION:com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- dgrant-mcs7835.cisco.com Recommended Action

Verify that DNS resolution is working correctly from the CRS engine. Try using an IP address instead of the DNS name.

Unable to create provider—Operation timed out

 

Possible Cause

The CRS engine does not have IP connectivity with the Cisco Unified CallManager.

 

Full Text of Error Message

101: Mar 24 11:37:42.153 PST %MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- Operation timed out 102: Mar 24 11:37:42.168 PST %MIVR-SS_TEL-7-EXCEPTION: com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- Operation timed out

 

Recommended Action

Check the IP address that is configured for the JTAPI provider on the CRS server. Check the default gateway configuration on the CRS server and the Cisco Unified CallManager. Make sure no IP routing problems exist. Test connectivity by pinging the Cisco Unified CallManager from the CRS server.

Unable to create provider—null

Possible Cause

No JTAPI provider IP address or host name get configured, or the JTAPI client is not using the correct version.

 

Full Text of Error Message

 

%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider -- null Recommended Action

 

Verify that a host name or IP address is configured in the JTAPI configuration. If the JTAPI version is incorrect, download the JTAPI client from the Cisco Unified CallManager Plugins window and install it on the CRS server.

 

MIVR-SS_TEL-1-ModuleRunTime

 

Failure  Symptom

This exception usually occurs when the JTAPI subsystem is unable to initialize any ports.

Possible Cause

The CRS server can communicate with the Cisco Unified CallManager, but is unable to initialize any CTI ports or CTI route points through JTAPI. This error occurs if the CTI ports and CTI route points are not associated with the JTAPI user.

 

Full Text of Error Message

 

255: Mar 23 10:05:35.271 PST %MIVR-SS_TEL-1-ModuleRunTimeFailure: Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_SS,Exception=null Recommended Action

 

Check the JTAPI user on the Cisco Unified CallManager and verify that CTI ports and CTI route points that are configured on the CRS server associate with the user.

 

 

JTAPI Subsystem is in PARTIAL_SERVICE

Symptom The following exception displays in the trace file:

 

MIVR-SS_TEL-3-UNABLE_REGISTER_CTIPORT

 

Possible Cause The JTAPI subsystem cannot initialize one or more CTI ports or route points.

Full Text of Error Message

 

1683: Mar 24 11:27:51.716 PST %MIVR-SS_TEL-3-UNABLE_REGISTER_CTIPORT: Unable to register CTI Port: CTI Port=4503, Exception=com.cisco.jtapi.InvalidArgumentExceptionImpl: Address 4503 is not in provider's domain.

1684: Mar 24 11:27:51.716 PST %MIVR-SS_TEL-7-EXCEPTION: com.cisco.jtapi.InvalidArgumentExceptionImpl: Address 4503 is not in provider's domain.

 

Recommended Action

 

The message in the trace tells you which CTI port or route point cannot be initialized. Verify that this device exists in the Cisco Unified CallManager configuration and also associates with the JTAPI user on the Cisco Unified CallManager.

 

Security Issues

This section provides information about security-related measurements and general guidelines for troubleshooting security-related problems. This section contains information on the following topics:

 

• Security Alarms

• Security Performance Monitor Counters

• Reviewing Security Log and Trace Files

• Troubleshooting Certificates

• Troubleshooting CTL Security Tokens

• Troubleshooting CAPF

• Troubleshooting Encryption for Phones and Cisco IOS MGCP Gateways

 

NOTE: This section does not describe how to reset the Cisco Unified IP Phone if it has been corrupted by bad loads, security bugs, and so on. For information on resetting the phone, refer to the Cisco Unified IP Phone Administration Guide for Cisco Unified CallManager that matches the model of the phone.

 

For information about how to delete the CTL file from Cisco Unified IP Phone models 7970, 7960, and 7940 only, see the Cisco Unified CallManager Security Guide or the Cisco Unified IP Phone Administration Guide for Cisco Unified CallManager that matches the model of the phone.

 

 

Security Alarms

Cisco Unified CallManager Serviceability generates security-related alarms for X.509 name mismatches, authentication errors, and encryption errors. The Serviceability GUI provides the alarm definitions.

 

Alarms may get generated on the phone for TFTP server and CTL file errors. For alarms that get generated on the phone, refer to the Cisco Unified IP Phone Administration Guide for Cisco Unified CallManager for your phone model and type (SCCP or SIP).

 

 

Security Performance Monitor Counters

Performance monitor counters monitor the number of authenticated phones that register with Cisco Unified CallManager, the number of authenticated calls that are completed, and the number of authenticated calls that are active at any time. Table 3-1 lists the performance counters that apply to security features.


Table 3-1 Security Performance Counters

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: