Hi James,
I've also dealt with this sort of issue. It's important to understand what's causing the calls in progress to be at such a high value even during non business hours. A reboot of CUCM server(s) would clear the problem temporarily if it is a defect. The issue I worked on required indepth troubleshooting to help establish it was a defect in the CUCM version. Let me share some details regarding this:
"Calls in Progress" represent the number of calls that are currently in process. It includes all active calls also (Calls Active is defined as the number of streaming connections that are currently active (in use), in other words the number of calls which actually have a voice path connected. When a phone goes off hook, it is a call in progress until it goes back on hook. If all calls that are in progress are connected, the number of calls in progress and the number of active calls will be the same. Call Control updates these counters. The CallInProgress counter is incremented when Cdcc process is started and decremented when the process is stopped.
Either the Call Control (most likely) is not working properly or we have a problem with the RTMT itself.
I had to analyze 24 hours worth of CUCM SDI and SDL traces (~20GB zipped!) to catch the issue, where Cdcc's were getting hung. This was related to call pickup groups and was a defect in 8.x.
If you happen to see the problem again post the reload of the CUCM server(s), here are the things to collect and analyze:
++ Collect the following from the CLI of the Publisher:
show status
show hardware
show perf query class "Number of Replicates Created and State of Replication"
show perf query counter "Cisco Locations" CallsInProgress
show perf query counter "Cisco CallManager" CallsInProgress
show perf query class "Cisco Lines"
++ Are you able to identify which location(s) are showing non-existent calls?
++ Kindly let me know if you see any low memory alerts / alarms in RTMT. If yes, kindly provide me with the alert / alarm details.
++ Application and system log from publisher.
++ Via GUI - during off hours, proceed to restart the following services on serviceability to each server, first publisher and then subscribers.
Cisco RIS Data Collector
Cisco AMC Service
++ Via CLI -
utils snmp hardware-agents restart
Additional steps to troubleshoot:
1. Enable trace Cisco RIS Datacollector to detailed
2. Enable trace Cisco AMC service to debug
3. Enable trace Risdc logs. To enable, Go to the service parameter
page of "Cisco RIS Data Collector" (RisDC) service and enable the
"Ris Logging" service parameter. PS: enable this trace only in
the server that we are having the problem.
4. On the PC that will run the RTMT: Start RTMT go to Edit > Trace
Setting > debug. The log will be at: c:\Documents and
Settings\\.jrtmt\log\rtmt.log
5. Now, schedule a maintenance window. In the maintenance window to
stop/start the Cisco CallManager Service on the node reporting high value for "calls in progress". Note that
will drop all the calls.
6. check if the calls in progress goes to zero.
7. If not reboot the server
8. Check if the calls in progress goes to zero.
9. When you get the counter back to zero, capture every couple of hours:
10. output of CLI show perf query counter "Cisco CallManager CallsInProgress"
11. screenshoot for the RTMT
12. Screen-shoot of
https://:8443/ast/ASTIsapi.dll?GetPreCannedInfo&Items=getCallActivityRequest
We want to find a moment were we get a good discrepancy between what the RTMT shows and what the CLI commands report. At this point collect all the logs and the traces.
HTH.
Regards,
Harmit.
