Please confirm the status of the CAR essential services. Under Serviceability pages (In the upper right dropdown on the Administration pages):
- System> Service Parameters> Cisco CallManager service
- CDR Enabled Flag
- CDR Log Calls with zero duration flag
- Call Diagnostics Enabled
- System-->Enterprise Parameters-->CDR Parameters
- CDR File Time Interval: 1
- Cluster ID: StandAloneCluster
- Tools -> Feature Services
- Cisco SOAP - CDRonDemand Service (This should be running on the First Node only)
- Cisco CAR Web Service (Again, First Node only)
- Tools -> Network services
- Cisco CDR Repository Manager (this one should be running on the First Node only)
- Cisco CDR Agent (these should be running on all the call processing nodes)
- Cisco CAR Scheduler (this should be running on the First Node only)
- If all the above services are running okay, then, as a first step, can you please try this:-
- Disable the "CDR Enabled Flag" parameter in all the servers in your cluster.
- Restart the "Cisco CDR Agent" service in all the servers in your cluster.
- Restart the "Cisco CDR Repository Manager" service in the publisher server.
- Restart the "Cisco Database Layer Monitor" service in all the servers in your cluster.
- Enable again the "CDR Enabled Flag" parameter in all the servers in your cluster.
If the above steps don't address the issue, One possibility for this error could be that the loader is failing to load the records into CAR tables. One would need the following information (screenshots, outputs of show commands, output of 'run sql' queries', traces and logs) to figure out what's going on:-
For tracing:
- Please enable the following trace levels in the Publisher (Cisco Unified CallManager Serviceability->Trace->Configuration) :
- Cisco CAR Scheduler (trace level: Debug)
- Cisco CAR Web Service (trace level: Debug)
- Cisco CDR Agent (trace level: Debug)
- Cisco CDR Repository Manager (trace level: Debug)
/* Set max No. of files to 200. Set Max. file size to 2M. */
- Make sure CDR Load is not disabled in CAR and there is a checkmark on "continously load 24/7"; set the "Load CDR & CMR" Time, Loading interval and Duration to be the moment you perform this test:
System->Scheduler->CDR Load (you will need to restart CAR Scheduler service to make the changes effective)
- Restart the following services (Cisco Unified CallManager Serviceability->Tools->Control Center - Feature/Network Services:
- Cisco CAR Scheduler
- Cisco CAR Web Service
- Cisco CDR Repository Manager
- Cisco CDR Agent
- Recreate the issue (i.e. try searching for the records) and note down the exact time you performed the search and what extension you searched for.
- Next, after 15 to 30 minutes, use the RTMT tool to collect the following logs files. This will show us all the process of loading the cdr files to the CAR tool (from preserve to processed folder).
- 'Cisco CAR Web service'
- 'Cisco CAR Scheduler'
- 'Cisco CDR Agent'
- 'Cisco CDR Repository Manager '
- 'Cisco Tomcat Stats Servlet'
- 'Cisco Tomcat'
- 'Event viewer-application log'
- 'Event viewer-System log'
- 'Install and Upgrade logs'
- 'Cisco CDR Files on Publisher Processed'
This document can help in the process of enabling and collecting traces: Set Up Cisco CallManager Traces for Cisco Technical Support
- Please take the output of the following command when run from the CLI:
admin:run sql select max(datetimeorigination) from car:tbl_billing_data
/* That query will show us the time of the latest CDR record inserted in the CAR database. Please collect the result of that command and note know the exact time when you executed the query. */
- Also, please check the status of the CAR Scheduler using CAR tool's EventLog ( System --> Log Screens --> Event Log ). Please provide me the screenshot.
- Collect the output for the following,to know
- how many records are there in the CAR database and
- see the output of the tbl_system_preferences table:-
- run sql select * from tbl_system_preferences
- run sql select * from car:tbl_event_log
- run sql select * from car:tbl_scheduler where name='DailyCdrLoad'
- run sql select count(*) from car:tbl_billing_data
- run sql select count(*) from car:tbl_billing_error
- run sql select count from car:tbl_error_id_map
- run sql select count from car:tbl_dump_calldetailrecord
- run sql select count from car:tbl_dump_calldetailrecorddiagnostic
- Also, Please provide the output of the below command so I may verify the status of the loader since November the 19th:-
run sql select * from car:tbl_event_log where start_time > '2007-11-19 00:00:00' and job_description='DailyCdrLoad'
- I'll also need to see whether the files are moving through the file system as they should. Here's how the CDR management works in your version of CUCM:
- CDR file(s) are written in 'activelog cm/cdr/' on local nodes
- CDR file(s) are transferred by the CDR Agent(s) from the different nodes to the 'trans' folder on the CDRM node.
'activelog cm/cdr_repository/trans/'
- CDR file(s) are transferred by the local CDR Agent on the CDRM node to 'activelog cm/cdr_repository/tmp/' and are ready to be processed.
- If any destination (billing servers) or CAR is active, the CDR file(s) are transferred to the preserve folder with date subdirectory.
'activelog cm/cdr_repository/preserve/<date>'
- From here the CDR files will be transferred to the billing destination(s) or CAR.
- When successful these files are placed in the processed folder with the same date subdirectory
'activelog cm/cdr_repository/processed/<date>'
- When the transfer fails to CAR the files are placed in 'CAR' folder inside the <date> sub-folder.
'activelog cm/cdr_repository/car/<date>'
- When the transfer fails to a destination (billing server), the files are placed in 'Destination<nr>' folder with date.
'activelog cm/cdr_repository/destination[1-3]/<date>'
- CDRM will try to transfer these files till success, till these files get too old or till they get removed by human intervention.
- So please take access to the server via the CLI and provide the output of the "file list" commands when run on all the above folders.
