CM 6.1.2 CDR not writing to SUB

Answered Question

Ladies and Gentlemen:

Architecture

1 PUB

2 SUBs

We just migrated from CM 4.x to CUCM 6.1.2

We are completely migrated and don't care about old 4.x records.

Every thing seemed to go fine,

but we noticed that we have just a few call records on one of our SUB Servers (SUB1) I am looking at Serviceability>>>CDR Analysis>>extension number

All phones register to the SUB1 or SUB2. The records are fine on SUB2

All Gateways register to SUB1 or SUB2.

We ran a trace file and see that all the call records are being written to the PUB just fine but, they never make it to the SUB1 db for some reason.

Can anyone give me a logical place to start looking? To fix the problem.

Thanks, Tom

I have this problem too.
0 votes
Correct Answer by htluo about 7 years 10 months ago

Tom,

The work flow above is to give you some ideas how CDR/CAR work.

Depending on different symptoms, we follow different paths to troubleshoot.

For example, if you suspected call records for one of the subscriber were missing, you troubleshoot like this:

1) Check the "service parameter" for that SUB. Make sure CDR flag was turned on.

2) Pick a phone that was registered to that SUB and make a test call.

3) Pull the CCM trace from that SUB. You should see something like below. This is CDR data being written to CDR file.

01/19/2009 22:15:22.661 CCM|EnvProcessCdr::outputCdrData CDR data - 1,2,7039,43064648,1232424915,2,0,0,"6009","",0,0,4,0,24576,4,20,0,0,0,0,0,0,"0","0",43064649,2,0,-1836951542,"6002","6002","htluo",0,16,4,-1836951542,19358,4,20,0,0,0,0,0,0,"0","0",1232424918,1232424922,"6002","5de6ff2a-2199-4e86-b0b6-b22977e76d28","DallasPT","","DallasPT","DallasPT",4,"SEP0010C6E27EC1","SEP001E7A24429A",0,12,0,0,0,0,0,"StandAloneCluster",0,"","",0,"",3,3,0,0,64,64,""

4) When you check the "preserve" or "processed" folder, pay attention to the file names. It'll be in the format of:

cdr____

For example, "cdr_CM6_02_200901200416_63" means it's a CDR file for "CM6" cluster, node 02, Jan 20, 2009, 04:16 GMT.

HTH

Michael

http://htluo.blogspot.com/

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (5 ratings)
Loading.
htluo Fri, 01/16/2009 - 14:58

what do you mean "they never make it to the SUB1 db"? could you be more specific?

CDR flat files are created on each node with service parameter "CDR Enabled Flag" set to true. (you might want to check this on SUB1).

CDR flat files are transferred to publisher for processing.

CAR scheduler on publisher will process those flat files and insert into database.

HTH

Michael

My blog:

http://htluo.blogspot.com/

Michael,

Sorry for being so vague. I thought the CDRs were written to the PUB and then transferred/written to the SUB1. Is it just the opposite?

Should the FLAG be enabled on all Servers or just the Servers that have registered phones?

Should the scheduler be enabled on all Servers or just the Servers that have registered phones?

I need to check my FLAGS and Schedulers in either case. Thank you for your response.

Tom

htluo Sat, 01/17/2009 - 07:58

The flag should be enabled on the servers that have registered phones.

CAR scheduler should only runs on PUB.

My question is "how did you find out CDR not writing to SUB?". What command or what tools you used to find out?

thanks!

Michael,

I was working with [email protected] from TAC in the RTP. We did several trace files together. Sri said that the CDRs are being written to the PUB but not transferring over to the SUB via flat file. I have to admit that the trace procedure is a little confusing for me at this point. I assumed that Sri knew. He seemed very knowledgeable on the topic.

I also ran the CDR Analysis Report in the Serviceability page on the PUB using my extension as a test.

I have been using this document as a guide for services.

http://supportwiki.cisco.com/ViewWiki/index.php/Troubleshooting_CUCM_5.x_/_6.x_CDR_Analysis_and_Reporting_%22Data_Missing%22_issues

This document says (Enable again the "CDR Enabled Flag" parameter in all the servers in your cluster.) We have all phones registering to SUBs only.

You are saying the flag should be enabled on the servers that have registered phones only. Right?

I just checked the flag is enabled on all the Servers PUB, SUB1 and SUB2.

The Scheduler is running on the PUB and SUB1 not SUB2 which works fine.

Could this be the problem? Thanks Tom

htluo Sun, 01/18/2009 - 08:12

Please see attached diagram for CDR workflow.

Here's the detailed explanation:

1) You made a call with IP phone.

2) Call was connected.

3) You hang up the phone.

4) Call was disconnected.

5) Let say, if the IP phone was registered to SUB1. The "Cisco CallManager" process on SUB1 will write the call record to a local file. We call this "CDR flat file". Those files are stored in cm/cdr folder. Command to see this folder is "file list activelog cm/cdr detail". However, in normal situation, you won't be able to see the CDR files (I'll explain later). But you will see the timestamp on "_cdrIndex.idx" being updated.

6) On SUB1, there's a process called "CDR Agent". This process' job is to keep moving CDR files from SUB to PUB. In a normal situation, the files are moved pretty quickly. That's why you won't be able to see any files on SUB1 with command "file list activelog cm/cdr detail".

7) On PUB, CDR files from all SUBs are put in cm/cdr_repository folder. (you may use "file list activelog cm/cdr_repository detail" to view the content of this folder). There are several sub-folders. The ones you're interested in would be "preserve" and "processed".

8) Files would be put in "preserve" folder first. "CAR Scheduler" will process the files in "preserve" folder, insert the data into CAR database, then move the files to "processed" folder.

9) If you saw lots of files in "preserve" folder but not in "processed" folder, that means "CAR Scheduler" has not processed those files yet. It could be Scheduler waiting for the time to come or Scheduler has a problem.

Please note: all timestamps (including folder names) are in UTC time. A call made on Jan 18, the CDR file might be in 20090119 folder.

Hope this helps.

Michael

http://htluo.blogspot.com/

Attachment: 
rob.huffman Sun, 01/18/2009 - 08:57

Hi Michael,

Nice work here! +5 points for this great explanation of a rather complicated process.

Cheers!

Rob

Michael,

I agree with Rob this is a nice diagram and explanation. Thanks you.

I ran admin:file list activelog cm/cdr_repository/ on the preserve and processed directories. On the PUB.

dir count = 32, file count = 0 in the preserve and processed directories. I guess this means the scheduler is working properly.

I have tried some different options with my services but I think there are still some issues. Do you feel that the document below is accurate? I notice a slight difference to what you suggested for services.

http://supportwiki.cisco.com/ViewWiki/index.php/Troubleshooting_CUCM_5.x_/_6.x_CDR_Analysis_and_Reporting_%22Data_Missing%22_issues

Thanks Tom

htluo Mon, 01/19/2009 - 16:49

"file list activelog cm/cdr_repository" will only show you the content of the repository folder.

You have to dig deeper, ie. "file list activelog cm/cdr_repository/preserve/20090119".

HTH

Michael

Correct Answer
htluo Mon, 01/19/2009 - 20:52

Tom,

The work flow above is to give you some ideas how CDR/CAR work.

Depending on different symptoms, we follow different paths to troubleshoot.

For example, if you suspected call records for one of the subscriber were missing, you troubleshoot like this:

1) Check the "service parameter" for that SUB. Make sure CDR flag was turned on.

2) Pick a phone that was registered to that SUB and make a test call.

3) Pull the CCM trace from that SUB. You should see something like below. This is CDR data being written to CDR file.

01/19/2009 22:15:22.661 CCM|EnvProcessCdr::outputCdrData CDR data - 1,2,7039,43064648,1232424915,2,0,0,"6009","",0,0,4,0,24576,4,20,0,0,0,0,0,0,"0","0",43064649,2,0,-1836951542,"6002","6002","htluo",0,16,4,-1836951542,19358,4,20,0,0,0,0,0,0,"0","0",1232424918,1232424922,"6002","5de6ff2a-2199-4e86-b0b6-b22977e76d28","DallasPT","","DallasPT","DallasPT",4,"SEP0010C6E27EC1","SEP001E7A24429A",0,12,0,0,0,0,0,"StandAloneCluster",0,"","",0,"",3,3,0,0,64,64,""

4) When you check the "preserve" or "processed" folder, pay attention to the file names. It'll be in the format of:

cdr____

For example, "cdr_CM6_02_200901200416_63" means it's a CDR file for "CM6" cluster, node 02, Jan 20, 2009, 04:16 GMT.

HTH

Michael

http://htluo.blogspot.com/

Michael,

This is a follow up to the CDR problem I had. I had to take the problem SUB1 off line yesterday. The PUB and SUB1 would not accept any changes yesterday. For example call FWD, but SUB2 would. SUB2 is across town connected by fiber on a different subnet.

Now that I took SUB1 off line the CDR and changes are working fine. Virtue of the fact the problem box is no longer there. TAC wanted us to apply a series of ES, but I thought I should just rebuild the box from scratch/patch and put it back on-line.

I can not help think through all of this that it might be a permissions issue. I was thinking gateway issue too but the PUB and SUB1 are right next to each other in the rack. The only thing that worries me is that I put in new gateways (2851's) from 6608's.. 2 days before the big migration job. I will let you know how things turn out. CDR was working fine before all of this.

I appreciate your help. I have much more experience now with the CLI and Trace Files.

I consider this issue resolved because I think the underlying problem is something other than CDR.

Tom

Vaijanath Sonvane Sun, 08/16/2009 - 23:54

Hi Michael,

We are having one Publisher and 2 Subscribers with version 7.1.2.20000-2. We observerd that, after 10-15 days the CCM service of only one Subscriber restarts automatically . We have collected the Syslog & Alarm logs and we are getting the below error:

"

Aug 11 13:15:21, KCMCS02, Error, Cisco CallManager, ccm: 7220: Aug 11 07:45:21.211 UTC : ÌM_CALLMANAGER-CALLMANAGER-3-CallManagerFailure: Indicates some failure in the Cisco CallManager system. Host name of hosting node.:KCMCS02 IP address of hosting node.:10.50.1.11 Reason code.:3 Additional Text [Optional]: Cluster ID:KCMCS01-Cluster Node ID:KCMCS02, 5886

Aug 11 13:15:21, KCMCS02, Error, Cisco CallManager, ccm: 7221: Aug 11 07:45:21.212 UTC : ÌM_CALLMANAGER-CALLMANAGER-3-CallManagerFailure: Indicates some failure in the Cisco CallManager system. Host name of hosting node.:KCMCS02 IP address of hosting node.:10.50.1.11 Reason code.:3 Additional Text [Optional]:CCM Intentional Abort: SignalName: PublishRetryAfterTimer, DestPID: PublishEPA[2:100:81:13] Cluster ID:KCMCS01-Cluster Node ID:KCMCS02, 5887

"

Does anyone faced this problem? Is there any solution for this?

Need help...

Actions

This Discussion