Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
5362
Views
40
Helpful
18
Replies

CUCM replication mismatch (deep research)

GlKlondike
Level 1
Level 1

‎07-19-2017 07:56 AM - edited ‎03-17-2019 10:49 AM

Hi,

We have two servers Pub and Sub

After replication problem, execute reset replication cluster, but setup not completed 

DB and Replication Services: ALL RUNNING

DB CLI Status: No other dbreplication CLI is running...

Cluster Replication State: BROADCAST SYNC Completed on 1 servers at: 2017-07-19-17-55
Last Sync Result: SYNC COMPLETED 603 tables sync'ed out of 603
Sync Errors: 4 Errors or failures were found
Use this command to see the output- file view activelog /cm/trace/dbl/2017_07_19_17_53_02_dbl_repl_cdr_Broadcast.log

DB Version: ccm9_1_2_11900_12
Repltimeout set to: 300s
PROCESS option set to: 1

Cluster Detailed View from callmanager1 (2 Servers):

PING CDR Server REPL. DBver& REPL. REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) RPC? (ID) & STATUS QUEUE TABLES LOOP? (RTMT) & details
----------- ------------ ------ ---- -------------- ----- ------- ----- -----------------
callmanager1 10.0.3.11 0.051 Yes (2) Connected 0 match No (2) PUB Setup Completed
callmanager2 10.0.4.11 4.861 Yes (4) Connected 0 match N/A (4) Setup Completed

We have 4 sync errors, for example one of this:

Jul 18 2017 15:44:01 ------ Table scan for
ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1 0 0 1 0


The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:abcdd7a0-4c70-46f6-9e58-d56bf1d409ff
------------------------------------------------------------------
WARNING: replicate is not in sync

Ok, i understand, problem in replicationdynamic table.

Now show this table on Pub, execute run sql select * from replicationdynamic

admin:run sql select * from replicationdynamic
pkid fkprocessnode datetimestamp
==================================== ==================================== =============
abcdd7a0-4c70-46f6-9e58-d56bf1d409ff b5efd7a0-4c70-46f6-9e58-d56bf1d409ff 1497420013

Same command on Sub

admin:run sql select * from replicationdynamic
pkid fkprocessnode datetimestamp
==================================== ==================================== =============
abcdd7a0-4c70-46f6-9e58-d56bf1d409ff b5efd7a0-4c70-46f6-9e58-d56bf1d409ff 1497420013

This value the same on Pub and Sub, why cucm not sync?

Plz help, i dont have a choice to call TAC

Labels:
0 Helpful
18 Replies 18

Jitender Bhandari
Cisco Employee
Cisco Employee

‎07-19-2017 08:07 AM

Hi,

Try below steps the way they are listed and let me know how it goes.

1) Stopped the replication on all the nodes by running the command "utils dbreplication stop" (1st on sub's and then on pub)
(2) Ran the command "utils dbreplication dropadmindb" on all the nodes (1st the pub and then the sub's)
(3) Ran the command "utils dbreplication reset all" on the publisher.

(Rate if it helps)

JB

GlKlondike
Level 1
Level 1
In response to Jitender Bhandari

‎07-20-2017 12:30 AM

Thanks for answer, i try it before, this not help.

Sync log after cluster reset, same error in the table, maybe we can delete this row in sql request?

It just one row in credentialdynamic  table.

Jul 20 2017 10:35:53 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1126 0 0 1 0


The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 10:35:59 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end


Jul 20 2017 10:35:31 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1 0 0 1 0


The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:abcdd7a0-4c70-46f6-9e58-d56bf1d409ff
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 10:35:36 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic end ---------

0 Helpful

Jitender Bhandari
Cisco Employee
Cisco Employee
In response to GlKlondike

‎07-20-2017 01:16 AM

Hi

Lets start from basics 

Can you attach the below output in .txt file to post, all from PUB

1) show network cluster

2) utils diagnose test

3) utils dbreplication status

4) show tech network hosts

JB

GlKlondike
Level 1
Level 1
In response to Jitender Bhandari

‎07-20-2017 01:31 AM

Thank you for taking time 

I attach, what you need.

Now i compared credentialdynamic  table with the field where sync show error

Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733

On Pub

eb4f68a2-744a-401b-a41e-5515d68c5733 0         0                 0              1497420200      

66892066-fa9c-4789-816e-9fa6d15adb32 1497420200    1497420145      

On Sub        

eb4f68a2-744a-401b-a41e-5515d68c5733 0         0                 0              1500537611       66892066-fa9c-4789-816e-9fa6d15adb32 1500537610    1500537611              

In a field timelastaccessed datetimestamp lastsuccessfullogintime value not the same, how you see above.

This pkid connected to Application user "axluser"

How i can change this value in the table?

Preview file
8 KB
0 Helpful

Jitender Bhandari
Cisco Employee
Cisco Employee
In response to GlKlondike

‎07-20-2017 01:42 AM

Hi

I think a cluster reboot has already been done, is that correct?

can you attach the output of .txt file for below.

file view activelog /cm/trace/dbl/2017_07_20_10_33_40_dbl_repl_cdr_Broadcast.log

JB

GlKlondike
Level 1
Level 1
In response to Jitender Bhandari

‎07-20-2017 02:04 AM

Yeah, it's correct

File attached. It is first today cluster setup and automatically next try.

Thanks

Preview file
362 KB
Preview file
464 KB
0 Helpful

Jitender Bhandari
Cisco Employee
Cisco Employee
In response to GlKlondike

‎07-20-2017 02:58 AM

Hi

Most of the things look good with your replication other then below

Jul 20 2017 10:24:24 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 1 0 0 1 0


The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:abcdd7a0-4c70-46f6-9e58-d56bf1d409ff
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 10:24:29 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_243_replicationdynamic end ---------


Jul 20 2017 10:24:45 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 8
g_callmanager2_ccm9_1_2_11900_12 1126 0 0 8 0


The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 10:24:51 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end ---------


Jul 20 2017 10:25:09 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_561_callforwardhistorydynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 627 0 0 0 2
g_callmanager2_ccm9_1_2_11900_12 627 0 0 2 0


The repair operation completed. Validating the repaired rows ...
Validation completed successfully.
Jul 20 2017 10:25:11 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_561_callforwardhistorydynamic end ---------


Jul 20 2017 10:25:11 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_560_callforwarddynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 183 0 0 0 1
g_callmanager2_ccm9_1_2_11900_12 183 0 0 1 0


The repair operation completed. Validating the repaired rows ...
Validation completed successfully.
Jul 20 2017 10:25:13 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_560_callforwarddynamic end ---------

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

DB CLI Status: No other dbreplication CLI is running...

Cluster Replication State: BROADCAST SYNC Completed on 1 servers at: 2017-07-20-10-36
Last Sync Result: SYNC COMPLETED 603 tables sync'ed out of 603
Sync Errors: 4 Errors or failures were found
Use this command to see the output- file view activelog /cm/trace/dbl/2017_07_20_10_33_40_dbl_repl_cdr_Broadcast.log

DB Version: ccm9_1_2_11900_12
Repltimeout set to: 300s
PROCESS option set to: 1

Cluster Detailed View from callmanager1 (2 Servers):

PING CDR Server REPL. DBver& REPL. REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) RPC? (ID) & STATUS QUEUE TABLES LOOP? (RTMT) & details
----------- ------------ ------ ---- -------------- ----- ------- ----- -----------------
callmanager1 10.0.3.11 0.040 Yes (2) Connected 0 match No (2) PUB Setup Completed
callmanager2 10.0.4.11 5.874 Yes (4) Connected 0 match N/A (4) Setup Completed


"REPL. LOOP?" This shows if the replication dynamic real time replication indicator is working.

Can you try "utils dbreplication rebuild all" once

JB

GlKlondike
Level 1
Level 1
In response to Jitender Bhandari

‎07-20-2017 03:12 AM

Thanks, yeah, i know this error, cucm not repair table credentialdynamic, because 

On Pub

eb4f68a2-744a-401b-a41e-5515d68c5733 0         0                 0              1497420200      

66892066-fa9c-4789-816e-9fa6d15adb32 1497420200    1497420145      

On Sub        

eb4f68a2-744a-401b-a41e-5515d68c5733 0         0                 0              1500537611       66892066-fa9c-4789-816e-9fa6d15adb32 1500537610    1500537611              

In a field timelastaccessed datetimestamp lastsuccessfullogintime value not the same, how you see above.

I think this problem connected with another, because i do not connect to Pub with login "axluser" (login/pass error), but on Sub connected is good.

What i can do to write normal value in this table

How i know, command "utils dbreplication rebuild all", just replace next command:

  • utils dbreplication stop

  • utils dbreplication dropadmindb or dropadmindbforce

  • utils dbreplication reset

I use it today.

0 Helpful

Jitender Bhandari
Cisco Employee
Cisco Employee
In response to GlKlondike

‎07-20-2017 03:39 AM

Hi,

I think you have already tried 

utils dbreplication repairtable credentialdynamic all on PUB correct?

JB

0 Helpful

GlKlondike
Level 1
Level 1
In response to Jitender Bhandari

‎07-20-2017 03:48 AM

Yeah, i try it too, same result

Maybe you have idea how fix this situation


To determine if replication is suspect, look for the following:
(1) Number of rows in a table do not match on all nodes.
(2) Non-zero values occur in any of the other output columns for a table

First, a cdr list of the replication status of the servers

SERVER ID STATE STATUS QUEUE CONNECTION CHANGED
-----------------------------------------------------------------------
g_callmanager1_ccm9_1_2_11900_12 2 Active Local 0
g_callmanager2_ccm9_1_2_11900_12 4 Active Connected 0 Jul 20 10:17:22


Jul 20 2017 14:45:29 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------

Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 1

g_callmanager2_ccm9_1_2_11900_12 1126 0 0 1 0


The repair operation completed. Validating the repaired rows ...
Validation failed for the following rows:
data mismatch
key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
WARNING: replicate is not in sync
Jul 20 2017 14:45:35 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end ---------

command failed -- WARNING: replicate is not in sync (178)

Running a cdr check after repair to insure table is in sync

Jul 20 2017 14:45:36 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic start --------

data mismatch

key: pkid:eb4f68a2-744a-401b-a41e-5515d68c5733
------------------------------------------------------------------
Node Rows Extra Missing Mismatch Processed
---------------- --------- --------- --------- --------- ---------
g_callmanager1_ccm9_1_2_11900_12 1126 0 0 0 0
g_callmanager2_ccm9_1_2_11900_12 1126 0 0 1 0

WARNING: replicate is not in sync
Jul 20 2017 14:45:37 ------ Table scan for ccmdbtemplate_g_callmanager1_ccm9_1_2_11900_12_1_452_credentialdynamic end ---------

command failed -- WARNING: replicate is not in sync (178)

end of the file reached

0 Helpful

Jitender Bhandari
Cisco Employee
Cisco Employee
In response to GlKlondike

‎07-20-2017 04:01 AM

Hi,

I have seems some similar issues reposted like this getting fixed with repair commands, but this one seems to have stuck.

Can you provide below

utils create report database

and informix from Both server for last 6 hours.

JB

GlKlondike
Level 1
Level 1
In response to Jitender Bhandari

‎07-20-2017 04:42 AM

Thank you for not leaving me

Attached log, rename .tar to .zip

callmanager1 - pub

callmanager2 - sub

Sorry but i don't know what you mean about informix log for last 6 hours.

callmanager2_db_report_jul_20_17_15_35.zip
callmanager1_db_report_jul_20_17_15_21.zip
0 Helpful

Jitender Bhandari
Cisco Employee
Cisco Employee
In response to GlKlondike

‎07-20-2017 05:22 AM

Can you also collect informix logs from RTMT

http://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200787-How-to-Collect-Traces-for-CUCM-9-x-10-x.html

JB

0 Helpful

Jitender Bhandari
Cisco Employee
Cisco Employee
In response to Jitender Bhandari

‎07-20-2017 05:35 AM

Hi,

Try below and let me know if it helps

1. Run -- utils dbreplication repairtable certificate all
2. After the previous step completes run -- utils dbreplication repairtable certificateservicecertificatemap all
3. After the previous step completes run -- utils dbreplication repairtable certificatehashmap all
4. After the previous step completes run -- utils dbreplication repairtable certificatetrustrolemap all
5. After the previous step completes run -- utils dbreplication repairtable certificateprocessnodemap all
6. After the previous step completes run -- utils dbreplication runtimestate
7. Send the output of the runtimestate

JB

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents