DB Replication Issue - CUCMv8.5 - UCS Virtual Environment

svictoryshanmugam · ‎04-17-2012

Dear Experts,

I work for a Global University IP Telephony(CUCM v8.5) Team and the site has one publisher and two subscribers hosted on a Virtual Environment(Vmware-UCS) in a single Server. An unplanned power outage has caused the publisher subscriber DB replication issue. I did the following but in vain.

1. "utils dbreplication stop" on both the subsribers

2. "utils dbreplication stop" on the publisher

3. "utils dbreplication reset all" on the publisher

And it did not help resolve the problem. After that I also tried doing the "utils dbreplication repair" on the publisher with no change in the db replication.

I do see some posts where the mention about rebooting the publisher and subscriber. If that be the case please do mention the difference between rebooting and restarting via CLI.

Please help me and I sincerely thanks for all your help.

THANKS, Saravanan

svictoryshanmugam · ‎04-17-2012

I am also finding that any incorrect NTP settings would cause DB replication. I did see some NTP warnings/critical in the RTMT until few days ago but essentially they are gone in the last two days but the db replication problem still persists. When I checked NTP Server settings I did find the publisher IP is mentioned as the NTP server. I just wanted to add this additional information. Thank you !

Joseph Martini · ‎04-17-2012

What do you replication states currently show as "utils dbreplication runtimestate"? If the UCS lost power and shutdown you might want to check for file system corruption by running the recovery disk on each server and checking for errors. I'm not sure if the UCS has a write-cache battery to help prevent file system issues in the event of a power loss. There's also a documentation bug to note that a rebuild should be done if you find errors when running the recovery disk CSCth53322.

svictoryshanmugam · ‎04-18-2012

Dear Joe, Please find the attached CM database Report that from the Publisher.

Sivakumar Shanmugam · ‎04-18-2012

Hello,

Could see one server 10.112.5.11 is down.That's the reason you end up seeing bad replication across the cluster.

Is it down due to maintanence ?

Thanks

Siva

svictoryshanmugam · ‎04-18-2012

Hi Siva,

I am able to ping the subscriber 10.112.5.11 from the pub 10.112.5.10 and vice versa. Pls find the db replication status from both the pub and sub1.

DB Replication Status from the Publisher

==============================

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

Cluster Replication State: Replication status command started at: 2012-04-17-17-19
Replication status command COMPLETED 530 tables checked out of 530
No Errors or Mismatches found.

Use 'file view activelog cm/trace/dbl/sdi/ReplicationStatus.2012_04_17_17_19_11.out' to see the details

DB Version: ccm8_5_1_10000_26
Number of replicated tables: 530

Cluster Detailed View from PUB (6 Servers):

                                PING            REPLICATION     REPL.   DBver& REPL.   REPLICATION SETUP
SERVER-NAME     IP ADDRESS      (msec) RPC?    STATUS          QUEUE   TABLES LOOP?   (RTMT) & details
-----------     ------------    ------ ----    -----------     -----   ------- -----   -----------------
cm-pub 10.112.5.10     0.025   Yes     Connected       0       match   Yes     (3) PUB Setup Completed
cm-sub2 10.112.6.10     0.273   Yes     Connected       0       match   Yes     (3) Setup Completed
cm-sub1 10.112.5.11     0.147   Yes     Off-Line        N/A     DB-diff No      (3) Not Setup
cm-sub3 10.112.5.12     0.118   Yes     Connected       0       match   Yes     (3) Setup Completed
cm-tftp1        10.112.6.15     0.323   Yes     Connected       0       match   Yes     (3) Setup Completed
cm-sub4 10.112.6.11     0.249   Yes     Connected       0       match   Yes     (3) Setup Completed

DB Replication Status from the Subscriber

==============================

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

Cluster Replication State: Only available on the PUB

DB Version: ccm8_0_2_40000_1
Number of replicated tables: 519

Cluster Detailed View from SUB (6 Servers):

                                PING            REPLICATION     REPL.   DBver& REPL.   REPLICATION SETUP
SERVER-NAME     IP ADDRESS      (msec) RPC?    STATUS          QUEUE   TABLES LOOP?   (RTMT)
-----------     ------------    ------ ----    -----------     -----   ------- -----   -----------------
cm-pub 10.112.5.10     4.31    Yes     Active-Failed   410358          DB-diff N/A     (3)
cm-sub2 10.112.6.10     0.274   Yes     Active-Failed   403818          DB-diff N/A     (3)
cm-sub1 10.112.5.11     0.025   Yes     Connected       0       match   N/A     (3)
cm-sub3 10.112.5.12     0.145   Yes     Active-Failed   403818          DB-diff N/A     (3)
cm-sub4 10.112.6.11     0.308   Yes     Active-Failed   403690          DB-diff N/A     (3)
cm-tftp1        10.112.6.15     0.289   Yes     Active-Failed   403818          DB-diff N/A     (3)

Thanks,

svictoryshanmugam · ‎04-18-2012

Also the output of the command "utils dbreplication status" from the Publisher:

admin:file view activelog cm/trace/dbl/sdi/ReplicationStatus.2012_04_18_10_59_42.out

SERVER ID STATE STATUS QUEUE CONNECTION CHANGED

-----------------------------------------------------------------------

g_cm_pub_ccm8_5_1_10000_26 2 Active Local 0

g_cm_sub2_ccm8_5_1_10000_26 3 Active Connected 0 Apr 17 16:22:50

g_cm_sub3_ccm8_5_1_10000_26 5 Active Connected 0 Apr 17 16:22:18

g_cm_sub4_ccm8_5_1_10000_26 7 Active Connected 0 Apr 17 16:24:57

g_cm_tftp1_ccm8_5_1_10000_26 6 Active Connected 0 Apr 17 16:21:57

end of the file reached

I am not able to subscriber 1 at all... You can see sub1 is missing.

Thank you all.

Sivakumar Shanmugam · ‎04-18-2012

Hello,

From your logs, could see

cm-sub1 10.112.5.11 0.147 Yes Off-Line N/A DB-diff No (3) Not Setup

Here we have to see many factors liks rhost,sql,connectivity,hosts,etc.May i request you to open a TAC case to troubleshoot the issue.

Thanks

Siva