CUC Randomly Losing NTP Synchronization

Quintin.Mayo · ‎06-19-2018

I am working on a issue for our collaboration cluster which consists of CUCM Pub & Sub, and CUC Pub & Sub. We have notice the CUC Pub randomly loses synchronization with our local NTP server. No other devices in the cluster are experiencing the issue. I can't seem to find the root cause why this is happening, can anyone inform on how to troubleshoot this type of issue? All NTP reachability, clock drift,and stratum tests completes successfully. Any direction in this matter would be greatly appreciated.

Thanks,

Ratheesh Kumar · ‎06-19-2018

Hi there

Can you find for any errors/alert in RTMT ? Please check for CPU spikes. Whats the version you are running on ?

Can you run the diagnose module commands and see its getting passed

utils diagnose module ntp_reachability
utils diagnose module ntp_clock_drift
utils diagnose module ntp_stratum

Hope this helps!

Cheers
Rath!

***Please rate helpful posts***

Quintin.Mayo · ‎06-19-2018

Hi,
Thanks for the quick response. Below is the output for the recommended commands.

Starting diagnostic test(s)
===========================

test - ntp_reachability : 5 second(s) left
test - ntp_reachability : 0 second(s) left

test - ntp_reachability : Passed

Diagnostics Completed
The final output will be in Log file: platform/log/diag5.log
Please use 'file view activelog platform/log/diag5.log' command to see the output
--------------------------------------------------
admin:utils diagnose module ntp_clock_drift

Log file: platform/log/diag1.log

Starting diagnostic test(s)
===========================

test - ntp_clock_drift : 5 second(s) left
test - ntp_clock_drift : 0 second(s) left

test - ntp_clock_drift : Passed

Diagnostics Completed

The final output will be in Log file: platform/log/diag1.log

Please use 'file view activelog platform/log/diag1.log' command to see the output
-----------------------------------------------------
admin:utils diagnose module ntp_stratum

Log file: platform/log/diag2.log

Starting diagnostic test(s)
===========================

test - ntp_stratum : 5 second(s) left
test - ntp_stratum : 0 second(s) left

test - ntp_stratum : Passed

Diagnostics Completed
-----------------------------------------------------
admin:show process load cpu
top - 11:11:25 up 217 days, 12:38, 1 user, load average: 0.55, 0.49, 0.44
Tasks: 262 total, 1 running, 261 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.9%us, 8.3%sy, 1.9%ni, 75.6%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 5994288k total, 5861108k used, 133180k free, 188892k buffers
Swap: 2064284k total, 1281080k used, 783204k free, 2402956k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2588 root 20 0 121m 9052 6560 S 1.9 0.2 61:35.98 CiscoSyslogSubA
12039 Admin 30 10 432m 64m 14m S 1.9 1.1 0:02.25 java
14436 openspee 20 0 141m 5224 1596 S 1.9 0.1 1206:48 NSSserver
17074 Admin 30 10 15168 1336 900 R 1.9 0.0 0:00.01 top
20849 cumbxsyn 30 10 453m 179m 7256 S 1.9 3.1 7717:36 CuMbxSync
24408 jetty 20 0 1194m 97m 2632 S 1.9 1.7 707:52.66 jetty
25610 ccmservi 20 0 697m 93m 5364 S 1.9 1.6 893:55.40 amc
28887 informix 20 0 721m 374m 374m S 1.9 6.4 8974:44 unityoninit
1 root 20 0 19492 1628 1200 S 0.0 0.0 19:00.49 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
----------------------------------------

admin:file view activelog platform/log/diag2.log

06-19-2018 11:10:03 Diagnostics Version: 1.0.0
06-19-2018 11:10:03 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]
06-19-2018 11:10:03 Hardware Model: VMware
06-19-2018 11:10:03 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]
06-19-2018 11:10:03 Version: 11.0.1
06-19-2018 11:10:03 disk_space: Is valid module: True
06-19-2018 11:10:03 disk_files: Is valid module: True
06-19-2018 11:10:03 service_manager: Is valid module: True
06-19-2018 11:10:03 tomcat: Is valid module: True
06-19-2018 11:10:03 tomcat_deadlocks: Is valid module: True
06-19-2018 11:10:03 tomcat_keystore: Is valid module: True
06-19-2018 11:10:03 tomcat_connectors: Is valid module: True
06-19-2018 11:10:03 tomcat_threads: Is valid module: True
06-19-2018 11:10:03 tomcat_memory: Is valid module: True
06-19-2018 11:10:03 tomcat_sessions: Is valid module: True
06-19-2018 11:10:03 tomcat_heapdump: Is valid module: True
06-19-2018 11:10:03 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml
06-19-2018 11:10:03 validate_network: val: true
06-19-2018 11:10:03 validate_network: Is valid module: True
06-19-2018 11:10:03 validate_network_adv: Is valid module: False

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 43) :
06-19-2018 11:10:03 raid: getting cpu speed [/usr/local/bin/base_scripts/sd_hwdetect CPUSpeed]
06-19-2018 11:10:03 raid: CPU Speed: 2400
06-19-2018 11:10:03 raid: model = VMware
06-19-2018 11:10:03 raid: Is valid module: True
06-19-2018 11:10:03 system_info: Is valid module: True
06-19-2018 11:10:03 ntp_reachability: Is valid module: True
06-19-2018 11:10:03 ntp_clock_drift: Is valid module: True
06-19-2018 11:10:03 ntp_stratum: Is valid module: True
06-19-2018 11:10:03 sdl_fragmentation: Is valid module: True
06-19-2018 11:10:03 sdi_fragmentation: Is valid module: True
06-19-2018 11:10:03 ipv6_networking: IPV6INIT=no
06-19-2018 11:10:03 ipv6_networking: IPv6 initialized: no
06-19-2018 11:10:03 ipv6_networking: False
06-19-2018 11:10:03 ipv6_networking: Is valid module: False
06-19-2018 11:10:03
06-19-2018 11:10:03 --> executing test [ntp_stratum], fix: fixauto, stop on error: False
06-19-2018 11:10:03
06-19-2018 11:10:03 ntp_stratum: ------------------
06-19-2018 11:10:03 ntp_stratum: Analyzing the stratum level of the Syncing NTP Server...
06-19-2018 11:10:03 ntp_stratum: Executing command /usr/local/bin/base_scripts/ntp_check_stratum.sh -D -f /tmp/tmpz6ZdDX.ntp

options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 43) :
06-19-2018 11:10:03 ntp_stratum: return code is: 0
06-19-2018 11:10:03 ntp_stratum: result: 0, message: Passed

end of the file reached

Ratheesh Kumar · ‎06-19-2018

Thanks. Diag tests seems to be good. Whats the version you are running and also did you see any errors in Syslogs ?
What kind of NTP are you using ? Is this affecting only CUC, hows the CUCM going ?

Hope this Helps
Cheers
Rath!

***Please rate helpful posts***

Quintin.Mayo · ‎06-19-2018

Hi,
The issue is only with the one CUC device which is the Pub. All other devices aren't experiencing any issues. I'm currently checking the syslogs.

admin:show status

Host Name : naco-uc-uc1
Date : Tue Jun 19, 2018 12:01:32
Time Zone : Eastern Daylight Time (America/Detroit)
Locale : en_US.UTF-8
Product Ver : 11.0.1.21900-11
Unified OS Version : 6.0.0.0-2

Quintin.Mayo · ‎06-19-2018

Hi,
Sorry, I didn't post the ntp server. The ASA is used as the NTP server.

Ratheesh Kumar · ‎06-19-2018

Hi there

I could find any bugs with this version of CUC. Except this

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd70334

Can you try these or just point NTP to CUCM and see that helps you.

1. utils network ping x.x.x.x (ip address of NTP server)

2. utils ntp reset

3. utils ntp status

Hope this helps!

Cheers
Rath!

***Please rate helpful posts***

Quintin.Mayo · ‎06-20-2018

Hi,
I checked the system logs on the call manager and unity nodes both are indicating connectivity issue with the NTP server which is an ASA, I believe this is the issue. Can you confirm that the ASA shouldn't be used as an NTP server for the collaboration environment? I found some old posts informing the ASA can only be a client and not a master, and most are using a switch or router for their NTP implementation. Is there any documentation on type of devices that should be used?

Ratheesh Kumar · ‎06-20-2018

These are the Cisco recommendation for NTP

Cisco highly recommends configuring the publisher to point to a Stratum-1, Stratum-2, or Stratum-3 NTP server to ensure that the cluster time is synchronized with an external time source.

Cisco recommends synchronizing Unified CM with a Cisco IOS or Linux-based NTP server. Using Windows Time Services as an NTP server is not recommended or supported because Windows Time Services often use Simple Network Time Protocol (SNTP), and Linux-based Unified CM cannot successfully synchronize with SNTP.

The external NTP server specified for the primary node should be NTP v4 (version 4) to avoid potential compatibility, accuracy, and network jitter problems. External NTP servers must be NTP v4 if IPv6 addressing is used.

Cisco IOS and CatOS NTP Time Synchronization

Time synchronization is also important for other devices within the network. Cisco IOS routers and Catalyst switches should be configured to synchronize their time with the rest of the network devices via NTP. This is critical for ensuring that debug, syslog, and console log messages are time-stamped appropriately. Troubleshooting telephony network issues is simplified when a clear timeline can be drawn for events that occur on devices throughout the network.

Hope this helps
Cheers
Rath!

***Please rate helpful posts***