Ntp assoc detail showing wrong time ?

tommy.schneider · ‎08-01-2016

Dear All,

I have an issue with some new switches (3650) where i can't get NTP to work.

I have 2 NTP servers using GPS and syncing all the infrastructure. they are working fine with all the stuff i have currently online. (ADs, etc...)

I have configured them on the switches, and when i do a "show Ntp Assoc detail" : it shows that my 2 NTPs are insane and invalid because they are 7 hours late than the current time (!)

Ok... so i decided to sync the switches with the Active Directories... and there : the switches says that the NTP servers are 25 minutes late than the current time !

The offset is far too big, so the switchs refuse to sync the time.

I've set the clock manually, but it doesn't change a thing.

I've checked, both GPS NTPs and both ADs : they are perfectly in sync and showing the right time.

I cannot check with others NTP as the infrastructure is fully isolated from Internet.

Someone has an idea about why the switches are seing a wrong time ?

Also, i can't reboot the switches (Production environment).

Maybe another clue : "term mon" is showing stuff with the wrong time...

chrihussey · ‎08-01-2016

Do you have the NTP sources defined as a peer or server in the switch configs? If you are peering that might be the issue and you should change it to defining a NTP server:

ntp server x.x.x.x

If this is not the case, could you supply a sanitized config?

tommy.schneider · ‎08-01-2016

Hi Chrihussey,

The others NTP servers have been set up as "servers" on the switch, not as peers.

No encryption, no IP limitations : switchs access the NTPs directly on the VLAN

Here's the running-conf (Without the encryption and interfaces part)

Current configuration : 20187 bytes
!
! Last configuration change at 20:07:35 CEST Mon Aug 1 2016 by admin
! NVRAM config last updated at 10:14:09 CEST Thu Jul 21 2016 by admin
!
version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
service compress-config
!
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-vrf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
logging userinfo
logging buffered informational
!
no aaa new-model
clock timezone cest 0 0
clock summer-time CEST recurring last Sun Mar 2:00 last Sun Oct 3:00
clock calendar-valid
switch 1 provision ws-c3650-48tq
switch 2 provision ws-c3650-48tq
!
!
!
!
!
!
!
!
qos queue-softmax-multiplier 100
!
!
!
diagnostic bootup level minimal
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
spanning-tree vlan 1-4094 priority 24576
hw-switch switch 1 logging onboard message level 3
hw-switch switch 2 logging onboard message level 3
!
redundancy
 mode sso
!
!
!
class-map match-any non-client-nrt-class
!
policy-map port_child_policy
 class non-client-nrt-class
 bandwidth remaining ratio 10
!
ip default-gateway xxxxx
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
ip route xxxxxxxx
!
interface Vlan533
 ip address 172.18.47.250 255.255.248.0
!
!
logging trap debugging
logging source-interface Vlan541
logging host xxxxxx transport udp port 1113
!
snmp-server community public RO
!
!
line con 0
 login local
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 privilege level 15
 login local
line vty 5 15
 privilege level 15
 login local
!
ntp source Vlan533
ntp server 172.18.41.120 version 3
ntp server 172.18.41.121 version 3
wsma agent exec
 profile httplistener
 profile httpslistener
!
wsma agent config
 profile httplistener
 profile httpslistener
!
wsma agent filesys
 profile httplistener
 profile httpslistener
!
wsma agent notify
 profile httplistener
 profile httpslistener
!
!
wsma profile listener httplistener
 transport http
!
wsma profile listener httpslistener
 transport https
!
ap group default-group
end

Here's what the "show ntp assoc" look (Here, it's targeting both AD,172.18.3.54 is one of the two GPS/NTP)

 address ref clock st when poll reach delay offset disp
 ~172.18.41.120 172.18.3.54 2 51 64 7 6.000 6475.01 17.259
 ~172.18.41.121 172.18.41.120 3 14 64 377 8.000 6570.43 17.471
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

With "detail", command launched at 20:19 CEST

#show ntp assoc detail
172.18.41.120 configured, ipv4, insane, invalid, stratum 2
ref ID 172.18.3.54 , time DB4A0E22.9E4BACA6 (20:06:26.618 CEST Mon Aug 1 2016)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 31.25 msec, root disp 10557.64, reach 37, sync dist 10600.45
delay 5.00 msec, offset 6483.8421 msec, dispersion 17.46, jitter 6.34 msec
precision 2**6, version 3
assoc id 41268, assoc name 172.18.41.120
assoc in packets 10, assoc out packets 10, assoc error packets 0
org time 00000000.00000000 (01:00:00.000 utc Mon Jan 1 1900)
rec time DB4A10D8.06FFE5FE (20:18:00.027 CEST Mon Aug 1 2016)
xmt time DB4A10D8.06FFE5FE (20:18:00.027 CEST Mon Aug 1 2016)
filtdelay = 5.00 6.00 6.00 4.00 3.00 3.00 3.00 3.00
filtoffset = 6483.84 6477.45 6475.01 6475.73 6481.90 6482.00 6486.10 6474.60
filterror = 16.60 17.62 18.62 19.64 20.50 20.53 20.56 20.59
minpoll = 6, maxpoll = 10

172.18.41.121 configured, ipv4, insane, invalid, stratum 3
ref ID 172.18.41.120 , time DB4A0DB1.E47EEE1C (20:04:33.892 CEST Mon Aug 1 2016)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 62.50 msec, root disp 10774.39, reach 377, sync dist 10844.10
delay 6.00 msec, offset 6585.5617 msec, dispersion 17.49, jitter 17.62 msec
precision 2**6, version 3
assoc id 41265, assoc name 172.18.41.121
assoc in packets 209, assoc out packets 209, assoc error packets 0
org time 00000000.00000000 (01:00:00.000 utc Mon Jan 1 1900)
rec time DB4A10FB.20E96D16 (20:18:35.128 CEST Mon Aug 1 2016)
xmt time DB4A10FB.20E96D16 (20:18:35.128 CEST Mon Aug 1 2016)
filtdelay = 6.00 6.00 8.00 6.00 5.00 7.00 5.00 5.00
filtoffset = 6585.56 6576.73 6570.43 6575.66 6564.26 6569.96 6561.06 6563.83
filterror = 16.60 17.59 18.59 19.57 20.53 21.49 22.45 23.44
minpoll = 6, maxpoll = 10

Thanks

chrihussey · ‎08-01-2016

Two things to try:

Remove the "clock calendar-valid" config.

It should only be used if there are not any authoritative clock sources on the network and this is not the case.

Also:

Are you are on UTC time? Try removing the "clock timezone 0 0". Don't think it is needed. From the documentation:

To set the time zone for display purposes, use the clock timezone global configuration command. To set the time to Coordinated Universal Time (UTC), use the no form of this command.

In both cases it may be a good idea to remove and then add the NTP server configs back in just to restart the synchronization process.

tommy.schneider · ‎08-02-2016

Clock calendar-valid has been removed.

Reset of the timezone : done

I also remove all the ntp parameters (the terminal showed me that NTP was "uninitalized" from all the interfaces

Set back the NTP servers.

It is still unable to sync.

I have another "strange" thing : I've manually set the clock to match the one on the NTP (to test the behavior).
The switch has started to sync... and the clock got completely out-of-sync (it went from "correct time" to "30 minutes away" and is now saying that the NTPs are insane

I've tried this 3 times... and i got the same behavior.

tommy.schneider · ‎08-02-2016

It seems to be "a little" better regarding the behavior

But clock is still not syncing, and the two NTPs (here's : ADs) are still considered as insane

#show ntp assoc detail
172.18.41.120 configured, ipv4, insane, invalid, stratum 2
ref ID 172.18.3.54 , time DB4AFA22.F806A1DA (11:53:22.968 CEST Tue Aug 2 2016)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 31.25 msec, root disp 10201.01, reach 377, sync dist 10258.87
delay 6.00 msec, offset 103525.8511 msec, dispersion 17.47, jitter 20.98 msec
precision 2**6, version 4
assoc id 25915, assoc name 172.18.41.120
assoc in packets 117, assoc out packets 117, assoc error packets 0
org time 00000000.00000000 (00:00:00.000 cest Mon Jan 1 1900)
rec time DB4AFB20.B8482B11 (11:57:36.719 CEST Tue Aug 2 2016)
xmt time DB4AFB20.B8482B11 (11:57:36.719 CEST Tue Aug 2 2016)
filtdelay = 6.00 6.00 6.00 6.00 6.00 6.00 7.00 5.00
filtoffset = 103525. 103522. 103515. 103522. 103555. 103549. 103545. 103558.
filterror = 16.60 17.57 18.53 19.49 20.45 21.43 22.43 23.42
minpoll = 6, maxpoll = 10

172.18.41.121 configured, ipv4, insane, invalid, stratum 3
ref ID 172.18.41.120 , time DB4AF9B2.4E335FA8 (11:51:30.305 CEST Tue Aug 2 2016)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 62.50 msec, root disp 10264.05, reach 377, sync dist 10332.86
delay 5.00 msec, offset 103528.9713 msec, dispersion 17.49, jitter 17.07 msec
precision 2**6, version 4
assoc id 25916, assoc name 172.18.41.121
assoc in packets 117, assoc out packets 117, assoc error packets 0
org time 00000000.00000000 (00:00:00.000 cest Mon Jan 1 1900)
rec time DB4AFB33.B8F3E2BB (11:57:55.722 CEST Tue Aug 2 2016)
xmt time DB4AFB33.B8F3E2BB (11:57:55.722 CEST Tue Aug 2 2016)
filtdelay = 5.00 6.00 5.00 6.00 6.00 6.00 5.00 6.00
filtoffset = 103528. 103538. 103537. 103539. 103534. 103549. 103549. 103558.
filterror = 16.60 17.57 18.58 19.58 20.57 21.58 22.55 23.51
minpoll = 6, maxpoll = 10

#show clock
11:56:52.329 CEST Tue Aug 2 2016


#show ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**10
ntp uptime is 1085600 (1/100 of seconds), resolution is 4000
reference time is 00000000.00000000 (00:00:00.000 cest Mon Jan 1 1900)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 0.08 msec, peer dispersion is 0.00 msec
loopfilter state is 'FSET' (Drift set from file), drift is 0.000000000 s/s
system poll interval is 64, never updated.

chrihussey · ‎08-02-2016

Assuming that you NTP and ADs are functioning properly you may be dealing with a bug. The closest I could find for the 3650 platform is CSCuv05123 which is tied to another bug dealing with taking a long time to synch with NTP and drifting values. Not exactly what you are experiencing but close enough. There are other bugs that I came across on other platforms that may apply.

Either open a TAC case or simply try a new code. That may be your best route at this point.

chrihussey · ‎08-02-2016

One other thing, with the configuration adjustments you have made try directing the switch to the NTP servers and see what that does.

mkwaramb · ‎03-08-2018

try checking ntp version for server and client

Richard Burts · ‎03-09-2018

The original post was going back and forth between attempting to sync with an NTP server and with AD. Windows typically uses a simplified time service and IOS will typically not sync its NTP with a Windows device. It is not clear what kind of NTP server was in the original discussion and what might have impacted syncing with it.

HTH

Rick

HTH

Rick