NTP: Common issues and troubleshooting

[Rajat Chauhan]

• Before setting up routers to sync with time sources
• Common Issues
• FAQs
• Miscelleneous
• Useful commands & debugs
• References

Before setting up routers to sync with time sources

• Check what kind of router/software version you are using: 
  • Certain lower end platform supports only SNTP.
  • Check version specific release notes.
  • Use Cisco Feature Navigator

• Check what IOS you are using: 
  • Look for potential bug
  • Check version specific release notes.
  • Not able to confirm? Contact TAC

• Check if you need to poll any public time server: 
  • Choose that is closer to you geographically: NTP Servers

• Check if there is any firewall or access list on the router: 
  • Confirm UDP port 123 is open throughout the network for NTP packets.

Alternate design option

If you have a high end router, use it as a "NTP master" and have it poll to public time servers, and everyone else poll this high end router. In the event of synchronization failure with the public time server, you can rely on the hardware clock of this high end router.

Common Issues

Sync to W32 based time service (Most Windows Implimentations)

W32Time uses Simple Network Time Protocol (SNTP a subnet of NTP) for time synchronization. SNTP and NTP use the same network-packet format. The main difference between SNTP and NTP is that SNTP doesn't provide the error-check and filtering functions that NTP provides. Cisco router and switches use NTP and allow for all error-checking and filtering functions provided by NTP v3.

Known bug: CSCed13703 - NTP will not sync, flags server as insane, invalid

An IOS system may be unable to synchronize to an NTP server despite being able to transmit to and receive packets from the server. This may be seen with a Windows system running the w32time service.

"show ntp associations detail" will show that the server is flagged as "insane, invalid". The "root dispersion" value will be seen as being in excess of 1000 ms, which will cause the IOS NTP implementation to reject the association.

Workaround:

Instead of running the w32time service on the Windows system, use NTP 4.x - refer http://www.eecis.udel.edu/~mills/ntp/html/hints/winnt.html

Note:

The bug is Junked because the behavior is normal and actually cant be "resolved" as such. Another workaround can be use of SNTP since it's not concerned with root dispersion value, it is not recommended however.

Error: Strata too high - too many indirections from sensor to master NTP server

NTP uses the concept of a "stratum" to describe how many NTP "hops" away a machine is from an authoritative time source. That error message indicates that the NTP stratum reported by the NTP server is too high. The stratum is a number between 1 and 15 that indicates how far removed that server is from a precision reference clock. Generally systems that are directly sync'ed to an atomic clock report their stratum as 1. A host that is synced to a stratum 1 NTP server but is also serving as an NTP server for other hosts would report it's stratum as 2 to those hosts. And so on, with each successive layer of servers having a stratum that is 1 higher than its parent.

If the sensor attempts to sync to a server that reports its stratum as 15, then the sensor's stratum would be 16, which is illegal, so the sensor instead rejects the server and displays the "Strata too high" message.

Also if you are using a Linux host as an NTP server and has hard-coded the stratum that it reports rather than let it calculate the stratum automatically. If it is a linux or unix box, the ntp server is configured by the file /etc/ntp.conf, and the "fudge" command is used to hard-code the stratum. The server always reports a stratum value 1 higher than the fudge value to its clients.

FAQs

What are the possible reasons a router can't sync up with public time servers?

• Possible Bug
• Access control lists that don't permit UDP port 123 packets to come through
• Mis-configuration on router
• Public time server down
• NTP server software on NT or UNIX is misconfigured
• Too much traffic on the router or on the way to the server.

When does a NTP master consider itself to be synchronized?

There are several ways a NTP master router considered itself to have synched up.

• Receipt of authoritative NTP time from another source
• Manual setting of the clock
• Manual reading of the calendar chip via "clock calendar-read"
• Setting of the clock at boot time the calendar chip, but only if "clock calendar-valid" is configured (high end router only)

Can we use a GPS clock to the router?

Yes, Cisco routers can connect to a GPS clock through its AUX port. However, Cisco only support GPS clocks from Symmetricom[22] (former Telecom-solutions). However, on a 2600 and 3600, this function is not supported because the AUX port is considered as “floating”, i.e. not fixed.

For router that used support GPS clock, use this command under the line [num]:

ntp refclock telecom-solutions pps cts

Can we adjust the NTP poll interval on a router?

No, the NTP protocol specifies how often the polling interval is required based on how various phase locked loops are performing. One situation is when the router can talk to the NTP server, so it gradually increase the poll interval to reduce network overhead. Another situation is when the router is talking to a bad NTP server (e.g. NTP server with large dispersion), the router will also increase the poll interval. Minimum Poll period specified in the RFC is 64 seconds (NTP.MINPOLL) and the maximum is 1024 seconds (NTP.MAXPOLL).

What stratum number should I use on a NTP master?

Any stratum number that is above 15 is considered as unsynchronized. That's why you will see stratum 16 in those routers which clocks are unsynchronized in "show ntp status". If the master is synchronized with an public NTP server, make sure the stratum number on the NTP master line is one or two higher than the highest stratum on the public servers you are polling.

What’s the difference between NTP broadcast and NTP client/server mode?

These are two separate mode in talking to the server. Actually, in the broadcast mode, the clients listen. In client/server mode, the clients poll the server. You can use NTP broadcast if there is no WAN link is involved because of its speed. If you go across a WAN link use the client/server mode (by pollling) is more preferable. Broadcast mode is designed for a LAN because otherwise many clients may need to poll the server and create a lot of packets on the network. NTP multicast is not yet available in NTPv3, forthcoming in NTPv4.

Why is a Cisco Router a better NTP server than a PC?

• Cisco's IOS NTP code is a full NTP server implementation.
• NTP code in the IOS adjusts (or "disciplines") the system clock every second by the Local Clock routine involving a Phase Lock Loop. A PC doesn't usually have this kind of feature. Most of the Windows implementations are SNTP only. Unix does have a full NTP implementation.
• Win2000 has a clock granularity of 10ms. This is too large for a precise protocol like NTP. If a Win2000 machine is used as a NTP server, it's clock may result in a 10ms jitter on the network but NTP is designed for a jitter of 1ms or less.

What is the address 127.127.7.1?

This is the reference clock address for the Cisco router when the router acts as a NTP master. If the router has not been synchronized with any NTP server, the router will use this address as the Reference ID.

Miscelleneous

NTP broadcast

NTP broadcasts are never forwarded. The "ntp broadcast" command will cause the router to originate NTP broadcasts on the interface on which it is connfigured. The "ntp broadcast client" command will cause the router to listen to NTP broadcasts on the interface on which it is configured.

The normal NTP behavior is to have the client poll the server, and the server to respond to the client. On a multi-access segment, this can consume unnecessary bandwidth. A different way to do this would be to configure the server to broadcast this information to the segment, and have all the clients listen for the broadcast. You have to configure this on all routers involved.

NTP update-calendar

This command synchronizes the sytem clock (software clock) with the calendar clock (a chip on high end routers, including 4500, 4700, 7000, 7200, 7500, with a rechargable backup battery). Though called a “calendar” it stores both time and date. NTP updates only change the system clock, and "ntp calendar-update" will transfer this update to the calendar. The update will be done only if NTP time is synchronized.

On the low end routers (including 2600 Series, 3600 Series, and the 4000), after a reload, its system clock will be initialized to March 1, 1993 because there is no permanent time storage on the routers. The clock on the calendar will run on its own, therefore, if a configuration doesn’t allow the calendar to be synched up with NTP (which affects system clock), it is possible to have very different “show clock” and “show calendar.” Often the system clock is called the “software clock” and the calendar “hardware clock”.

Useful commands & debugs

show clock
show ntp associations [detail]
show ntp status

Before executing any debugs, ensure to turn on the service timestamps:

Service timestamps debug datetime localtime msec

debug ntp packet (NTP packet)

To view actual NTP packet and various parameters

debug ntp validity (Sanity check for incoming NTP packets from a server)

This will only report the failed tests. Validity (sanity) tests are specified in the RFC1305 to test the reply packet received.

debug ntp authentication (NTP authentication debug)

To show what authentication key ID is being used during NTP authentication.

debug ntp events (NTP events)

To show system NTP events like the followings: System Restart, System Fault, Synchronization Change, Peer Stratum Change, Clock Reset, Bad Date/Time, Clock Exception. Also Peer NTP events like these: IP Error, Authentication Failure, Peer Unreachable, Peer Reachable, Peer Clock.

debug ntp select (NTP clock selection debug)

To show which server is being eliminated in the clustering algorithm. If no preferred peer is specified, NTP will only keep 3 (NTP.MINCLOCK) ntp server and eliminate the rest.

Other Useful commands

calendar set:

Exec command. Setting the calendar info manually. On a 7000 or 4500 routers, this date/time info is pick up by the system clock only if you reload the router, or if you use the "clock read-calendar" commands. This command is also affected by the "clock timezone" and "clock summer-time" settings.

clock set:

Exec command. Setting the system clock only and according to the clock timezone and clock summer-time settings. At startup, a low end router will reset its system clock to March 1, 1993. A high end router will read its time from the calendar.

ntp update-calendar:

Global configuration command. NTP usually changes only the system clock, this allows NTP also updates the date/time info on the calendar. The update will be done only if the NTP time is synchronized, otherwise, the calendar will keep its own time and unaffected by the NTP time or system clock. Always use this on the high end routers lest the calendar will be so widely different from the NTP time, that a network disaster that cut out the network from public time source will cause problem to the clocks on the internal network.

clock calendar-valid:

Global configuration command. Declare the calendar information to be valid and synchronized. Recommended to be used on the NTP master. If this is not configured, the high end router that has the calendar will still think its time is unauthoritative even it has the NTP master line.

clock update-calendar:

Exec command. To update the date/time info on the Calendar ship manually. This is a mirror command of "clock read-calendar".

clock read-calendar:

Exec command. To copy the date/time info from the clock chip to the system clock. This is a mirror of the "clock update-calendar" command.

sntp server:

Global configuration command. To configure a Cisco 800, Cisco 1003, Cisco 1004, Cisco 1005, Cisco 1600, Cisco 1720, or Cisco 1750 router to use the Simple Network Time Protocol (SNTP) to request and accept Network Time Protocol (NTP) traffic from a stratum 1 time server

References

Basic System Management Commands

NTP: Best Practices White Paper

Home of The NTP Project

RFC: NTPv3

[heiko.gerstung]

You can download a prepacked binary NTP installer for Windows (all versions since W2K) from http://www.meinbergglobal.com/english/sw/ntp.htm ...

[Ahtesham08]

Very good Article. however in most of the organizations NTP servers are seen as luxury. I faced an NTP server couple of days back.. i have created this blog where i put all my findings and experiences on T-SHOOTING..

http://goo.gl/3ZxX1L have look.

[jmolefe]

What is the difference in .GPS., .GPSe., and .GPSt. on ref clock  from the show ntp associations outputs ?