NTP issues

[Cory Dryden]

Hi guys,

I started a discussion last week about my two routers that are not synching time with a server on the internet. They both go through a firewall but are open for ntp.

After a day of screwing around last week they magically resynched and I thought all was well, however I have come back in this morning and they are down again.

I restarted ntp by removing the ntp server statements and readding. I saw on the firewall a UDP connection closing after 66hrs. But when I restart by typing statements in again, no synching occurs even though firewall states it builds an outbound connection.

It sounds like it might not be closing the connection, so possibly timed out???

I have used NTP query tool to confirm the server status and it comes back as a good connection.

Any ideas?

[smitesh kharecha]

Hi Cory,

Any NAT on firewall ?

Try NTP with update source interafce...

btw can you share your NTP configs please.

Regards,

Smitesh

[Cory Dryden]

Hi mate,

NAT is on the firewall, seems to be working fine.

I have not heard of the ntp update source int so will look into it.

Here is the config on both routers

ntp clock-period 17180056

ntp update-calendar

ntp server 203.161.12.165 prefer

ntp server 194.35.252.7

[Richard Burts]

Cory

The ntp source interface has been supported in IOS for a long time. In instances where your router has multiple paths to the NTP server it may be helpful to specify the source interface. And perhaps it may help to specify the source address when going to an Internet NTP server (depending on how the access rules in your firewall are written).

Is it possible that your router(s) rebooted? On routers that do not have the internal clock chip when they reboot they start their clock from an arbitrary time. And if a router attempts to sync with an NTP server but its clock is WAY off then it can take a long time to achieve sync.

I find that sometimes the command show ntp association detail provides helpful information. Perhaps you could post that output for us? (and maybe the output of show version as well)

HTH

Rick

[Cory Dryden]

Ok here is the sh ver on the routers

#1

Cisco IOS Software, 2800 Software (C2800NM-ADVSECURITYK9-M), Version 12.4(6)T, RELEASE SOFTWARE (fc1)

#2

Cisco IOS Software, 2801 Software (C2801-ADVSECURITYK9-M), Version 12.4(6)T, RELEASE SOFTWARE (fc1)

The sh ntp asso is the same on both

      address         ref clock     st  when  poll reach  delay  offset    disp

~203.161.12.165   0.0.0.0          16     -    64    0     0.0    0.00  16000.

~194.35.252.7     0.0.0.0          16     -    64    0     0.0    0.00  16000.

* master (synced), # master (unsynced), + selected, - candidate, ~ configured

Thank you for your time guys.

[Cory Dryden]

Oh and also, no rebooting has occured. Its just so strange when it was working but then all of a sudden stopped.

[Richard Burts]

Cory

I was hoping to see several things in the output of show version. You gave us the details of what code it is running but not some of the other things I was hoping to see. And you gave us the output of show ntp association but not of the show ntp association detail that I asked for.

If the routers are 2800 then resetting the clock to an arbitrary date on reboot should not be an issue. But obviously something in interfering with NTP.

HTH

Rick

[Cory Dryden]

203.161.12.165 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (10:00:00.000 QLD Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (10:00:00.000 QLD Mon Jan 1 1900)
rcv time 00000000.00000000 (10:00:00.000 QLD Mon Jan 1 1900)
xmt time D2915FB8.04C4BD3C (15:23:04.018 QLD Tue Dec 13 2011)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

127.686 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (10:00:00.000 QLD Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (10:00:00.000 QLD Mon Jan 1 1900)
rcv time 00000000.00000000 (10:00:00.000 QLD Mon Jan 1 1900)
xmt time D2915FB8.050EEFB7 (15:23:04.019 QLD Tue Dec 13 2011)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

Cisco IOS Software, 2800 Software (C2800NM-ADVSECURITYK9-M), Version 12.4(6)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Wed 22-Feb-06 22:54 by ccai

ROM: System Bootstrap, Version 12.4(1r) [hqluong 1r], RELEASE SOFTWARE (fc1)

MONHUB1 uptime is 1 week, 29 minutes
System returned to ROM by bus error at PC 0x40CFEDBC, address 0x42E1BC at 14:53:51 QLD Tue Dec 6 2011
System restarted at 14:54:41 QLD Tue Dec 6 2011
System image file is "flash:c2800nm-advsecurityk9-mz.124-6.T.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 2811 (revision 53.51) with 249856K/12288K bytes of memory.
Processor board ID F^&^&^&*

2 FastEthernet interfaces
8 terminal lines
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity enabled.
239K bytes of non-volatile configuration memory.
62720K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

Here mate, sorry.

[Richard Burts]

Cory

Thanks. You have now given me exactly what I asked for. And I am sorry to say that it does not shine as much light on the issue as I had hoped that it would. The ntp ref ID 0.0.0.0 makes it look like the router has not communicated with the NTP server. And I find that curious since you say that it had been i sync with the server.

Is it possible that access to the Internet (especially to the NTP servers in the Internet) has been interrupted?

HTH

Rick

[Cory Dryden]

The internet is working fine and I can see that the routing is working fine as the firewall logs are denying a ping to the server when I try from the router.

Its almost like when it sends an ntp packet it does not know where to go?

[Cory Dryden]

When I look at the firewall logs after removing the ntp server statements and readding them, I see no outbound UDP connections being formed.

[Richard Burts]

Cory

That is an interesting observation.

What do you get from the command show ip route 203.161.12.165

What do you get from a traceroute to that address?

HTH

Rick

[Cory Dryden]

Routing entry for 203.161.12.165/32

  Known via "static", distance 1, metric 0

  Routing Descriptor Blocks:

  * 172.16.101.235

      Route metric is 0, traffic share count is 1

Traceroute does nothing, even though it hits a layer 3 switch and the firewall.

MONHUB1#tracerout 203.161.12.165

Type escape sequence to abort.
Tracing the route to ntp.tourism.wa.gov.au (203.161.12.165)

  1  *  *  *
  2  *  *  *
  3  *  *  *
  4  *  *  *
  5  *  *  *
  6  *  *  *
  7  *  *  *
  8  *  *  *
  9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *

[Richard Burts]

Cory

The output of traceroute is very interesting and suggests that there may be a problem with basic IP connectivity. The static route indicates that the next hop is 172.16.101.235. Can you verify that this is reachable?

It also occurs to me to ask whether there have been changes recently, on this router or on other devices on its path toward the NTP server?

HTH

Rick

[Kishore Chennupati]

Hey mate,

You said they were working before. Did you reboot your routers after that by any chance?

1. Also AFAIK in order to have NTP setup and working you need to have a working clock configuration on your router as wel. I mean configure your router properly to the time zone etc. Please check that as well "sh clock" should show if the time is out of whack.!!

can you paste " sh run | i clock" here?

2.  If the above thing is good then may the IOS needs to be changed. I for some reason dont like "T" trains If ntp is workling intermittently I would check for any IOS Bugs as well. how about a "debug ntp validity" on the box?

Edit:

. Just saw your logs, your box is in queensland and the ntp is in wa  not much far . still in Oz. Im from Melb btw

   how about this. do you have a router elsewhere? . Maybe configure  NTP on that and see if it works for testing.

HTH

Regards

Kishore