Best practices for connecting to NTP servers
1. How many NTP servers should I use?
There is an old saying that goes: "A man with one watch knows what time it is. A man with two watches is never sure."
If you have only one clock, that is the only one you can trust. If you have two and they start to show different times, it is difficult to know which one has gone wrong. To guarantee accuracy, you need at least three clocks. If two of them show the same time, you can be relatively sure when the third clock has gone wrong. The more clocks you have showing the same time, the more sure you can be that they are right and that any clock showing a different time is incorrect.
The same principle holds true for getting time over the Network Time Protocol (NTP). If your client allows it, you should connect to multiple NTP servers. In the case that your client only allows you to specify a single NTP server, use: ntp.netnod.se (or nts.netnod.se if your client supports Network Time Security).
2. How do I connect to Netnod’s NTP servers?
There are approximately 3,000 publicly available NTP servers on the Internet today. You obviously can’t connect to them all, but connecting to up to a few dozen is perfectly reasonable. You should pick those that are close to your network and that are from trusted time providers such as Netnod.
Netnod currently has 12 different NTP servers that are in secure bunkers spread in six different locations throughout Sweden (Gothenburg, Luleå, Malmö, Stockholm*2 and Sundsvall). Each bunker has two atomic clocks and two NTP servers to ensure redundancy. If your client is in the Nordics, connecting to all of Netnod’s NTP servers gives you an effective way to ensure accurate time over NTP.
For those using chrony or ntpsec (two of the most common NTP clients on Linux systems today), you can enter the following in your chrony.conf or ntpd.conf:
pool gbg1.ntp.netnod.se iburst pool gbg2.ntp.netnod.se iburst pool lul1.ntp.netnod.se iburst pool lul2.ntp.netnod.se iburst pool mmo1.ntp.netnod.se iburst pool mmo2.ntp.netnod.se iburst pool sth1.ntp.netnod.se iburst pool sth2.ntp.netnod.se iburst pool sth3.ntp.netnod.se iburst pool sth4.ntp.netnod.se iburst pool svl1.ntp.netnod.se iburst pool svl2.ntp.netnod.se iburst
"iburst" tells chrony / ntpsec to send multiple queries when starting up to enable faster synchronisation with the server after a reboot.
3. How do I connect to other NTP servers?
In addition to Netnod's NTP servers, it may be helpful to use other freely available NTP servers such as:
pool time.nist.gov iburst pool ptbtime1.ptb.de iburst pool ptbtime2.ptb.de iburst pool ptbtime3.ptb.de iburst
NIST and PTB are the state measuring institutes in the USA and Germany, respectively, which also have publicly available NTP servers.
Google also has redundant NTP servers (time.google.com), but they do not really follow the standards when it comes to managing leap seconds. Most other NTP servers handle leap seconds according to the current IETF Best Current Practises, but Google smear the leap second over an entire day, which means they will run almost one second wrong when the leap second falls. As long as you use several other NTP servers, your NTP client will discover that Google is making a mistake and ignore them during that day. But if you only use Google's servers, or if you use about an equal number of servers that correctly leap smear and that do not, you are back to the problem of having two clocks showing different times and not knowing which to trust. Google's NTP servers are excellent if you only need time that is correct to a few seconds. But if you want to follow UTC more accurately than that, you should not use them.
You can also consider using pool.ntp.org; a group of volunteers who run their own publicly available NTP servers:
pool se.pool.ntp.org iburst
If you wish to use their servers, it is best to include your country code in the name of pool.ntp.org. So if you live in Sweden, you should use: se.pool.ntp.org. If you are in Germany, it would be: de.pool.ntp.org.
An issue with pool.ntp.org is that most of their servers are run by volunteers who, mostly, do not have good local clocks. They get their time from other NTP servers or, at best, a GPS receiver. Netnod's NTP servers are connected directly to atomic clocks that will deliver very accurate time for several months even without contact with other NTP servers or if the GPS service is, for some reason, interrupted.
Another issue with pool.ntp.org is that many of the volunteers most likely get their time from providers like Google that use leap smearing. It is impossible to know in advance which NTP servers in the pool will use leap smearing.
4. Why should I connect to multiple NTP servers?
The more NTP servers you are connected to, the more sure you can be that you will end up with the right time. Things like noise in measurements, delays, changes and network asymmetries tend to level out when you connect to many NTP servers, at least as long as the servers are reasonably independent of each other.
Ideally, the servers should also differ in what hardware they run on and what software they use. If a certain type of server software has a bug that causes everyone who runs it to simultaneously start responding with the wrong time, or if a certain type of hardware has a bug that causes a clock skip at the same time, then it is good to have other servers with different software or hardware that still provides the right time.
When choosing which NTP servers to use, you need to balance a good spread of different providers and/or implementations with the goal of connecting to servers that are as close to your network as possible. For example, if you have your client in Sweden, it would be better to connect to NTP servers located in Sweden or neighbouring countries.
Ideally, you should connect to servers located in different places and on different networks. If the network from a client to the USA experiences an asymmetric delay, the NTP client will think that all those clocks in the USA have started to go wrong with the same difference as the asymmetric delay. The negative effect will be smaller if your NTP client is also getting time from elsewhere.
5. Why should I care about asymmetry?
To explain how asymmetry can impair time accuracy, take a look at Figure 1 below.
Figure 1: symmetric and asymmetric network delays
In the example above, T is the local clock on the client. The client sends out an NTP request at that time. The server will receive the request and send a response with the time from the server's clock. The client receives the response 40 milliseconds after it sent out the request.
The only thing the client can be certain of is that the timestamp in the response from the server was made somewhere in that 40 ms window. But it does not know exactly when. For some applications, however, 40 ms is a fairly long time.
The client can make the assumption that the delays to and from the server are symmetric: simply divide the total delays by 2 and assume that that is the exact local time to which the server timestamp corresponds. In the symmetric case, 40ms/2=20 ms. So in this case you can be confident that the time received by your client is correct within plus/minus 20 ms and if the delays are symmetric the estimate will be spot on.
This is only an assumption though, and will not hold for an asymmetric network delay. As an example, a mobile phone usually has a faster downlink than the uplink. On such a network, it might take 30 ms for the request to reach the server but only 10 ms for the reply to get back to the client. If the client assumes that the delays are symmetric (but the truth is different, as in the asymmetric diagram above), this means that the client's estimate will be off. In the case of our example, assuming that the delays are symmetric when they are not would mean the estimate is 30-20=10ms off.
There is no way the client can tell the difference and know if the delays are symmetric or asymmetric. The client only knows about T, T+40ms and the timestamp it got from the server. As far as the client can tell, the two cases are identical. In the absence of outside information about the delays (e.g. measurement using a GPS receiver), assuming that the delays are symmetric will usually give a decent estimate, even when the delays are asymmetric. Of course, the estimate would be even better if the delays match the assumption and actually are symmetric.
A total delay of 40 ms is fairly typical for a decent home network to an NTP server with good network connectivity such as Netnod's. With fibre to the home, it can be below 10 ms. But if the NTP server is far away or the network is congested, the delay can easily reach a few hundred milliseconds. If the NTP client talks to a nearby NTP server where the total delays are only 10ms, the client will have a much smaller window of uncertainty than if the total delays are 100ms.
6. What if I can only connect to one NTP server?
Unfortunately, there are a lot of devices, such as home routers, where you can only enter the name of a single NTP server. In that case, you should choose a provider that has redundant infrastructure to ensure that you will get an answer even if there are problems somewhere.
Netnod uses a technology called "anycast" that provides a response from the Netnod servers as listed above. To use Netnod's anycast address, you should use "ntp.netnod.se". The IPv4 or IPV6 address you receive in response to that name will be routed to the nearest Netnod NTP server. For example, if you usually connect to the NTP server in Gothenburg called gbg1 and that becomes unavailable, gbg2 will take over. If the network from the computer to the bunker in Gothenburg is completely broken, one of the other Netnod NTP servers in a different location will take over. The system is designed to reroute queries to the nearest available server (and is something Netnod has perfected running DNS anycast for one of the world’s 13 root name servers).
7. How can I run my own NTP server?
If you have your own network, not all of your computers should connect to an external NTP server. If all computers on the local network run their own NTP client talking to a dozen NTP servers on the Internet, it is not certain that they will talk to the same ones. This means that the computers will often be out of sync with each other.
It is much better to have one NTP client on a local network computer that is connected to a dozen NTP servers around the world. Then you run an NTP server on that computer and let the NTP clients on the rest of the devices on your network get time from your NTP server. That way, all the machines on the local network will have the same time. If you have a large local network, it can be good for redundancy to have several local NTP servers. But you only need two or three, not a dozen.
If you want to go deeper into the best current practices for NTP, you can see the latest IETF draft here.
8. Why take time from Netnod?
Netnod has been recognised for world-leading time services by the Royal Swedish Academy of Engineering Sciences (IVA). Praising Netnod as a “star watchmaker” (“stjärnurmakare”), the President of IVA described Netnod’s time service as one of the best in the world.
Netnod’s time service, funded by the Swedish Post and Telecom Authority (PTS), uses a distributed timescale on multiple, autonomous sites throughout Sweden to provide a free time service available over IPv4 or IPv6. The time is traceable to the official Swedish time UTC(SP).
Each site has full redundancy: multiple servers, caesium clocks, and FPGA boards provide an extremely fast hardware implementation of NTP. This means you speak NTP directly to the FPGA chip. As there is no software involved, you get the most accurate time possible.
9. Further reading
If you want to set up an NTS client to take time from Netnod’s NTS servers, follow our detailed HOW TO here. You can also read our white paper on NTS available here.
If you want to know more detail about how Netnod operates it’s time service, read our earlier blogpost here.