Telstra outage: The night a network decided the year was 2006

The opposite is actually the case. If we do not have a common understanding of what “now” is, a lot of things we take for granted will stop working.

This summer, Australia realized this the hard way. Let me take this opportunity to give an example of why time matters to a modern society, what happened in this particular case, and what our key takeaways are.

On 8 July 2026, a large part of the mobile network run by Australia's largest cell phone operator Telstra stopped working with voice calls not getting through or text messages that didn't arrive. There were even calls to Australia's emergency number that did not get through. But the outage affected systems even further apart, such as trains, payment terminals, ticketing systems and EV chargers being disrupted.

No networks were attacked. Nobody accidentally cut the fiber. All systems had electrical power. The culprit, you may ask? A single GPS receiver in a single chassis in Melbourne coming back from scheduled maintenance believing the year was 2006, and the rest of the network was persuaded to believe it.

Telstra commissioned an independent review from the company Technology Audit Partners (TAP) and the report is a very interesting read, because the same type of failure could appear in many critical services, including some we depend on to keep people alive.

In order for many of these systems to function, time being correct, or at least the same everywhere, is crucial. And as everyone in the business knows, “correct” is a relative term. There is no exact time, only time held within a certain margin of a reference. How wide that margin may be depends entirely on what you are doing or in which business you operate in. 

Running a mobile network, like Telstra, is about as time-dependent as a business gets. 

Modern mobile communication explains why time matters

Modern cell phone protocols will not work without precision time. Mobile networks separate uplink data from downlink by either FDD (Frequency Division Duplex) or TDD (Time Division Duplex). 

FDD gives each direction its own slice of spectrum, so both can run continuously without colliding. 

TDD instead uses an entire single block of spectrum for both directions, alternating between transmitting and receiving in very short intervals. 

Since far more data usually flows down than up, FDD's fixed ratios leave much of the uplink spectrum idle, while TDD can shift the ratio to match the actual traffic. That is why most modern 5G spectrum, including Sweden's main 5G band at 3.5 GHz, is TDD. 

It is also why TDD depends on accurate time: every cell on the same frequency has to switch direction in step with every other. A cell that lets its clock drift will transmit data into its neighbour's receive window, with the result that the network will start jamming itself.

Rather than allocating spectrum on keeping the two directions apart, the industry chose to rely on time accuracy, and accepted a hard dependency on every cell agreeing about when "now" is. Thus, being dependent on time is a design choice. 

However, given how much we all depend on the systems being able to agree on “now”, it is somewhat puzzling that time is not given as much consideration as it deserves. And that is a lesson that is very clear from the published report.

So, what really happened?

Architecture of time distribution.

To begin with, it is vital to understand the architecture of time distribution.

Time distribution protocols all build hierarchies; Network Time Protocol (NTP), which Telstra has deployed according to the report, expresses its hierarchy in strata.

• Stratum 0 is the reference itself, for instance a GPS receiver, or Netnod’s atomic clocks.
• Stratum 1 is a machine synchronised directly to a stratum 0 reference, for example the NTP servers that Netnod provides.
• Stratum 2 synchronises from a stratum 1 server, stratum 3 from a stratum 2, and so on.

In Telstra's case, that hierarchy had a specific shape, at least to begin with. This design from 2010 had at the top stratum 1 sources at Australia's National Measurement Institute (NMI), which maintains the country's national time scale, much as the Research Institute of Sweden does in Sweden. Telstra drew time from those external references into two stratum 2 servers of its own, in Sydney and Melbourne, which in turn fed three stratum 3 servers, in Sydney, Melbourne and Perth.

Below them sat the clients. In this context that does not mean laptops or phones, but the entire mobile network infrastructure, for instance nodes handling handovers between cell sites. There were thousands of nodes all over a vast geography and every one of them needed to have the same idea of what “now” is, to within a few millionths of a second.

The TAP report describes this setup as “fit for purpose” and that it gave Telstra “a highly reliable and authoritative reference time source from NMI”. 

Stratum in itself does not say if the time is accurate, only the number of steps from a server to its reference. A stratum 1 server with a bad time reference is still a stratum 1 server.

Protection against bad time sources

NTP will therefore need defence against bad time sources. In fact, it has two different ones, and they do different things, both of which assume they are independent from each other.

1. Among otherwise comparable candidates, the lower stratum carries more weight. This is the mechanism that determines which source a client settles on.
    
2. NTP compares several sources and discards those that disagree with the rest. A single source claiming an implausible time is outvoted and dropped, regardless of how authoritative it claims to be.

Neither defence is specific to any particular disruption; together they protect against a broken receiver, a misconfigured server, or an external attack. But these protective measures only work if the time sources that the clients listen to are genuinely independent of each other.

Two ways to deploy NTP

NTP can be deployed in two ways. In client/server mode the relationship is declared and directional: a node takes time from those servers, and nothing else. 

The 2010 Telstra setup was in reality such a client/server model. Peering was allowed, but only at the same stratum level and the TAP report, as noted in the beginning, described this setup as "fit for purpose”.

The other way is a symmetric (peering) mode, where nodes exchange time mutually and settle on whichever source the algorithms currently favour.

Peering is flexible and survives the loss of a source gracefully. But it also means the topology in production is emergent rather than designed. What you documented is a setup that could quietly rearrange itself into a shape no one ever approved.

Telstra's 2020 upgrade

In 2020 the mobile core timing system was upgraded, and new hardware was installed, including a new NTP timing chassis. That installation introduced a few changes.

The first one was forced. The new chassis could not let a stratum 2 server feed a stratum 3 server inside the same box, so the two had to be wired across each other: Sydney's stratum 3 took its time from Melbourne's stratum 2, and Melbourne's stratum 3 from Sydney's. 

In reality, instead of having two stratum 2 sources, each site was left with only one. The TAP report notes that this degradation in redundancy was known and accepted. A second change was leaving the client/server-model in favour of the peering model. The report is not clear about the motivation, but it is reasonable to suggest that one wanted compensation for this loss of redundancy. With each site having just one source instead of two, letting the servers find their own replacements could give the impression of better resilience. 

The TAP report clearly states that the loss of resilience was known. However, it fails to find any evidence that the resulting risk of so-called “timing loops” was identified.

What is a timing loop?

A timing loop is the network equivalent of believing a rumour to be true by asking three people who all heard it from each other. Each one agrees, so it must be true. NTP works basically the same way: it compares several sources and discards whichever disagrees with the rest.

As you may recall from above, NTP has two defenses against bad time sources. The second one protects against timing loops, but only if the sources are independent of each other. In such a loop, sources that appear independent are in fact taking their time from each other, either directly or indirectly by tracing back through a shared reference.

Once a wrong value is circulating, the sources will start agreeing with each other and the vote will be in favour of the majority’s opinion, even though the value is wrong.

The protocol worked. The architecture did not.

Both of NTP’s types of defenses came to be disabled in Melbourne, but five years apart. Not deliberately, but by choices, each of them defensible on their own terms: a hardware limitation had to be worked around, and later, a recurring fault had to be stopped. Each decision solved the problem in front of it. Nobody was asked to look at the sum of all actions.

The second defence was the first one to be disabled. The introduction of peering in 2020 made timing loops possible, and five years later, such a loop showed up. In Melbourne a server started taking time from a node beneath itself. That should have set off alarm bells. However, since accurate time was still reaching the network by other paths, no real harm was done. The underlying problem, the circular dependency, was there, but no one issued a ticket about it.

The actual complaint was quite obvious. Melbourne kept losing contact with its only stratum 2 source in Sydney. With no fallback configuration, the server used peering to find a replacement, sometimes a node beneath it in the hierarchy. 

In October 2025, engineers activated the GPS receiver that had been sitting unused in the Melbourne chassis since 2020 and connected it to the stratum 3 server, as a replacement for the unreliable Sydney source.

By every visible measure it seemed to have worked. Melbourne now had a reliable source of its own and the alarms stopped. But the fix only addressed the symptom, not the root cause. Nobody established why Melbourne kept losing its Sydney source in the first place. The underlying problem was still present in the network by July 2026.

To make things even worse, nobody seems to have understood what activating the GPS card did to the architecture. By adding the GPS card, the Melbourne server went from a stratum 3 server to stratum 1. The engineers didn’t add a source next to the other ones. By promoting a server to the same rank as the national  reference, a new source was created at the very top. As far as NTP is concerned, they carry the same weight. 

Suddenly this GPS card in a chassis in Melbourne, installed as a workaround and reviewed by no one, became the most authoritative server in the hierarchy for the largest mobile network in Australia.

Needless to say, virtually nothing of the 2010 design remained.

By July 2026 the network had a single source that was both the most authoritative candidate available and unopposed, because the sources that could have contradicted it were downstream of it.

This behaviour was very difficult to spot. The network served accurate time every day for years. Architectures like this do not usually degrade gradually. They work, and they keep working, right up until they stop.

GPS week number rollover

The second ingredient is a well-known property of GPS.

GPS broadcasts time as a week number plus seconds-into-week, counted from an epoch that began in early January 1980. In the main civil GPS signal, the week number field is 10 bits, i.e. a maximum of 1,023 weeks. Every 1,024 weeks, or 19.6 years, the counter starts over. This has happened twice: in August 1999 and in April 2019.

Working out which number of epoch it is and adding the right multiple of 1,024 weeks, is the job of the receiver. And the information needs to be in its firmware. 

The problem that occurred in Australia was not a late consequence of any of the GPS rollovers. The card in Melbourne had passed through the second rollover in 2019 without trouble, because a receiver that keeps running also keeps counting. Each new week is simply added to the one before, and the question of which epoch it belongs to is never raised.

However, once you turn it off, that knowledge is gone. When it is turned on again, the receiver has to work out the epoch from scratch, and all it has to go on is what its firmware assumes. The firmware on the Melbourne card had not been updated. Upon start-up it fell back on the earlier epoch and placed the date 1,024 weeks in the past.

What happened next is best understood as the two defences being disabled when they were needed the most.

The first defence, the lower stratum carrying more weight, ranked the Melbourne server highest, because the attached GPS card promoted it to a stratum 1 server. This was according to NTP protocol and thus steered clients towards the one source which was 1,024 weeks wrong.

The second defence, outliers being voted down, was never engaged, because nothing was left to identify Melbourne as an outlier. NTP does not ask whether a date is plausible; it asks whether a source disagrees with the others. The 2010 setup had two stratum 2 servers. If one of them had started announcing the year 2006, the other one would have stayed with 2026 and no consensus would have been reached. That would not have been ideal, but at least the wrong date would not have spread. 

But Melbourne's stratum 2 counterpart had been switched off by the very same chassis replacement, and the remaining sources were downstream of Melbourne. As the wrong date spread, they began reporting it back. Agreement grew, and agreement is what the algorithm is looking for.

So the clients did what they were built to do. Once a majority of a client's sources agreed on November 2006, the client accepted the date, and the further the date travelled, the more convincing it became.

Neither defence malfunctioned. Both had simply been deprived of what they depend on: one needed a source worth ranking highest, the other needed sources capable of disagreeing. Two decisions, five years apart, had removed each in turn.

Key takeaways from the incident

Prioritise and classify time and frequency distribution as critical infrastructure. Manage it accordingly

Document all functions that can take the whole network with them, and put timing on that list. Classification is not paperwork; it is what determines change risk category, review depth, staffing levels, monitoring coverage and budget priority. Telstra's report is, at bottom, the story of one missing entry on that list and everything that followed from it.

Document the whole infrastructure, and every change to it

There was no central repository of NTP configuration, no golden configuration, and no documented record of the servers other than the devices themselves. Without records you cannot perform meaningful pre-checks, you cannot assess impact, and during an incident you cannot tell what "correct" looks like.

Build redundancy in competence

Two engineers performed the change, and both were on mandatory stand-down before the consequences of the GPS card reboot were understood.

Depth of expertise is a resilience property exactly like a redundant power feed. A single specialist, or a pair, means no second opinion, and no one to ask in the middle of the night when maintenance is usually done.

Run a security analysis of the time and frequency infrastructure

Treat timing as an attack surface like any other and analyse it accordingly.

Start with where time enters the organisation. A GNSS signal arriving from space is weak and unauthenticated, and can be jammed or spoofed by cheap equipment. If that signal is your only reference, someone outside your building can decide what time you think it is.

Then look at how it travels. Time distributed over a shared network can be intercepted and manipulated on its way to the client.

Then look at who is allowed to speak. Which servers may your clients accept time from, and who decided that? A source that nobody authorised is a source nobody is checking.

And do not stop at deliberate attack. A timing loop produces much the same effect as a successful spoofing attack: a source the network trusts, delivering a value nobody can contradict. 

Use point-to-point connections

It is easy to see the appeal of peering. It feels like resilience with sources that back each other up: a network that heals itself when a node disappears. But redundancy that arranges itself is not redundancy you can rely on. 

There are safer ways to achieve a similar level of robustness. Netnod runs dedicated point-to-point connections: every relationship is known and documented. Every source is known, and the topology stays the way we designed it. Redundancy comes from multiple independent sources deliberately configured, not from nodes negotiating amongst themselves.

Build an effective alarm organisation

Alarms from the timing platform were not in the standard monitoring tools, and were reviewed only during business hours by a handful of people. Client-side alarms carried neither the severity nor the detail to drive immediate action. Getting this right is organisational as much as technical: alarms reach 24x7 monitoring, severities reflect real consequence, each alarm carries an instruction for what to do about it, and someone owns the response. An alarm no one is on call for is documentation at best, not detection.

Use golden installations

For every class of timing device, keep a known-good reference build and configuration under version control, and check regularly and automatically that what is deployed still matches it. The point is to turn a question like "is this chassis correctly configured and patched?" from something only an expert can answer, and only slowly, into a comparison anyone can run in seconds.

Telstra had nothing of the sort. The TAP report found no such configuration and no record of what the servers should look like other than the servers themselves. The missing firmware update on the Melbourne GPS card had been there for six years, in plain sight. There was simply no automated process that would have flagged it to anyone.

Upgrade and evaluate software continuously

The firmware fix for the rollover behaviour existed and the vendor had published bulletins about it. Vendor notifications need a defined owner and a tracked path to action, and updates need to be applied on a schedule rather than when something forces the issue. 

Evaluate before deploying, in a lab, against the behaviour you actually depend on. Do not forget to verify afterwards. The Telstra changes were completed without anyone checking that the chassis served the correct date.

Redundancy

Redundancy in timing means, not only multiple sources, but independent ones that cannot converge on a common error.  Two servers fed by the same GNSS receiver is still one source, but counted twice. 

Netnod's own service is built on the principle of multiple autonomous nodes, each with independent atomic clocks and redundant servers that are traceable to Swedish National Time realization, UTC(SP).  

Replace equipment continuously

Timing infrastructure usually ages quietly. It keeps working, it rarely complains, and it is therefore a natural candidate when the budget gets trimmed. There will always be other components where the consequences of failure are more visible and therefore gets prioritised. 

Instead, plan replacement on a rolling cycle and design the target architecture first rather than accepting what new hardware installations impose on you. Keeping existing infrastructure healthy should be funded alongside new projects, not be paid for with the left overs.

Concluding remarks

The way Telstra handled the aftermath deserves praise. Commissioning and publishing the independent review is commendable. All providers of critical services, including Netnod, are better off because of this. 

The most unsettling part is perhaps that the outage occurred even though NTP worked just like it was intended to do. 

The problem was everything around it. Architectural choices, budget cuts, low staffing level, lack of proper monitoring, ownership or documentation; it all occurred because no one really appreciated just how vital time services can be. 

Let’s try to change that, shall we?