Physical Time, Clock Uncertainty, HLC & TrueTime

Wall-clock ordering silently destroys data

Stamping each write with now() and letting the highest timestamp win (last-writer-wins) feels obvious and is wrong in a way that silently destroys data. Knowing exactly why is a staff-level distinguisher.

Clocks drift, and drift is not small. Machine clocks are quartz oscillators that run fast or slow with temperature. NTP disciplines them over the network but leaves them off by anywhere from a few milliseconds to tens or even hundreds of milliseconds, and NTP can step a clock backward when it corrects. PTP does better (sub-microsecond in a datacenter) but needs special hardware. So at any instant, two nodes can disagree about "now" by tens of milliseconds.

Now run last-writer-wins on wall-clock timestamps. Node A's clock is 50 ms ahead. A user writes X on node B (correct value, real time T). A stale retry lands on node A, whose clock reads T+50 ms even though it happened first in real causal terms. LWW keeps the higher timestamp, so node A's write wins and node B's newer, correct write is silently discarded. No error, no log, just a lost update. This is the classic Cassandra LWW-under-skew data-loss story. Clock skew is a correctness input, not just a dashboard metric.

Hybrid Logical Clocks (HLC)

An HLC timestamp combines a physical component (kept close to NTP wall time) with a logical counter that breaks ties and preserves causality. On an event, HLC takes max(local physical clock, physical part of last seen timestamp) and, if the physical part did not advance, bumps the logical counter. The result: timestamps stay within a bounded distance of real NTP time (human-meaningful, roughly sortable), and they also guarantee that if A → B then HLC(A) < HLC(B), which pure wall clocks do not. HLC needs no special hardware, just NTP, which is why CockroachDB and MongoDB use it. Its limit: HLC gives causal ordering and monotonicity, but it cannot by itself give external (linearizable) consistency across nodes.

Google TrueTime

TrueTime attacks the problem from the hardware side. Every datacenter has GPS receivers and atomic clocks, and the TrueTime API returns an interval [earliest, latest] with a guaranteed bound: now() is somewhere in that window, and the uncertainty (epsilon) is typically a few milliseconds. Spanner uses this for commit-wait: when a transaction commits at timestamp t, Spanner waits out epsilon (until TT.now().earliest > t) before releasing locks and acknowledging. That deliberate wait guarantees any transaction that starts later gets a strictly higher timestamp, giving Spanner external consistency (linearizability) globally. The price: a couple of milliseconds of added commit latency on every write, plus GPS and atomic clocks in every datacenter.

Table

The middle row is why HLC is the default answer: it buys real causal ordering for the price of a counter. The top row is the one to name as a hazard, because its failure mode is silent.

Approach	How it orders writes	What it guarantees	What it costs
LWW on wall clock	highest timestamp wins	Nothing under skew: an older write can beat a newer one	Silent data loss, with no error to alert on
HLC	physical time (NTP) plus a logical counter	Causal and monotonic ordering	No special hardware
TrueTime	an [earliest, latest] interval plus commit-wait ε	External consistency, that is, global linearizability	GPS and atomic clocks in every datacenter, plus a few ms on every commit

Interview nuance: the choice is HLC versus TrueTime, and it is a hardware-versus-guarantee trade. If you control your datacenters and need global linearizable transactions, TrueTime-style bounded uncertainty plus commit-wait is worth the hardware cost. If you run on commodity cloud with only NTP, HLC gets you causal, monotonic timestamps for free, and you accept that you are not externally consistent without an extra coordination step. Saying "just use timestamps" without addressing skew is the tell that someone has not built this.

Recap: NTP/PTP drift is tens of milliseconds and real, LWW on wall-clock timestamps silently drops writes under skew, HLC gives causal + monotonic timestamps on plain NTP hardware, and TrueTime's bounded interval plus commit-wait buys global external consistency at the cost of GPS/atomic-clock infrastructure and a few ms per commit.