Leader Election, Leases, Fencing & Split-Brain

One leader, guaranteed, even when the network lies

Many systems need a single active primary: one node that owns writes, holds a lock, or coordinates work. The hard part is not electing one, it is guaranteeing there is never a second one acting at the same time, because the asynchronous network gives you no reliable way to tell a dead primary from a slow one.

Electing is the easy half. Run it through a consensus system: etcd, ZooKeeper, or Consul, or a Raft/Paxos group directly. The primary holds a lease: a time-bounded grant ("you are leader until T+10s") it must renew. If renewals stop, the lease expires and a new election runs. Leases need no per-request coordination, but they carry a hidden assumption: bounded clocks and bounded pauses.

The canonical failure

Leader L holds a 10-second lease and is mid-write. L suffers a stop-the-world GC pause (or the VM is descheduled, or a disk stall) for 15 seconds. From everyone else's view, L went silent, its lease expired, and a new leader L2 was elected and started writing. Then L wakes up. L does not know time passed. It believes it still holds the lease and completes its in-flight write. Now two leaders have both written: split-brain, and the data is corrupted. No clock was "wrong" and no bug was hit; a legal pause alone produced two active leaders.

Fencing tokens: enforce at the resource

Leases alone cannot fix this, because the paused leader's problem is that its own view of "do I still hold the lease" is stale. The fix lives at the resource. Every time leadership is granted, the coordinator hands out a monotonically increasing number (etcd revision, ZooKeeper zxid, a Raft term). The leader attaches that token to every write. The storage layer remembers the highest token it has seen and rejects any write with a lower token. When paused L wakes and writes with token 33, storage has already accepted L2's token-34 writes and rejects L. This is the piece a distributed lock must have: a lock that only tells the client "you have it" is unsafe, because the client can be paused between acquiring and using it. This is exactly Martin Kleppmann's critique of Redlock.

The token is checked where the damage would happen

Step 1 / 3

Stage 1 of 3: Leadership is granted with a monotonically increasing token: an etcd revision, a ZooKeeper zxid, or a Raft term.

Split-brain and partitions

Take a 5-node cluster split 3-2. Consensus-based leadership requires a majority quorum (3 of 5). The majority side can elect and keep a leader and stays writable; the minority side cannot reach quorum, steps down, and refuses writes. This is the CP choice (Raft/etcd/ZooKeeper). The AP alternative (Dynamo-style) lets both sides keep accepting writes and reconciles later with CRDTs/version vectors. Either way, fence the minority: the losing side must be provably unable to affect shared state. A 2-2 split of 4 nodes has no majority on either side, which is why consensus clusters use odd numbers.

Interview nuance: the answer that gets you hired names both halves: consensus/lease for who leads, and a fencing token enforced at the storage layer for why a paused old leader cannot corrupt state. Stopping at "etcd elects a leader" fails the follow-up "what happens during a GC pause?"

Recap: elect one leader via consensus and a lease, but because a GC pause can briefly create two leaders, enforce fencing tokens (monotonic numbers the storage rejects if stale); on a 3-2 split the majority stays writable (CP) or both sides reconcile (AP), and either way the minority is fenced.