Skip to main content

Replication Topologies & Consistency

Level 3: Level 3: Scaling the Data Tierhard35 minreplicationconsistencyconflict-resolution

Single-leader avoids conflicts, multi-leader buys local writes at the cost of write-write conflicts, leaderless uses R+W>N quorums; resolve conflicts losslessly and reason with PACELC.

Each topology buys a capability by exposing an anomaly

Once one leader is not enough (you need multi-region writes, or you want no write SPOF), you choose among three replication topologies, and each one buys a capability by exposing a specific class of anomaly. Knowing which anomaly you are signing up for is the whole skill.

Single-leader: all writes go through one node, which serializes them, so there are no write-write conflicts, and it is the easiest to reason about. The costs are that the leader is a write SPOF (failover is required and risky) and cross-region writers pay the latency to reach the one leader's region. This is the default for most OLTP systems.

Multi-leader: several leaders (typically one per region) each accept writes and replicate to the others. This gives low-latency local writes everywhere and survives a region outage for writes. The price is brutal: two leaders can accept conflicting writes to the same key concurrently, and you must define how to merge them. Use it when local write latency or offline/multi-datacenter operation genuinely requires it, not by default.

Check yourself
Two regional leaders accept concurrent edits to the same profile field. A teammate proposes: just keep whichever write carries the later timestamp. Is that a safe default?

Leaderless (Dynamo-style): any replica accepts a write, and the client (or a coordinator) writes to and reads from multiple replicas. Cassandra, DynamoDB, and Riak work this way. Consistency comes from quorums: with N replicas, if you require W replicas to ack a write and R to answer a read, then R + W > N guarantees the read set and write set overlap on at least one node, so a read sees the latest acked write. Common config is N=3, W=2, R=2. Tuning W and R trades consistency against availability and latency: W=1 is fast but weakly durable, R=1 can read stale data.

Check yourself

Match each behavior to the replication topology that produces it.

No write-write conflicts, because one node serializes every write
Low-latency local writes in every region, with concurrent conflicting writes to merge
Any replica accepts writes; reads are safe when 'R + W > N'
Write availability hinges on risky failover when one node dies
Sloppy quorums and hinted handoff keep writes flowing during node failures

Two more leaderless mechanics interviewers probe. Sloppy quorums with hinted handoff keep the system available during failures by letting writes land on temporary "stand-in" nodes when the home replicas are down, then handing the data off when they recover; this trades consistency for availability. Anti-entropy converges divergent replicas in the background: read repair fixes stale replicas noticed during a read, and Merkle trees let two replicas efficiently find and reconcile the exact ranges that differ.

Conflict resolution: where these designs live or die

  • Last-write-wins (LWW): pick the write with the highest timestamp, discard the rest. Simple, and Cassandra's default, but it silently loses data for concurrent writes and depends on clock sync.
  • Version vectors: track a per-replica counter so the system can tell whether two writes were concurrent or causally ordered, then surface genuine conflicts to the app or merge them.
  • CRDTs: data types (counters, sets, sequences) mathematically designed so concurrent updates always merge deterministically without loss.
  • Application merge: hand both versions to business logic (shopping-cart union is the classic Dynamo example).

Interview nuance: do not answer with a CAP binary ("CP or AP"). Reason with PACELC: if there is a Partition, choose Availability or Consistency; Else (normal operation) choose Latency or Consistency. Dynamo-style stores are PA/EL; a single-leader RDBMS is PC/EC. Then name the concrete anomaly a user sees ("two edits from two regions, one silently overwrites the other under LWW"), which shows you reason about data, not letters.

SINGLE-LEADER          MULTI-LEADER              LEADERLESS (quorum)
 all writes -> L         L(us) <--> L(eu)          client -> W of N replicas
 no conflicts            local writes, but         reads <- R of N
 leader = write SPOF     concurrent conflicts      R + W > N => overlap

Recap: single-leader avoids conflicts but has a write SPOF; multi-leader enables multi-region writes at the cost of write-write conflicts; leaderless uses R + W > N quorums (plus sloppy quorums, hinted handoff, read repair, and Merkle trees) for availability; resolve conflicts with LWW (lossy), version vectors, CRDTs, or app merge, and reason with PACELC and named anomalies rather than CAP.

Check yourself
The interviewer asks: 'So is your store CP or AP?' What is the strongest move?

Apply

Your turn

The task this lesson builds to.

Design the replication + consistency scheme for a globally-used note app where two users may edit from different regions; state exactly which stale reads and conflicts are possible.

Think about

  1. Where does each topology fit, and what conflicts does it create?
  2. How do quorum reads/writes (R + W > N) give strong-ish consistency?
  3. How is a write-write conflict resolved (LWW, version vectors, CRDT)?

Practice

Make it stick

A second problem on the same idea, so it survives past today.

Design the replication and consistency model for a DynamoDB-style shopping cart backing a large e-commerce site: N=3 replicas per key, writes must never be rejected (an 'add to cart' always succeeds even during a node or network failure), and a user must never lose an item they added from two devices.

Think about

  1. What quorum settings and mechanisms keep a write succeeding when home replicas are down?
  2. Why does LWW on the whole cart object violate the never-lose-an-item requirement?
  3. What makes deletes the subtle case in a merge-by-union cart?