Skip to main content

Quorums & Dynamo-Style Tunable Consistency

Level 5: Level 5: Distributed Systems Corehard30 minquorumstunable-consistencydynamo

R+W>N forces read/write overlap (quorum consistency, NOT linearizability); latency tracks the slowest quorum member, and sloppy quorums buy availability during partitions.

Raft gives one fixed answer; Dynamo hands you a dial

Dynamo-style systems (DynamoDB's underpinnings, Cassandra, Riak, ScyllaDB) let you choose three numbers per operation, trading durability, consistency, and latency against each other.

The three knobs:

  • N: the replication factor, how many nodes store each key (say 3).
  • W: how many replicas must acknowledge a write before the client is told it succeeded.
  • R: how many replicas must respond to a read before the client gets an answer.

The one rule to memorize is R + W > N. When that holds, the set of nodes a read touches and the set a write touched must overlap in at least one node (pigeonhole: two subsets of N whose sizes sum to more than N cannot be disjoint). That overlapping node has seen the latest write, so a read is guaranteed to observe at least one copy of the freshest value.

Check yourself
N=3, R=2, W=2, so 'R + W > N' holds and every read set overlaps every acknowledged write set. Does this configuration give you strong consistency (linearizability)?
  N=3 nodes: [1][2][3]
  W=2 write acked by {1,2}
  R=2 read from {2,3}   -> overlap = node 2 -> sees latest
  since 2+2 > 3, no read/write pair can miss each other

What R+W>N does NOT give you

This is the number-one trap. Quorum overlap guarantees a read sees the latest acknowledged write, but it does not give you linearizability. Concurrent writes to different quorums can produce conflicting versions that must be reconciled with version vectors (Dynamo returns siblings) or last-write-wins by timestamp (Cassandra, which silently drops the loser). A read during an in-flight write may see the old or new value depending on timing, and there is no guarantee about the order two clients observe events in. If you need true linearizability, you need consensus (Raft/Paxos), not quorums. Claiming "R+W>N gives strong consistency" is the classic wrong turn; it gives quorum consistency, which is weaker.

Latency, sloppy quorums, and intent

Latency is bounded by the slowest node in the quorum. A write with W=2 of N=3 waits for the 2nd-fastest replica. Raising W or R toward N makes latency track a higher tail percentile: with N=3, R=3, one slow node (GC pause, hot disk) drags every read to its p99. Mitigate with speculative/hedged reads (send to R+1, take the first R) and keep W and R as low as the consistency requirement allows.

Sloppy quorum and hinted handoff trade consistency for availability. In a strict quorum, if the W home replicas are unreachable, the write fails. A sloppy quorum writes to the next W healthy nodes on the ring even if they are not the key's usual owners, storing a hint so those temporary holders forward the data back once the rightful replicas recover. Writes stay accepted during partitions at the cost of a window where a strict-quorum read might miss the value.

Check yourself

Sort each behavior by which quorum flavor produces it.

A write fails when the key's W home replicas are unreachable
Writes keep succeeding during a partition, accepted by stand-in nodes on the ring
A window where a read can miss a write that was already acknowledged
The 'R + W > N' overlap guarantee actually holds

Interview nuance, map numbers to intent: W=N maximizes durability but breaks writes if any replica is down. R=1, W=N gives fast reads and slow fragile writes. R=N, W=1 the reverse. W=1, R=1 is fastest and weakest (no overlap). Also mention flexible quorums (write and read sets defined to intersect without both being majorities) and witness replicas (vote for quorum without storing full data, cutting storage cost while preserving overlap).

Recap: N/R/W is a per-operation dial, R+W>N forces read/write overlap so a read sees the latest acknowledged write, but that is quorum consistency not linearizability, quorum latency tracks the slowest node in the set, and sloppy quorum plus hinted handoff buy availability during partitions at the cost of consistency.

Check yourself

Last pass before you pick your own N, R, and W: sort what quorum overlap actually buys you.

A read sees at least one copy of the latest acknowledged write
Concurrent writes to different quorums resolve to one agreed value
All clients observe writes in the same order
Overlap still holds while a sloppy quorum is accepting writes on stand-in nodes
Read latency tracks the slowest replica in the read set

Apply

Your turn

The task this lesson builds to.

Choose N/R/W for a session store that must survive one AZ loss and still serve fast reads, and state the consistency you actually get.

Think about

  1. What does R+W>N guarantee, and what does it NOT guarantee?
  2. Why is quorum latency bounded by the slowest node?
  3. How do sloppy quorum and hinted handoff trade consistency for availability?

Practice

Make it stick

A second problem on the same idea, so it survives past today.

Design the replication settings for a Cassandra-backed IoT telemetry store ingesting 500k writes/sec from sensors across two regions, where writes must almost never be rejected but analytics reads can tolerate seconds of staleness. Choose consistency levels and explain what breaks if a region is partitioned.

Think about

  1. Why does LOCAL_QUORUM beat EACH_QUORUM for the never-reject-writes requirement?
  2. Where does R+W>N hold in this design, and where deliberately not?
  3. What reconciles the regions after a partition heals?