Skip to main content

Multi-Region & Multi-AZ Architecture

Level 7: Level 7: Reliability, Resilience & Operationshard35 minmulti-regionreplicationfailover

Multi-AZ is cheap synchronous HA within a region; multi-region is expensive async protection against region loss; sync gives RPO~0 but latency and stall risk while async gives speed but lag/loss; active-active forces a consistency choice (single-writer-region, CRDTs, or a Spanner-class store); steer with health-based failover and actually test region evacuation.

Multi-AZ and multi-region are different tools

Multi-AZ and multi-region are not the same tool, and conflating them is a common tell. Know exactly what each buys and what it costs.

Multi-AZ spreads a system across Availability Zones: physically separate data centers within one region, tens of km apart, connected by fast, low-latency (single-digit ms) links. Because latency between AZs is tiny, you can replicate synchronously across them cheaply, so multi-AZ is the default for HA. It protects against a data-center failure (power, cooling, a fire in one building) but not against a whole-region outage, and not against region-wide control-plane failures.

Multi-region spreads across regions hundreds or thousands of km apart, with 50-150+ ms of round-trip latency between them. It protects against losing an entire region (natural disaster, region-wide provider outage, regulatory blackout). But that latency changes everything about data: synchronous replication across regions would add 100+ ms to every write, so you almost always replicate asynchronously, which means the remote copy lags and a region loss can lose the un-replicated tail. Multi-region is expensive (full or partial stacks in each region, cross-region data transfer, more operational surface) and it makes consistency genuinely hard.

The consistency crux (CAP made concrete)

Across a WAN partition you cannot have both strong consistency and full availability:

  • Sync replication: a write is acknowledged only after it lands in the remote region. RPO is zero, but every write eats the cross-region round trip, and if the remote region is unreachable your writes stall (you chose consistency over availability).
  • Async replication: acknowledge locally, ship to the remote region in the background. Writes are fast and stay available, but the remote copy lags (seconds), so a sudden region loss loses the in-flight tail (non-zero RPO), and reads from the remote region can be stale.

For active-passive multi-region (one region serves, the other is a hot standby), async is standard: the passive region trails by seconds, and on failover you accept that small RPO. Simple, one writer, no conflicts.

For active-active multi-region (both regions take writes), you now have two places accepting writes to the same data, and reconciling them is the whole problem. Options:

  • Single-writer-region per record: partition ownership so each record (or shard/tenant) has exactly one home region that owns its writes; other regions forward writes there or serve read-only. Avoids conflicts entirely at the cost of cross-region write latency for non-local records. This is the most common sane choice.
  • Conflict resolution: allow writes anywhere and reconcile. Last-writer-wins (simple, silently drops one update), vector clocks (detect conflicts, push resolution to the app), or CRDTs (conflict-free replicated data types that merge deterministically, great for counters/sets/carts, not for everything).
  • Globally consistent stores like Spanner or CockroachDB use synchronized clocks (TrueTime) and consensus to give strong consistency across regions, paying the latency, so you do not hand-roll conflict logic.

Traffic steering sits on top: GeoDNS (route by client location, but DNS TTL caching makes failover slow), a global load balancer / anycast (AWS Global Accelerator, Cloudflare) that health-checks regions and shifts traffic in seconds. Health-based failover moves traffic off a dead region automatically.

Interview nuance: two things separate strong answers. First, cell-based and shuffle-sharding thinking applies here: an active-active pair still shares a blast radius if a bad config or poison request replicates to both, so regions should fail independently and you must test region evacuation (actually drain a region) rather than assume it works. Second, do not claim multi-region gives strong consistency for free. It does not. You either pay cross-region latency (sync/Spanner) or accept eventual consistency and design conflict resolution. Saying "we go multi-region active-active and everything is consistent and fast" is the wrong turn interviewers wait for.

Recap: multi-AZ is cheap synchronous HA within a region; multi-region is expensive async protection against region loss; sync gives RPO~0 but latency and stall risk while async gives speed but lag/loss; active-active forces a consistency choice (single-writer-region, CRDTs, or a Spanner-class store); steer with GeoDNS/global-LB health-based failover and actually test region evacuation.

Apply

Your turn

The task this lesson builds to.

Design a multi-region deployment for a globally used app; decide active-active vs active-passive, data replication mode, and traffic routing/failover.

Think about

  1. What do multi-AZ and multi-region each protect against, and at what cost?
  2. Sync vs async replication: what is the RPO and latency tradeoff?
  3. How do you resolve conflicts in active-active?

Practice

Make it stick

A second problem on the same idea, so it survives past today.

Design the multi-region data layer for a global shopping-cart service (think Amazon-scale) where the cart must always accept 'add to item' writes even during a network partition, and no item a user added may silently vanish.

Think about

  1. Why choose AP over CP for the cart, and what does that cost?
  2. Why is last-writer-wins wrong, and what merges adds without loss?
  3. How do you handle quantities and removals without un-deleting forever?