Redundancy, Failover & Health Checking

No single point of failure

Availability starts with a simple rule: no component whose failure takes down the system may exist as a single instance. A single point of failure (SPOF) is any box, process, or record that has no live substitute. Redundancy is having N+1 or N+2 of everything, so losing one (or two) instances still leaves enough capacity to serve.

The trap is that SPOFs hide. Engineers dutifully run three web servers, then route all of them through one load balancer, one database primary, one DNS name backed by one provider, and one config service that every pod reads on boot. Each of those is a SPOF that quietly undoes the redundant web tier. A real audit walks the request path and asks, at every hop, "if this single thing dies, does traffic stop?" Load balancers need a redundant pair (or a managed multi-node LB like AWS ALB/NLB); the DB primary needs replicas plus automated promotion; DNS needs multiple providers or at least multiple authoritative servers; the config store needs a quorum (etcd/ZooKeeper run 3 or 5 nodes for exactly this reason).

Two shapes of redundancy

Active-active: every instance serves live traffic, so you use the capacity you pay for and failover is instant (just stop routing to the dead one). The cost is shared state, which is hard when instances are stateful (two DB primaries accepting writes will diverge). Active-passive: a hot or warm standby sits idle until the primary fails, then gets promoted. Simpler to reason about because only one instance owns the state, but you pay for idle hardware and you eat a failover lag while the standby takes over.

Health checking triggers failover

Failover has to be triggered by something, and that something is health checking. Three depths matter:

• Liveness: is the process up? (answers a TCP connect or a trivial /healthz). If it fails, restart the instance.
• Readiness: can this instance serve right now? (warmed caches, DB pool connected). If it fails, pull it from the LB pool but do not kill it.
• Deep / dependency check: can it reach its critical dependencies? Useful but dangerous: if your health check calls the shared database and the database blips, every instance fails its check at once, the LB pulls them all, and a minor blip becomes a total outage.

Interview nuance: the two failure modes interviewers probe are flapping and split-brain. Flapping is an instance that fails and recovers repeatedly, causing constant add/remove churn; you damp it with hysteresis (require N consecutive failures to eject, M consecutive successes to re-admit) and cooldowns. Split-brain is worse: during a network partition, a passive standby cannot tell "primary is dead" from "I just cannot reach the primary," promotes itself, and now you have two primaries taking writes. The fix is to never let a single node decide promotion. Use quorum-based leader election (Raft/Paxos, or a fencing token) so a minority side cannot win, and fence the old primary (STONITH, revoke its storage lease) before the new one takes over. Also plan failback: returning to the recovered primary is its own controlled operation, not automatic.

        clients
           |
     [ DNS: 2 providers ]
           |
   [ LB pair, active-active ]
        /        \
   web-1       web-2 ... web-N     (N+2, stateless, readiness-gated)
        \        /
   [ DB primary ]==async/sync==>[ replica ]
     leader elected via quorum; fence old primary on failover

Recap: eliminate every SPOF (LB, DB primary, DNS, config) with N+1/N+2 redundancy, pick active-active for instant failover or active-passive for simpler state, gate traffic with liveness/readiness/deep checks, and use quorum election plus fencing and hysteresis to avoid split-brain and flapping.