Disaster Recovery: RTO/RPO & Strategies
RTO is tolerable downtime, RPO is tolerable data loss, both set per tier; pick the cheapest rung on the backup -> pilot-light -> warm-standby -> active/active ladder that meets the tier's numbers; handle corruption and ransomware (not just region loss) with immutable/air-gapped backups; and prove it with restore drills and game-days.
DR is a set of promises, proven by drills
Disaster recovery is not "we have backups." It is a set of promises about how fast you come back and how much data you lose, made per system, and proven by drills. The two numbers that anchor everything are RTO and RPO.
RTO (Recovery Time Objective) is the maximum tolerable downtime: how long the system can be unavailable before the business is unacceptably harmed. RPO (Recovery Point Objective) is the maximum tolerable data loss, measured in time: if you can lose at most 5 minutes of data, your RPO is 5 minutes, which means your recovery point can be no older than 5 minutes before the disaster. RTO is about the clock; RPO is about the data. A checkout system might need RTO of minutes and RPO near zero; a marketing analytics warehouse might be fine with RTO of a day and RPO of an hour.
The strategy ladder
These numbers set your strategy, because recovery speed costs money. The industry ladder, cheapest and slowest first:
Backup & restore: RTO hours to days. Periodic snapshots to durable storage (S3, cross-region); on disaster you provision infra and restore. RPO is only as good as your backup cadence. Cheapest; fine for non-critical tiers.
- Backup & restore: periodic snapshots to durable storage (S3, cross-region). On disaster you provision infra and restore. Cheapest, RTO in hours, RPO as good as your backup cadence. Fine for non-critical tiers.
- Pilot light: keep the critical data continuously replicated to the DR region and a minimal always-on core (the database), but application servers are off. On disaster you start and scale the app tier. RTO in tens of minutes.
- Warm standby: a scaled-down but fully functional copy runs in the DR region all the time. You fail over and scale up. RTO in minutes.
- Multi-site active/active: full capacity live in two or more regions, traffic already flowing to both. A region loss is just a traffic shift. RTO/RPO near zero, and the most expensive by far.
The senior move is to tier your systems and apply a different rung to each. Your payment ledger might warrant warm standby; your recommendation model can live on backup & restore. Spending active/active money on a system whose users would not notice an hour of downtime is a classic waste.
Interview nuance: name the disaster types
Name the disaster types, because they need different recovery. Region loss (fire, flood, power) is what most people picture and multi-region solves. Data corruption or a bad migration is different: it replicates instantly to your standby, so failover just gives you the corrupted data faster. You recover corruption with point-in-time restore from immutable backups, not failover. Ransomware is different again: it may sit dormant and encrypt your backups too, so you need immutable, air-gapped (or object-lock) backups the attacker cannot reach. A DR plan that only handles "the region went away" fails the other two.
And the line that separates real DR from theater: an untested backup is not a DR plan. Backups silently rot, restore scripts break, permissions drift, and the one time you need it you discover the restore takes 14 hours or fails. Real DR means restore-drills, documented runbooks, and periodic game-days where you actually fail over and time it against your stated RTO.
Recap: RTO is tolerable downtime, RPO is tolerable data loss, both set per tier; pick the cheapest rung on the backup -> pilot-light -> warm-standby -> active/active ladder that meets the tier's numbers; handle corruption and ransomware (not just region loss) with immutable/air-gapped backups; and prove it with restore drills and game-days or it is not a plan.
Apply
Your turn
The task this lesson builds to.
Set RTO/RPO per tier for an e-commerce platform and pick a DR strategy (backup/restore, pilot light, warm standby, multi-site active/active) for each, justifying cost.
Think about
- What do RTO and RPO mean concretely?
- How does the strategy ladder trade cost against recovery time?
- Why is an untested backup not a DR plan?
Practice
Make it stick
A second problem on the same idea, so it survives past today.
Design the DR plan for a hospital's electronic health records (EHR) system where regulators require RPO under 15 minutes and RTO under 1 hour, and explain how you would prove the plan works before an auditor asks.
Think about
- How does continuous replication plus WAL archiving hit RPO under 15 minutes?
- Why does a regulated safety-critical system rule out the cheaper ladder rungs?
- How do you produce audit evidence that the plan actually works?