IaC, Environments & Progressive Delivery

Two failure modes ruin infrastructure delivery

Drift (staging and prod diverge because someone made a manual console change, so a deploy that passed staging breaks prod) and big-bang rollout (you ship to 100% at once, and if it regresses you have already taken an outage before you notice). This lesson kills both.

Infrastructure as Code fixes drift

Declare the desired infrastructure in Terraform/OpenTofu or Pulumi, keep it in Git, and apply through a pipeline, never by hand. Key discipline: remote state with locking (an S3 backend plus DynamoDB lock, or Terraform Cloud) so two engineers cannot corrupt state with concurrent applies, and modules so dev/staging/prod are the same module with different variable files. That gives environment parity: prod is staging with more replicas, not a different snowflake. Treat infra as immutable: to change a node you replace it, you do not SSH in and tweak it. Manual console changes are the cardinal sin because they are invisible to Git and cause the exact drift that makes staging a liar. You can catch drift by running terraform plan on a schedule and alerting on any non-empty diff.

Environment promotion: the same versioned artifact and same IaC modules flow dev to staging to prod. Config differs only by variables (replica counts, instance sizes, endpoints), ideally sourced from the same place, so promotion is "apply the tested module to the next environment," not "rebuild it."

Progressive delivery fixes big-bang rollout

  rolling     : replace pods N at a time; cheap, no extra capacity, slow to detect a bad version
  blue-green   : full parallel env, flip the router; instant rollback, but 2x capacity briefly
  canary       : send 1% -> 5% -> 25% -> 100%, watch metrics, auto-halt on regression

For a critical payments service you want canary with automated analysis and auto-rollback. Tools: Argo Rollouts or Flagger shift traffic in steps, and between steps they bake (hold and observe) while querying Prometheus for your SLIs: error rate, p99 latency, and a business metric like payment-authorization-success-rate. If any metric breaches its threshold during the bake, the rollout auto-aborts and shifts traffic back to the stable version. No human in the loop at 3am. Blue-green is the alternative when you cannot tolerate two versions serving simultaneously (it flips atomically) but it costs double capacity during the window.

Feature flags decouple deploy from release. Deploying code and releasing a feature become separate events: ship the code dark behind a flag (LaunchDarkly, Unleash, or a homegrown flag service), then turn it on for 1% of users independent of the deploy. This means you can roll back a feature instantly without redeploying, and you can deploy risky code safely because it is inert until flagged on.

Interview nuance: database migrations are the trap in any progressive rollout. Canary assumes old and new code run simultaneously, so a destructive migration in one deploy (drop a column the old version still reads) breaks the stable version mid-canary. Use expand/contract (a.k.a. parallel-change): first expand (add the new column, write to both, backfill), deploy code reading the new shape, then in a later deploy contract (drop the old column) once nothing reads it. Migrations must be backward-compatible across at least one version.

Recap: IaC (Terraform/OpenTofu) with remote-state locking and shared modules gives environment parity and kills drift; never make manual console changes; promote the same artifact dev to staging to prod; use canary with automated metric analysis and auto-rollback (Argo Rollouts/Flagger) for a payments service, blue-green when versions cannot coexist; feature flags decouple deploy from release; and use expand/contract so a migration never breaks the version still running during a canary.