Secrets & Key Management (KMS/HSM, Rotation)

Secrets unlock everything else

Secrets (DB passwords, API keys, signing keys, TLS private keys) are the credentials that unlock everything else, so how you store, distribute, and rotate them is a top-tier design problem. The failure everyone starts with is secrets in env vars, config files, or source control. Those leak through git history, CI logs, crash dumps, /proc, and container images, and they cannot be rotated or audited. The first principle is a dedicated secret store: HashiCorp Vault, AWS/GCP Secrets Manager, or a cloud KMS.

KMS vs HSM

A KMS is a managed key service with an API for encrypt/decrypt/sign where keys never leave the service. An HSM is the tamper-resistant hardware (often FIPS 140-2 Level 3 certified) that actually holds the root keys; managed KMS is usually HSM-backed. The pattern is a key hierarchy with a hardware-backed root of trust: the HSM holds the root KEK, which wraps intermediate keys, which wrap DEKs. Nothing sensitive exists in plaintext outside the hardware boundary, and you get a single audited choke point for every key operation.

Rotation without downtime

Naive rotation ("change the password, restart everything") causes an outage the moment the old credential dies before every client picks up the new one. The fix is versioned secrets with a dual-secret (overlap) window: create version N+1 while N still works, roll consumers over gradually, confirm nothing uses N, then revoke N. For encryption keys, decrypt with old-or-new during the window and re-encrypt lazily. This turns rotation from a risky event into a routine, reversible rollout.

The "secret zero" problem

If every secret lives in Vault, the app needs a credential to authenticate to Vault, so what protects that credential? Bootstrapping trust with a long-lived static token just moves the problem and recreates the thing you were avoiding. The modern answer is workload identity: the platform vouches for the workload so no pre-placed secret is needed.

• On Kubernetes / cloud, the workload gets a short-lived OIDC/JWT identity token from the platform (IRSA on EKS, GKE Workload Identity), and the secret store trusts that issuer. No static credential is ever placed on the box.
• SPIFFE/SPIRE issues a cryptographic SVID (an X.509 cert or JWT) that attests the workload's identity, which Vault or a mesh accepts.

The workload then fetches dynamic, short-lived secrets: instead of a shared static DB password, Vault generates a unique DB credential that lives 1 hour and is auto-revoked. A leak is self-limiting, and every credential is traceable to one workload.

 secret zero solved:
   platform (K8s/cloud) --signs--> short-lived OIDC/SVID for the pod
   pod --presents identity--> [Vault] --verifies issuer--> issues
        dynamic DB cred (TTL 1h), unique per pod, auto-revoked
   no static credential ever stored on the pod

Interview nuance: rotation and dynamic secrets are useless without least-privilege policies and per-access audit logging. Every secret read should be logged (who, which workload, when) so a leak has a blast-radius answer, and policies should scope each workload to only the secrets it needs. Pair this with leaked-credential scanning (pre-commit hooks, GitHub secret scanning) to catch the static keys that inevitably slip through, and auto-revoke on detection.

Recap: put secrets in a dedicated store rooted in an HSM-backed KMS, rotate with versioned dual-secret windows so there is no downtime, solve secret zero with platform-issued workload identity that hands out short-lived dynamic secrets, and log every access under least-privilege policies.