Idempotency & Deduplication

Idempotency, past the definition

Level 5's delivery-and-idempotency lesson established the key mechanics: attach a unique idempotency key to a request, do the work once, store the result keyed by that key, and return the stored result on any retry, all written atomically with the side effect. This lesson takes that as given and goes at the two things that actually break in production once you accept at-least-once delivery: the concurrent-duplicate race, and how long you must remember a key.

The concurrent-duplicate race

The concurrent-duplicate race is what interviewers probe. Two copies of the same request arrive at two servers at the same millisecond. A read-then-write check ("is this key present? no -> insert") has a race between the read and the write where both pass, both execute, and you double-apply. You must make the check-and-set atomic:

  INSERT INTO idempotency_keys (key, status, result)
  VALUES ($1, 'in_progress', NULL)
  ON CONFLICT (key) DO NOTHING;
  -- exactly one inserter wins the unique constraint; the loser
  -- re-reads the row and waits for / returns the winner's result

A unique constraint (or a Redis SET key value NX, or a DynamoDB conditional PutItem with attribute_not_exists) makes exactly one writer win. The loser reads back the row: if it is in_progress, it waits or returns 409/retry; if completed, it returns the stored result.

Store the result, not a boolean. This is the detail the race turns on. If the dedup store keeps only a "seen" flag, two concurrent duplicates both find "not seen," both execute, and there is no stored answer to hand back. Save the outcome (the created order id, the HTTP response body) keyed by the idempotency key, and return it verbatim on any repeat.

Sizing the dedup window

The dedup store keeps keys for a TTL, and that TTL must be at least as long as the longest window in which a duplicate can arrive. Two windows matter: the client retry horizon (how long clients keep retrying, usually minutes) and the broker replay/retention window (Kafka can replay days of history during a reprocess or consumer reset). If your dedup TTL is 1 hour but you replay a 3-day-old topic, every replayed message looks new and re-applies. Size the TTL to cover the replay window, or use a permanent natural key so replays are inherently safe.

The idempotency toolkit

The dedup store is the general fallback, but two cheaper flavors come first when the operation allows:

1. Naturally idempotent operations. SET status = shipped or an upsert keyed by a stable id lands in the same state however many times it runs. Design for this where you can. INCR balance is the opposite: repeating it corrupts state, so counters need explicit protection.
2. Idempotent by design via state machines. Model the aggregate as states with legal transitions (CREATED -> PAID -> SHIPPED). A command that tries an already-taken transition is a no-op, and a per-aggregate expected version (optimistic concurrency) rejects a replayed or stale command outright.
3. Enforced idempotency via a dedup store. For everything else, the idempotency-key-plus-stored-result machinery above, guarded by the atomic check-and-set.

Interview nuance: distinguish the idempotency key's scope. A client-supplied key dedups client retries of the same logical request. An event-id key dedups broker redeliveries. They are different keys guarding different duplicate sources, and a robust design often uses both.

Recap: past the L5 key mechanics, the two production hazards are the concurrent-duplicate race (resolve it with an atomic check-and-set on a unique constraint, storing the result not a flag) and the dedup window (size the TTL to cover both the client-retry and broker-replay horizons); prefer naturally idempotent operations or state-machine transitions before falling back to a dedup store.