Delivery Semantics, Idempotency & Exactly-Once Reality

The most misunderstood phrase in distributed systems

The correct mental model: exactly-once delivery over an unreliable network is impossible; exactly-once effect is achievable by combining at-least-once delivery with idempotency.

The three delivery semantics:

• At-most-once: send and forget. No duplicates, but possible loss. Fine for a metric sample, fatal for a payment.
• At-least-once: send and retry until acknowledged. No loss, but if the ack is lost the sender retries and the receiver may process twice: duplicates possible.
• Exactly-once (delivery): each message delivered and processed once, no loss, no duplicates. Over a real network you cannot have this at the delivery layer.

Why impossible? Because acknowledgements can be lost too. Sender sends, receiver processes, receiver's ack is dropped. The sender cannot distinguish "message lost" from "ack lost," so to avoid loss it must retry, and retrying risks a duplicate. A consequence of the two-generals problem: no finite exchange over a lossy channel makes both sides certain. So every real system that "cannot lose data" runs at-least-once and deduplicates.

The idempotency key

The client attaches a unique key to a request (Idempotency-Key: a1b2..., as Stripe does). The server, on first receipt, does the work and stores the result keyed by that idempotency key (with a TTL). On any retry with the same key, the server returns the stored result instead of redoing the work. The effect happened exactly once even though the request arrived multiple times. The critical detail: recording the key and performing the side effect must be atomic (same transaction), or a crash between them reopens the double-execution window.

Distinguish the operation types:

• Naturally idempotent: SET balance = 5, PUT user.email = x, delete by id. Applying twice yields the same state: safe to retry with no machinery.
• Non-idempotent: balance = balance + 100, "charge $50," "append to list." Applying twice doubles the effect. Make them safe with an idempotency key plus stored result, or convert to conditional/versioned updates (compare-and-set, or a unique constraint on the operation id).

Fencing tokens (Kafka EOS lives in Level 6)

One boundary to name and then hand off: Kafka's "exactly-once semantics" (EOS) convert at-least-once into effectively-once, but only within a Kafka-to-Kafka pipeline, never for external side effects like charging a card. Level 6's Kafka material owns that scoping in depth, so this lesson states it in one line and keeps its own distinct ground: fencing tokens.

Fencing tokens protect against a different failure: a stale operation from a delayed or paused actor. A process pauses (long GC), is presumed dead, a new one takes over, then the old one wakes and issues a now-stale write. A monotonically increasing fencing token attached to each operation, rejected by the storage layer if lower than the highest seen, neutralizes the zombie write. Idempotency keys stop duplicates of the same intent; fencing tokens stop stale operations from a superseded actor. Different problems, both needed.

Interview nuance: if you say "we use exactly-once delivery" you will get pushed. Say instead "at-least-once delivery plus idempotent processing gives exactly-once effect," and name where the dedup state lives and how it is made atomic with the side effect.

Recap: networks force at-most-once (may lose) or at-least-once (may duplicate); build exactly-once effect with an idempotency key whose stored result is written atomically with the side effect; and fencing tokens separately reject stale writes from a superseded actor (Kafka's own EOS scoping is covered in Level 6's Kafka material).