Cache Stampede, Thundering Herd & Hot Keys

A cache works right up until a popular key expires

In the instant a popular entry disappears, every concurrent request for it misses at once, and each one independently tries to rebuild it by querying the database. This is the cache stampede (thundering herd, dog-piling), and it is one of the most common ways a healthy system takes itself down: the cache was hiding, say, 10K req/s worth of a 300ms query, and now all of those requests hit the origin in the same window, so the DB suddenly has roughly 3,000 concurrent copies of a slow query and its CPU goes to 100%. Worse, because the DB is now slow, each rebuild takes longer, so more requests pile up before the first one finishes, and the cache never gets re-populated. The system spirals.

Three families of defense exist, and a good design layers them rather than betting on one.

Request coalescing (singleflight)

When a key is missing, only the first requester rebuilds it, and every other concurrent requester for the same key waits for that single in-flight rebuild and shares its result. Go's singleflight package is the canonical implementation, but the pattern is universal: a per-key lock serializes recomputation. The first thread acquires the lock and rebuilds; the rest block briefly and then read the freshly populated cache. This turns 3,000 concurrent DB queries into exactly one. If the lock is process-local you protect one app node; to protect the DB from a whole fleet you use a distributed lock (a short-lived Redis SET NX key) so exactly one node across the fleet rebuilds.

Beating the synchronized expiry

Even with coalescing, a hard TTL means the key vanishes at a single instant. TTL jitter spreads the expiry of a cohort of related keys over a window so they do not all expire together. Probabilistic early recomputation (the XFetch algorithm) refreshes a key slightly before its TTL, with a probability that rises as expiry approaches, so a single lucky reader rebuilds the value in the background while the still-valid cached value keeps serving everyone else. The key never actually expires under load. A simpler cousin is stale-while-revalidate: serve the stale value immediately and kick off one async refresh.

The genuinely hot key

Sometimes the problem is not expiry but sheer volume: one key (a viral tweet, a flash-sale SKU) is read so often that even a single Redis shard cannot serve it, because all requests for one key hash to one shard. Coalescing does not help here since the value is present; the shard is simply saturated. The fixes are key replication (write the value under N suffixed keys hotkey:0..N spread across shards and have clients read a random one) and a client-side near cache (L1) on each app server so most reads never reach Redis at all. Hot-key detection (per-key request rates) tells you which keys need this treatment.

Interview nuance: the subtlety interviewers push on is what happens right after a cache flush or cold start. A cold cache is a stampede on every key at once, so "just flush and warm up" is dangerous at scale. You warm the cache before taking traffic, or ramp traffic gradually, and keep coalescing on. Treating a flush as free is the classic mistake.

NAIVE (stampede)                 COALESCED (singleflight)
 key expires                      key expires
  req1 -> DB \                     req1 -> lock -> DB -> set cache
  req2 -> DB  }  N queries         req2..N -> wait -> read cache
  reqN -> DB /  DB at 100%         => exactly 1 DB query

Recap: stop expiry stampedes with request coalescing (singleflight) plus jittered TTLs and probabilistic early refresh, and handle a genuinely hot key with replication across shards or an L1 near cache, layering the defenses and never treating a cold cache as safe.