Distributed Rate Limiting

One global limit across N nodes

A single-node limiter is easy because one process owns the counter. Production limiters run on a fleet of gateway nodes behind a load balancer, and a user's requests land on any of them. If each of 20 nodes independently enforces "100 req/min," a user spraying across all 20 gets 2000 req/min, 20x the intended limit. The whole problem of distributed rate limiting is enforcing one global limit across N nodes without paying an unacceptable latency or availability tax.

Approach 1: centralized exact, with a shared store. Put the counter in Redis and have every node read-modify-write it. The trap is the race: two nodes doing GET count; if under limit INCR can both read 99 and both allow, overshooting. You must make the decision atomic. For simple counters, INCR returns the new value in one round trip, so the node increments first and rejects if the result exceeds the limit. For token bucket you need multiple fields updated together (refill then decrement), so you run a Lua script via EVAL, which Redis executes atomically. This gives exact global enforcement. The cost is a network round trip to Redis on every request (0.5 to 1ms added per call) and Redis becoming a hot dependency.

Approach 2: local approximation. Give each node a slice of the budget: 2000/min total across 20 nodes means 100/min per node, enforced purely in local memory with zero coordination. Fast, no shared dependency, but only correct when traffic is evenly balanced. If the load balancer sends a hot user disproportionately to a few nodes, those nodes throttle early while the global budget is underused. It also wastes budget: idle nodes' slices are unusable by busy nodes.

Approach 3: hybrid, the common production answer. Nodes enforce locally from a local token cache for speed, and asynchronously sync their consumption to Redis every short interval (say 100ms) to true up the shared view and re-divide the remaining global budget. This bounds overshoot to at most one sync interval's worth of traffic while keeping the hot path in local memory. Envoy's global rate limiting and many CDNs work roughly this way.

Interview nuance: you will always be asked "what if Redis is down?" A rate limiter must not become a single point of failure for the whole API. The standard answer is fail open: if the shared store is unreachable, fall back to permissive local limits and let downstream load shedding protect the backend. Also handle hot keys (one celebrity key hammering a single Redis slot) with key sharding or local caching, and clock skew / window alignment across nodes: use the store's time or logical windows, not each node's wall clock.

naive per-node (BROKEN)          hybrid (production)
 node1: 100/min                   local cache enforces fast
 node2: 100/min   x20 nodes       async sync -> Redis every 100ms
 ...              = 2000/min       true up + re-divide budget
 user sprays -> 20x limit         overshoot bounded to ~1 interval

Recap: naive per-node limits grant Nx, so either enforce exactly via atomic Redis ops (INCR / Lua, paying a per-request round trip) or approximate locally and async-sync to a shared store for bounded overshoot, and always decide the fail-open path plus hot-key sharding so the limiter never becomes the outage.