Design a Distributed Job Scheduler / Cron

Fire each job exactly once despite crashes

A distributed job scheduler fires jobs at their scheduled time (one-off or recurring) across a fleet of workers, and its defining challenge is firing each job exactly once even when workers crash mid-run. This is one of the hardest correctness problems in system design because "exactly once" collides with the reality that any worker can die or pause at any instant. The honest target is effectively-once through idempotency, not literal once-delivery.

Storage and the "due now" query

Jobs have a next-run timestamp, and the scheduler must efficiently find all jobs due in the current window without scanning everything. Index by run time: a database index on next_run_at, or time-bucketed storage where each bucket is a minute or second and workers poll the current bucket. A poller wakes every second, queries WHERE next_run_at <= now AND status = 'pending', and dispatches those jobs. At large scale you shard jobs across many buckets or partitions so no single poller is a bottleneck.

Leasing with a visibility timeout

When a worker picks up a job it does not just mark it running; it acquires a lease: it atomically sets status = running, locked_by = worker, lease_expires_at = now + T in a single conditional update (compare-and-set on status). Only one worker wins the CAS, so only one runs the job. If that worker crashes, its lease expires and the job becomes eligible again, so another worker retries it. Crucially the job is retried, not duplicated, because a live worker holds the lease and a dead one's lease simply expires. This is the same visibility-timeout pattern SQS uses.

Interview nuance: the subtle failure is a paused worker. Suppose a worker acquires the lease, then suffers a long GC pause or network partition past its lease expiry. Its lease expires, a second worker picks up the job and runs it, and then the first worker wakes up and also runs it: a double-run. A lease alone does not prevent this. The fix is a fencing token: each lease grant carries a monotonically increasing token, and any external system the job writes to (or the completion update) rejects a token lower than the highest it has seen. So the resumed old worker's write is fenced off. Bring up fencing unprompted here; it is the senior signal.

Exactly-once framing, clocks, misfires

You cannot guarantee a side effect runs exactly once across crashes, so combine at-least-once execution (retries via lease expiry) with idempotency. Give each job run an idempotency key so that if the job's action is retried, the downstream system dedupes it. Now a double-run produces a single effect.

For a cron job, on completion compute the next run and reschedule. Clock skew across machines means you should not rely on any single worker's clock for correctness; use the database's time or a logical ordering, and tolerate a small firing jitter. If the scheduler was down and missed a window, decide policy explicitly: catch up and run the missed occurrences, or skip to the next future one (misfire policy). Shard jobs by id or by time bucket so many pollers and workers run in parallel, add priority queues, and separate the scheduling tier from the execution tier.

poller: SELECT jobs WHERE next_run_at<=now AND pending
worker: CAS status pending->running, lease_expires=now+T, token=n++   (only one wins)
   crash -> lease expires -> another worker retries (token n+1)
   downstream write carries token; rejects token < max_seen (fencing)
   completion: idempotency_key dedupes the side effect

Recap: index jobs by run time and poll the due window, make a single worker win via a compare-and-set lease with a visibility timeout so crashes retry rather than duplicate, add fencing tokens to defeat the paused-worker double-run, achieve effectively-once with idempotency keys, and handle clock skew and missed windows with an explicit misfire policy.