Quarry Cloud asks for a scheduler that runs 4,320,000 jobs a day for its customers, with a 20x spike at the top of every hour. Deliver the scope, a peak dispatch estimate, a job API, a run table, the dispatch path, a fence for stale workers and a plan for the hourly spike.
How interviewers level this design
- Junior: stores due runs durably and dispatches them through a queue to workers.
- Mid-level: claims each run once with a uniqueness constraint, uses leases and at-least-once delivery, and retries with backoff.
- Senior: fences stale workers, flattens spikes, isolates noisy customers and explains why exactly-once effects need idempotent handlers.
Quarry Cloud runs millions of scheduled jobs a day for its customers: nightly exports, hourly syncs, reminders. Customers expect a job to start close to its scheduled time, to be retried when it fails and never to run twice at the same moment. Plenty of jobs are scheduled for the top of the hour, so load arrives in sharp spikes.
The core of the design is a durable list of due runs, a claim step that only one dispatcher can win, and workers that hold a lease while they run. Delivery is at least once, so job handlers must be safe to repeat, and a worker that freezes past its lease must not keep writing. You will estimate the spike, define the API and the run table, draw the dispatch path and close the two classic failure windows.
Claim each run once, deliver at least once with leases, fence stale workers and flatten top-of-hour spikes.