Designing a Distributed Job Scheduler: A System Design Interview Walkthrough
3 views
"Design a distributed job scheduler" is a favorite system design interview prompt because it touches almost every hard topic at once: durable storage, time, coordination, failure handling, and exactly-once semantics that are not really exactly-once. Here is a structured way to walk through it in 45 minutes.
1. Clarify requirements first
Ask what kinds of jobs we support: one-off delayed jobs ("send this email in 2 hours"), recurring cron-style jobs, or both. Ask about scale (thousands of jobs per second, or millions per day?), acceptable delay (is firing 2 seconds late fine?), and delivery guarantees. Most interviewers accept at-least-once execution with idempotent jobs. Write these assumptions down where the interviewer can see them.
2. Core API
Keep it small: schedule(job_spec, run_at or cron), cancel(job_id), get_status(job_id). The job spec contains the target (a queue topic or HTTP endpoint), payload, retry policy, and timeout.
3. Storage model
Store jobs in a durable database keyed by job_id, with a secondary index on next_run_time. For high scale, partition by a hash of job_id, and within each partition keep jobs bucketed by time (for example, one bucket per minute). This makes "find everything due in the next minute" a cheap range scan instead of a full-table query.
4. The dispatch loop
Scheduler workers each own a set of partitions, assigned through a coordination service or a lease table. Every few seconds a worker scans its partitions for jobs due soon, marks them as "claimed" with a lease expiry, and pushes them to an execution queue. Executors pull from the queue, run the job, and write back success or failure. If a worker dies, its leases expire and another worker takes over its partitions.
5. Failure handling
This is where strong candidates separate themselves. Talk about: retries with exponential backoff and a max attempt count; a dead-letter queue for jobs that keep failing; idempotency keys so that a job run twice (because of a lease expiring mid-execution) does not charge a customer twice; and clock skew, which is why you compare against a database or coordinator time rather than each machine's local clock.
6. Recurring jobs
After each successful run of a cron job, compute the next run time and write it back atomically with the status update. Never pre-generate a year of occurrences; it bloats storage and makes cancellation painful.
7. Scaling and hot spots
Popular times like midnight UTC create thundering herds. Mention adding jitter for jobs that do not need exact timing, and rate limiting per tenant so one customer cannot starve everyone else.
8. Wrap up with trade-offs
Summarize: at-least-once plus idempotency, time-bucketed partitions, lease-based ownership. Then name what you would revisit with more time, such as priority lanes or a hierarchical timing wheel for sub-second precision.
Practicing out loud
The hard part in a real interview is not knowing these pieces, it is presenting them in order while someone interrupts with follow-ups. Mock interviews help a lot. Some candidates also use tools like Linkjob AI (https://linkjob.ai), which gives real-time help during live job interviews and technical interviews, to keep their structure on track when the follow-up questions start coming fast.