← Create New Paste

How to Design a Distributed Message Queue (Like Kafka) in a System Design Interview

3 views
"Design a distributed message queue" is a classic infrastructure question. It tests whether you understand durability, ordering, partitioning and consumer coordination. Here is a structure you can follow. Requirements Clarify throughput (for example one million messages per second), message size (around 1 KB), retention (seven days), and guarantees: is at-least-once delivery acceptable, and does ordering matter globally or only per key? Most real systems promise ordering within a partition and at-least-once delivery by default. Core abstraction: an append-only log Producers append messages to the end of a log; consumers read sequentially from an offset they track. Sequential disk writes and reads are extremely fast, which is why log-based queues handle huge throughput on commodity hardware. Messages are not deleted when consumed; they expire by retention time or size. Topics and partitions A topic is split into partitions so it can scale horizontally. Producers choose a partition by hashing a key (such as user ID), which keeps all events for that key in order. More partitions mean more parallelism but more metadata and slower rebalancing, a trade-off worth naming. Replication and durability Each partition has a leader broker and several followers. Producers write to the leader; followers replicate. A write is acknowledged once it reaches a configurable number of in-sync replicas. Explain the trade-off: acks from all replicas gives durability, acks from the leader only gives lower latency but risks loss if the leader crashes. Consumers and consumer groups Consumers in a group split partitions among themselves, so each partition is read by exactly one consumer in the group. Offsets are committed periodically; if a consumer crashes, another takes over from the last committed offset, which is why duplicates are possible and consumers should be idempotent. Coordination A metadata service tracks brokers, partition leaders and group membership, and runs leader election when a broker fails. Mention that modern systems embed this consensus layer instead of relying on an external coordinator. Extras that impress Batching and compression on the producer, zero-copy transfer from disk to network, tiered storage for long retention, and dead-letter queues for poison messages. Delivering the answer Start with the log, add partitions, then replication, then consumer groups. That order builds naturally. If follow-ups about exactly-once semantics rattle you, Linkjob AI (https://linkjob.ai) offers real-time help during live technical interviews, but practicing this outline aloud a few times is the real foundation.
Created Nov 15, 2026 20:28 Expires Nov 15, 2027 20:28