Designing a Web Crawler: A System Design Interview Walkthrough
5 views
"Design a web crawler" is a classic system design interview question. It looks simple, fetch pages and follow links, but it quickly turns into a discussion about scale, politeness, deduplication, and fault tolerance. Here is a clear way to structure your answer.
1. Clarify the scope
Ask what the crawler is for: building a search index, monitoring a set of sites, or archiving. Ask about scale (for example, one billion pages per month), whether we handle only HTML or also images and PDFs, and how fresh the data needs to be. A reasonable assumption: about 400 pages per second on average, HTML only, recrawl popular pages more often.
2. High-level components
- Seed URLs and a URL frontier (the queue of URLs to fetch)
- Fetcher workers that download pages
- DNS resolver with caching
- Parser that extracts links and content
- Content deduplication
- URL filter and "seen" set
- Storage for raw pages and metadata
3. The URL frontier is the heart of the design
A naive single FIFO queue fails for two reasons: politeness and priority. Split the frontier into front queues by priority (based on PageRank-like signals or update frequency) and back queues by host, so each host is handled by one worker at a time with a delay between requests. This keeps you from hammering a single site while still keeping all workers busy.
4. Politeness and robots.txt
Fetch and cache robots.txt per host, respect crawl-delay, and identify your crawler with a clear user agent. Interviewers like hearing that you treat politeness as a hard requirement, not a nice-to-have.
5. Deduplication
There are two kinds. URL dedup: normalize URLs (lowercase host, strip fragments, sort query params) and check a seen set; at billions of URLs a Bloom filter in front of a key-value store saves a lot of memory. Content dedup: hash page content (or use SimHash for near-duplicates) so mirrors and session-ID variants are not stored many times.
6. Handling traps and failures
Spider traps, such as infinite calendar pages, can eat your crawl budget. Limit URL length and depth per host. Use timeouts, retries with backoff, and checkpoint the frontier to durable storage so a crash does not lose progress.
7. Scaling out
Partition hosts across crawler nodes with consistent hashing so each host belongs to one node, which keeps politeness local. Store pages in a distributed blob store and metadata in a wide-column database.
8. Freshness
Track how often each page changes and schedule recrawls adaptively: news homepages every few minutes, static docs every few weeks.
Wrapping up
Close by summarizing the trade-offs: priority versus politeness, memory versus accuracy in the seen set, and freshness versus crawl cost. Then mention what you would add with more time, like JavaScript rendering for dynamic pages.
Practice tip
Explaining all of this clearly while an interviewer asks follow-up questions is a skill on its own. Practice out loud with a friend or a timer. Some candidates also use Linkjob AI (https://linkjob.ai), which provides real-time help during live job interviews and technical interviews, to keep their design discussion structured under pressure.