Design a Web Crawler — System Design Interview Walkthrough

A web crawler is a distributed breadth-first traversal of the web graph — and the entire interview is about three constraints that make that BFS hard at scale: politeness (don't hammer a site), deduplication (don't crawl the same content twice), and progress (don't get trapped or starved). Nail the URL frontier design and those three constraints, and the rest is a standard pipeline. This applies the standard system design framework.

1. Requirements

Functional: start from seed URLs, fetch pages, extract links, and keep crawling outward; store page content for downstream use (e.g., a search index); re-crawl for freshness. Out of scope (say so): the search index/ranking itself, JavaScript rendering (mention it's a heavier variant needing a headless browser).

Non-functional, quantified: scalable (billions of pages, finish a full crawl in days); polite (respect robots.txt and per-host rate limits, ~1 request/sec per host); robust (handle timeouts, traps, malformed HTML); fresh (re-crawl important pages by change frequency); and extensible (today you parse HTML, but the parser should be a set of per-content-type modules so images, video, and PDFs slot in later without touching the engine).

Clarify the scope out loud — it's graded in the first three minutes. A crawler's scope is wide open, so pin it: Which content types? (start with HTML, design for more — see extensibility above). Which schemes? (HTTP/HTTPS now; the design generalizes to others). How big? (commit to ~10B pages so every later estimate has something to multiply). Respect robots? (yes). Then name the two properties that make this genuinely hard, because they shape everything downstream: the web is too large to finish (you crawl only a fraction, so you must prioritize) and it changes while you crawl (so freshness is a moving target, not a one-shot job).

2. Scale and the throughput math

To crawl ~10B pages in ~5 days at ~100 KB/page → ~1 PB of raw content (→ object storage). Add ~500 bytes of metadata per page (URL, status, content hash, timestamps) and that's 500 B × 10B ≈ ~5 TB more — a rounding error next to the petabyte. Provision storage and the fleet to ~70% utilization so a burst has somewhere to go (1 PB / 0.7 ≈ 1.4 PB), and budget separately for the in-memory seen-set indexes that dedup needs — sized in Section 6. The interesting number is the rate:

10B pages / (5 × 86,400 s) ≈ ~23,000 pages/sec.

Can a fleet sustain that? Bandwidth is not the limit — the whole crawl pulls only 23,000 × 100 KB ≈ 2.3 GB/s (~18 Gbps) of aggregate download, trivial for a fleet, and a single instance with ~200 Gbps could in principle pull ~250k small pages/sec. But DNS waits, per-host politeness delays, and parsing dominate, so real throughput is a tiny fraction of that — call it a few thousand pages/sec per instance, which puts the crawl at ~10 machines. The crucial insight: a per-host limit of 1 req/sec sounds slow, but there are millions of distinct hosts, so crawling thousands of them in parallel gives huge aggregate throughput. Politeness limits a host, not the crawler. The numbers say storage goes to a blob store, and the frontier and politeness scheduling are the real engineering, not the fetching.

3. Core components & architecture

1. URL Frontier — the heart: enforces priority (crawl important pages first) and politeness (cap requests per host).
2. DNS resolver (+ cache) — turns hosts into IPs; the hidden bottleneck (Section 5).
3. Fetchers — a worker pool that downloads pages (respecting robots.txt + timeouts).
4. Parser / link extractor — pulls out links and content.
5. Dedup — drop already-seen URLs and already-seen content.
6. Storage — raw pages in an object store; metadata in a DB.

4. The crux: the URL frontier

This is where the interview is won. The frontier is two layers of queues:

• Front queues (priority): a URL is assigned a priority (page importance/freshness) and routed to a priority band — higher bands are drained more often.
• Back queues (politeness): one logical queue per host, each with a "next allowed" timestamp. A router maps each URL to its host's back queue, and a selector hands a worker the next back queue whose next-allowed time has passed.

This two-layer design lets you crawl aggressively overall while staying polite per host — naming the front/back-queue split (priority × politeness) is the senior signal. The frontier is distributed (partition hosts across nodes), so no single host's politeness limit throttles the whole crawl.

The frontier lives on disk. Hundreds of millions of pending URLs won't fit in memory, so the queues are backed by disk: each keeps a small in-memory enqueue buffer (flushed to disk when it fills) and a dequeue buffer (refilled from disk in the background), so workers pull from RAM and never block on I/O. It's the same trick a log-structured store uses — batch the random work, keep the hot edge in memory.

BFS by default, DFS within a host. The traversal is breadth-first — breadth surfaces the well-linked, important pages first — with one deliberate exception: once you've opened a connection to a host, crawl several of its URLs depth-first before moving on, so HTTP keep-alive amortizes the TCP+TLS handshake across pages instead of paying it per fetch. Politeness still gates the rate; this just spends each connection well.

5. DNS: the hidden bottleneck

Every fetch must resolve a host to an IP, and a cold DNS lookup costs tens to hundreds of milliseconds. At thousands of fetches a second that is enormous — historically DNS resolution has consumed up to ~70% of a fetch thread's time. So a serious crawler runs an aggressive DNS cache per crawler node (respecting TTLs) plus multiple resolvers to parallelize and avoid being throttled by one. Most fetches then skip the lookup entirely. Mentioning DNS at all separates candidates who have thought about a real crawler from those who haven't.

6. Deduplication (two kinds)

URL dedup. Maintain a "seen URLs" set. Storing billions of URLs exactly is expensive, so a Bloom filter (probabilistic, tiny, allows rare false positives) sits in front of a durable store. The memory math is the senior detail — drag the sliders:

About 9.6 bits per URL gives a 1% false-positive rate, so ~12 GB for 10B URLs — and the baseline that makes that impressive is the exact alternative: storing even an 8-byte checksum per URL (never mind the full text) is ~80 GB, and a real index hundreds of GB to a TB. The cost of the Bloom filter is a rare false positive (occasionally skipping a genuinely new URL), which is an acceptable trade. In front of it, keep a small in-memory cache of the most-popular hosts' membership — a handful of hosts account for a huge share of links, so the hit rate is high.

Content dedup. Different URLs often serve identical content (mirrors, session params, tracking query strings). Hash the page content — or a SimHash for near-duplicates — and skip content you've already stored. Size it like everything else: ~10B documents × an 8-byte checksum ≈ ~80 GB (a 64-bit fingerprint risks a few birthday collisions at 10B docs, so use a ~128-bit one — ~160 GB — if exact dedup matters). Either way it's too big to pin entirely in RAM, so run it as an LRU cache backed by a durable store on disk, hot checksums in memory and the long tail spilled. Normalizing URLs (strip fragments, sort query params) before the URL dedup also kills a huge fraction of duplicates up front.

7. Politeness & robots.txt, concretely

Politeness is more than a per-host timestamp:

• Cache robots.txt per host (it changes rarely) and honor its Disallow rules and Crawl-delay.
• Per-host lock. Back the back-queue with a lock (e.g., a Redis SET key val NX PX ttl) so two workers never hit the same host at once, and so a rate-window reset doesn't cause a thundering herd on that host.
• Store host rules in a Domain table (Section 9) alongside the next-allowed time, so politeness state survives restarts.

8. The fetch loop

1. Pull a ready URL from the frontier (host politeness satisfied).
2. Resolve the host via the DNS cache; check robots.txt (cached); skip if disallowed.
3. Fetch with a timeout; handle non-200s, redirects, and retries with backoff.
4. Store raw content in the object store; record url → contentHash, fetchedAt.
5. Parse links; normalize and run URL + content dedup; enqueue new links with a priority.

Fetch the bytes once. Three stages need the page — the link extractor, the content-hash dedup, and the storage writer — so buffer the downloaded bytes a single time and let all three read from that buffer instead of re-downloading: small pages stay in memory, large ones spill to a temp file. One fetch, many readers.

9. Data model

The seen-set (Bloom filter) is the fast path in front of the URL table; the URL table is the durable record. The Domain table is what makes politeness survive a restart.

10. Partitioning the crawl across machines

One node can't hold the frontier, the seen-sets, or the throughput, so the crawl is partitioned by host: each node owns a slice of the hostname space and runs the frontier, the URL-seen set, and the content-seen set for the hosts it owns. Co-locating all three on the owning node is the point — politeness, URL dedup, and content dedup for a host are consulted together on every fetch, so keeping them on one machine turns three cross-node round-trips into three local lookups.

Map hosts to nodes with consistent hashing (hash(host) → node): adding a node remaps only a slice of hosts, and when a node dies its hosts are reassigned to its neighbors instead of triggering a full reshuffle. Run many logical partitions over fewer physical nodes so you rebalance by moving partitions, not rehashing keys. The shard key is the host, never the URL — sharding by URL would scatter a single host's politeness and dedup state across the fleet, which is exactly what co-location is there to prevent. (This is the same rule as everywhere else: shard by the entity you read together.)

11. Fault tolerance

A multi-day crawl will lose workers, so make progress durable:

• The frontier is a durable queue (e.g., SQS-style) with a visibility timeout: a pulled URL is hidden, not deleted, until the worker acks success — so a crashed worker's URL automatically reappears for another worker.
• Retries with exponential backoff for transient failures (timeouts, 5xx, rate-limit responses), and a dead-letter queue after N attempts so a poison page (malformed, gigantic, or a trap) doesn't block the pipeline.
• Because the URL table records fetchedAt, a full restart resumes from where it left off rather than re-crawling everything.
• That last point is checkpointing by another name. A week-long crawl can't restart from zero, so it snapshots its progress continuously — the durable queue's position plus each node's fetchedAt watermark are the checkpoint. No separate snapshot job: the queue and the ledger already let any node resume from the last known-good point.

12. Trade-offs, follow-ups & what to say at each level

Trade-offs to name unprompted: politeness vs throughput (the back-queue design is this trade — millions of hosts in parallel); crawler traps (infinite calendars, session-id URLs → cap depth/length, normalize URLs, lean on content dedup to escape); freshness vs cost (re-crawl by change frequency, not uniformly); Bloom false positives vs memory; robustness (timeouts, malformed HTML, bans). And name the consistency stance: a crawler is firmly AP — dedup is best-effort (a Bloom false positive or a missed checksum just costs a rare re-fetch), and raw pages sit in an erasure-coded object store, so re-crawlability makes strict durability a non-goal.

Follow-ups a senior interviewer asks next: JavaScript rendering (a headless-browser fetcher fleet for SPA pages), a priority re-crawl scheduler (sitemaps + change-rate estimation), multi-content-type parsing (the per-MIME extractor modules from the extensibility requirement), and monitoring (pages/sec, frontier depth, error rates).

What to say at each level. Mid: the frontier, fetch loop, Bloom dedup, object storage. Senior: the front/back-queue split with priority and politeness, DNS caching, Bloom + content-dedup sizing, the disk-backed frontier, the fault-tolerant queue. Staff+: the throughput math and "politeness limits a host not the crawler," host-partitioning with consistent hashing and co-located seen-sets, checkpointing for week-long resumability, the re-crawl scheduler, and trap handling — unprompted.

Why interviewers use this one

It moves the candidate into distributed scheduling territory — a priority-and-politeness queue, probabilistic dedup, DNS, and fault tolerance — rather than CRUD. It's the same 7-step framework; the crux just shifts to the frontier and dedup instead of the database (as in the URL shortener) or fan-out (as in Instagram).

Amit Singh is a Senior SDE at Amazon, a Claude Certified Architect, and the instructor at AlgoEngineer. Distributed-scheduling problems reward thinking about the constraints, not the happy path. We pressure-test exactly that in the System Design course.