Design a web crawler

UpdatedAugust 24, 2026

•2 min read•View as Markdown

Danny Crasto

I am developer/code-reviewer/debugger/bug-fixer/architect/teacher/builder from dubai, uae

Part of seriesSystem Design

At the start we seed a few webpage which recursively add more pages from the links found on the seed pages.

Uses Cases

Used for search engine indexing, web archiving, web minining (scraping), web monitoring (copyright checks)

Estimations

To design a crawler, find it's purpose (indexing), pages per month (billion), page type (dynamic, html, pdfs), storage duration (years) and handling duplicate content (keep first).

1 billion pages per month

Throughput: 1 billion queries / 30 days / 24 hours / 3600 secs = ~400 QPS
Storage: 1 billion pages * 500kb / page = 500TB / month * 5 years = 30 PB

Attributes

Crawlers should be robust (flakey servers, malicious links), polite (don't abuse host servers), extensible (support new types)

Workflow

URL frontier

The component that groups and prioritizes urls that need to be processed. To ensure that the crawler respects the server, it will group urls from the same host before pushing them to a worker to process, limiting the number of requests downloaded per host by the processor. It can also reorder urls based on pagerank priority. Links are added to the frontier and processed with a BFS queue as DFS could get too deep. There could be potentially millions of urls to process. To reduce memory requirements most can be kept on disk and retrieved iteratively.

HTML downloader

There could be several which will download the actual file content. It will download the file and store it if new. It will also parse content and extract urls. They will filter out seen urls and restricted paths of the site's robots.txt file. Modern webpages would need to be server-side rendered to extract dynamic content. To reduce DNS lookup which could take between 10-200ms, the IPs are cached. To work with slow pages, a timeout has to be used, as well as handling processing errors gracefully and hashing content to skip duplicates. The state of downloaders is preserved so in the event of a node failure, it can pick up from the last known state.

References

PageRank

This article is part of the system design series where I am summarizing chapters from The System Design Interview: Volume 1 / Volume 2 amongst other related content

#scalability #backend-architecture #design #distributed-systems

75 views

Comments (4)

Join the discussion

Sanoary Istiaque18d ago

I really enjoyed this breakdown of designing a web crawler. I especially liked how it focuses on the practical engineering challenges rather than just explaining the basic concept. The discussion around crawling at scale, handling URLs, and managing resources makes the topic much easier to think about from a real-world perspective. Great read for anyone interested in web infrastructure and backend engineering.

Debjyoti Roy18d ago

I was also working on this topic , this is helpful

Nube Colectiva19d ago

I'm researching this topic and your article is helpful, thank you 👍🏼

OnlineProxy19d ago

the 500tb a month number always bugs me in these writeups because nobody asks what happens after you dedup. if youre storing first copy only and hashing content youre not looking at raw page count times average size, youre looking at unique bodies times size times whatever replication factor you picked, and those two diverge fast once overlapping crawls hit the same articles or product pages from different urls. i've seen real pipelines where content hash dedup alone cuts raw ingest by 30 to 40 percent before compression even touches it. same deal with the qps number, 400 aggregate doesnt tell a worker how many requests it can throw at one host, you need a separate token bucket or delay per domain or you get banned by the first site with any rate limiting at all. treating those two numbers as clean inputs instead of things that need their own logic is where people get tripped up in interviews

System Design

Part 7 of 7

We will run though the design of actual distributed systems and concepts, abridged and modernized. [Photo by Med Badr Chemmaoui via unsplash]

Start from the beginning

DDD: Summary

Here is a summary / cliff’s notes of the Domain Driven Design Quickly eBook Domain is what you trying to encode for which is owned by domain experts/specialist of the area The domain model is an abs