// Distributed Data Building Blocks

Design a web crawler

System Design track · 20 min

By the end of this lesson you can
  • Draw the crawl loop: seeds, frontier, fetchers, parser, dedup, and back to the frontier.
  • Explain why the frontier keeps one queue per website, and how that keeps the crawler polite.
  • Name the three things that break real crawlers (rudeness, traps, crashes) and the defence for each.

Every search engine, price tracker and web archive starts with the same humble program: download a page, read its links, add them to a list, repeat forever. That program is a web crawler.

The idea fits in one line. The engineering is everything around it: billions of pages, websites that will block you if you are rude, and a to-do list that must never forget where it stopped.

Size the crawl first

You start from a few seed URLs: trusted pages full of links. Everything else is found along the way.

Say the goal is one billion pages a month. A month is about 2.6 million seconds, so 1,000,000,000 ÷ 2,600,000 ≈ 400 pages per second, day and night.

One machine fetching 50 pages per second would need 1,000,000,000 ÷ 50 = 20 million seconds, about 8 months, to do one month's work. You need at least 8 machines, and more for headroom. So this is a distributed system from the very first sketch.

The frontier is the heart of the system

The URL frontier is the queue of links waiting to be downloaded. It answers two questions: which page next, and how fast may we hit each website.

Politeness means one queue per website (per host). All the links for one site wait in their own line, and a worker keeps at most one request open per site, with a pause between requests. A site with a thousand pages gets visited steadily, never flooded.

Priority runs alongside: pages that change often or matter more jump ahead. A news homepage beats a forum thread from 2009. One big shared queue would fail at both jobs. It could fire fifty requests at one unlucky site in the same second.

Fetchers, and the DNS you forgot

Fetchers are simple workers that keep no state. Each one takes a link from a site's queue, downloads the page, and passes the bytes on. If one dies, another picks up the work.

The hidden bottleneck is DNS, the service that turns a name like example.com into an IP address. Each new site name means a lookup that can take hundreds of milliseconds, and public DNS servers will slow you down if you flood them. So every fetcher keeps a DNS cache: name to IP, refreshed when it expires. A crawl that touched ten million sites answers most lookups from memory.

robots.txt, the file where a site says what crawlers may visit, gets the same treatment: download it once per site, cache it, and obey it.

Remove duplicates twice, on purpose

The parser pulls out the text and the outgoing links. Then two filters save you from wasted work.

First, a fingerprint of the page: hash the content and compare. Many URLs serve the same page (mirrors, tracking parameters like ?utm=...). If you have seen that hash before, skip storing and processing it again.

Second, the seen-URL set: before a new link joins the frontier, check whether it was already queued. At a billion URLs, a normal hash map would need too much memory. A Bloom filter holds the set in a few gigabytes of RAM. It sometimes says “seen” for a link it never saw. The only cost of that rare mistake is skipping one page, and the web will offer it again later.

Crawler traps are real

Some sites create endless links, on purpose or by accident. A calendar page links to next month, forever. A search page has endless filter combinations. Your crawler will happily fill its frontier with one site's junk while the rest of the web waits.

The defences are boring and they work: a maximum depth from each seed, a page budget per site, and a limit on URL length. On the open web, traps are normal, so plan for them from day one.

The frontier must survive a crash

A month-long crawl will meet dead workers, full disks, and at least one bad deploy. If the frontier lives only in memory, one crash wipes out every link you found but have not fetched yet. That is days of work.

So save the frontier to disk or to a replicated store, and checkpoint progress often: which pages are done and which are in flight. After a crash you resume; you do not restart. Fetchers can die freely. The queue may not.

URL frontierper-host politeness queuesFetcherswith DNS cacheParserextract links + contentDedup filtersURL-seen Bloom + content fingerprintNew linksback to the frontiera loop with no end
Follow the loop: seeds and new links feed the frontier, per-site queues pace the fetchers, and the parser removes duplicates and sends fresh links back to the frontier.
The idea

Take away the scale and a crawler is a queue with good manners. The frontier decides what comes next and how gently to ask. The fetchers are interchangeable muscle. Dedup is two cheap checks that keep the queue clean. Everything hard about this system lives in that queue: priority, politeness and persistence.

Test yourself

Why does the frontier keep a separate queue for each website instead of one big priority queue?

Answer in your head first, then check.

Takeaway

Draw it as a loop: seeds in, a frontier that paces each site, fetchers with a DNS cache, a parser, two layers of dedup, and links back to the frontier. Then spend your time on what actually breaks crawlers: politeness, traps and crashes. That is where you show you think about failure, not just boxes.

Do it this week

Write a fifty-line polite crawler in any language: one seed page, a per-site map that waits one second between requests, a seen-URL set, and a depth limit of three. Point it at a site with a calendar and watch the depth limit save you. That small program holds every idea in this lesson except the scale.

What you can do now
  • Draw the crawl loop: seeds, frontier, fetchers, parser, dedup, and back to the frontier.
  • Explain why the frontier keeps one queue per website, and how that keeps the crawler polite.
  • Name the three things that break real crawlers (rudeness, traps, crashes) and the defence for each.

Distributed Data Building Blocks