ScaleCrafts

Design a web crawler

MediumQueuesPolitenessDedup

Download billions of web pages for a search engine, following links while being polite to every site.

Functional requirements

  • Crawl from seed URLs and follow links
  • Store page content for indexing
  • Recrawl pages that change often

Non-functional requirements

  • Never overload a single website
  • Skip duplicate pages and URLs
  • Keep running when machines fail

Scale

  • 1B pages per month
  • About 400 pages per second
  • Average page is 500 KB

How you’ll be scored

11 checks covering your diagram and your notes, and any valid approach counts. Hints appear for anything you miss. Common mistakes, such as single points of failure or components you never explain, cost points. How grading works

What interviewers look for

  • Clarify before you draw. State what is in and out of scope. For “Design a web crawler”, that starts with: Crawl from seed URLs and follow links; Store page content for indexing; Recrawl pages that change often. Interviewers notice candidates who pin the requirements down first.
  • Turn the scale into numbers. The scale (1B pages per month; About 400 pages per second; Average page is 500 KB) should become requests per second, storage and bandwidth, and those figures should drive your choices. Estimates with units carry more weight than adjectives.
  • Show how each quality is met. Your design should visibly deliver: Never overload a single website; Skip duplicate pages and URLs; Keep running when machines fail. Saying that it does isn’t enough; point to the part of the design that makes it true.
  • Expect follow-ups on the hard parts. Interviewers usually probe queues, politeness and dedup here. This is a medium problem, so interviewers expect you to weigh two reasonable options, such as consistency against latency, and to say why you picked one.
  • Explain failure and trade-offs. Say what happens when a component fails, where a single point of failure would hurt, and what you chose not to build. Naming a trade-off is a stronger answer than claiming there isn’t one.

Related problems

Loading your workspace…