Project: a web crawler
Mallow Search asks you to crawl a fictional approved collection of 6,000 hosts and 18 million candidate pages over 100,000 seconds. Deliver requirements, rate estimate, durable frontier model and a checked crawler board. Done means per-host politeness, access checks after redirects, duplicate-safe retry and bounded redirect loops, plus a clear statement when host limits prevent the target rate.
Mallow Search needs a crawler for a fictional collection of public partner sites. Its frontier contains URLs to visit, but several URLs can identify the same page and some hosts respond slowly. An unrestricted worker pool can overwhelm one host while other workers sit idle. Schedule by host and keep the next permitted fetch time explicit.
A crawl is repeatable work. A worker can fetch a page and crash before marking the frontier entry complete, so deduplicate content and keep stable fetch identities. Normalize only the URL components that the site contract treats as equivalent; blindly removing every query parameter can merge different pages. You'll build a durable frontier, enforce per-host limits, and produce a checked crawl pipeline that handles retries, redirect loops and duplicate discovery.
frontier(host, normalized_url, next_fetch_at)
Per-host scheduler -> bounded worker -> object storage -> index events
Build a polite crawler with durable scheduling and duplicate-safe output.