A crawler that stays on topic.

How it works

1. Seed the queue. Two starting articles go into a FIFO queue and a visited set.
2. Fetch & parse. Pop the next URL, download the HTML, extract the visible page text and every outgoing /wiki/ link.
3. Score relevance. Lower-case the page text and count how many of the related terms appear. Two or more distinct hits → the page is on-topic.
4. Save on-topic pages. Relevant pages are written to the collection along with the terms that matched.
5. Enqueue neighbours. Every valid, not-yet-seen outgoing link is appended to the queue.
6. Repeat until the target page count is reached or the queue drains.

Link filtering skips non-article namespaces (Wikipedia:, Special:, Talk:, Help:, File:, Category:, Template:, Portal:, fragment links, and the Main Page).

The original run — topic

Term coverage in the 2018 crawl

How many of the 462 saved pages mention each term.

Reference implementation

The original crawler is a ~130-line Python script (web_crawler/crawler_doc/webcrawler.py) using urllib + BeautifulSoup. The live demo on this site is a JavaScript re-implementation of the same algorithm that runs entirely in your browser against the MediaWiki API — no backend, which is what lets it host as a static site.

queue = [seed_one, seed_two]
while queue and saved < target:
    page = fetch(queue.pop(0))
    hits = {t for t in related_terms if t in page.text.lower()}
    if len(hits) >= 2:
        save(page, matched=hits)
    for link in page.outgoing_wiki_links:
        if valid(link) and link not in visited:
            queue.append(link); visited.add(link)