Skip to content
polite-crawler

Fetch once. Read many times.

polite-crawler is a web crawler for research and for AI agents. It obeys robots.txt, sends one request at a time to each site, and keeps each page on your disk. The next question about a page costs no request.
Start in two minutesTry the robots.txt tester
Librarynpm install polite-crawler
Claude Codeclaude mcp add polite-crawler -- npx -y polite-crawler mcp
A crawl of eight sitesThe real scheduler of the package, on a simulated clock at 2× speed.
en.wikipedia.org0 of 5 pages
docs.example.org0 of 4 pages
museum.example0 of 2 pages
api.busy.example0 of 3 pages
news.example0 of 3 pages
shop.example0 of 3 pages
members.examplerobots.txt refuses: no request
blog.example0 of 3 pages

Each site gets one request at a time, and the sites run in parallel.

Requests sent
0
Answered from the store
0
Most requests at once to one site
1
Time on the clock
0.0 s
  • A request
  • A rest
  • A 429
  • A refusal
  • From the store

Why a polite crawler

Section titled “Why a polite crawler”

Research reads the same sources again and again. A plain web fetch goes back to the site each time, and nothing stays. This crawler fetches a page one time, keeps the response, and answers the next questions from a local index. Each site sees one request at a time, from a crawler that tells its name.

It obeys the site

The crawler reads robots.txt as RFC 9309 specifies. It obeys Retry-After, and it stops at a challenge, a 402 or a 403. Test a robots.txt.

It reads the cheap way

The crawler reads the JSON-LD, the visible text and the links with no browser. It opens a headless browser only for a page that needs JavaScript. See what it reads.

It answers from the disk

Each response goes into a capture log. A full-text index finds passages, and a quote check finds the exact text to cite. How it works.

For AI agents

Section titled “For AI agents”

One command gives Claude four tools. The tools fetch pages politely, search the store, read a range of a document and check a quote.

Terminal window
claude mcp add polite-crawler -- npx -y polite-crawler mcp

The same server works in Claude Desktop, Cursor and each MCP client. Set up the MCP server.

As a library

Section titled “As a library”
import { openCrawler } from "polite-crawler";


const crawler = openCrawler({ dir: "./crawl" });
const page = await crawler.fetch("https://en.wikipedia.org/wiki/Web_crawler");


crawler.find("robots exclusion"); // ranked passages, with no request
crawler.verify(page.doc!, "robots.txt protocol"); // the exact span and the capture to cite

Tools to try

Section titled “Tools to try”

These tools run the code of the package in your browser. Nothing goes to a server.

  • robots.txt tester: paste a robots.txt, and find the rule that permits or refuses each path, for each user agent.
  • What the crawler sees: paste HTML, and get the JSON-LD, the visible text and the decision about JavaScript.
  • Opening-hours parser: paste the hours from a page, and get the week that the crawler reads.