robots.txt tester

Paste a robots.txt and the paths to test. The tester shows if each user agent can fetch each path. Select a result to see the group and the rule that decided it.

A line can be a path or a full URL. The tester reads a URL as its path and its query, and it adds a missing / to a path. For your own user agent, write a token, for example MyBot, or paste a full User-Agent header. The tester then matches the product token, as RFC 9309 specifies.

The tester uses parseRobots and decide from polite-crawler, the same code that the crawler uses before each request. Your text stays in your browser.

A typical siteBlocks AI crawlersRefuses training, permits assistantsWildcards and $Crawl delay and content signals

robots.txt# Keep the search pages and the account pages out of the crawl User-agent: * Disallow: /search Disallow: /account/ Allow: /account/help Sitemap: https://www.example.com/sitemap.xml

Paths or URLs to test, one on each line/ /search?q=bread /account/orders /account/help /blog/robotsYour own user agent: a token or a full User-Agent header

Groups

1

Crawl delay for PoliteCrawler

none

AI input

not stated

Path	MCP serverPoliteCrawler and Claude-User	PoliteCrawler	Claude-User	GPTBot	ClaudeBot	Googlebot
/	✓Permitted	✓Permitted	✓Permitted	✓Permitted	✓Permitted	✓Permitted
/search?q=bread	×Refused	×Refused	×Refused	×Refused	×Refused	×Refused
/account/orders	×Refused	×Refused	×Refused	×Refused	×Refused	×Refused
/account/help	✓Permitted	✓Permitted	✓Permitted	✓Permitted	✓Permitted	✓Permitted
/blog/robots	✓Permitted	✓Permitted	✓Permitted	✓Permitted	✓Permitted	✓Permitted

Select a result to see the rule that decided it.

1. # Keep the search pages and the account pages out of the crawl
2. User-agent: *
3. Disallow: /search
4. Disallow: /account/
5. Allow: /account/help
7. Sitemap: https://www.example.com/sitemap.xml

How a robots.txt decision works

RFC 9309 is the standard for robots.txt since September 2022. The crawler applies these rules:

1. Find the group. A group is one or more User-agent lines and the rules after them. The crawler uses the groups that name its product token. When no group names it, it uses the group *.
2. Merge the groups. When two groups name the same token, their rules apply together.
3. Find the longest match. The rule with the longest path pattern that matches the path wins. When an Allow and a Disallow have the same length, Allow wins.
4. Read the wildcards. A * matches any sequence of characters. A $ at the end of a pattern matches the end of the path.
5. Permit by default. When no rule matches, the crawler can fetch the path. The crawler can always fetch the file /robots.txt.

The match of the token ignores case, and it uses only the product token: Googlebot/2.1 matches the group googlebot.

Agents that act for a person

An AI assistant can fetch a page because a person asked for it. Anthropic names this agent Claude-User, and OpenAI names it ChatGPT-User. These agents are different from the crawlers that collect training data, for example ClaudeBot and GPTBot.

When the crawler fetches a page for an assistant, it obeys two sets of rules. It obeys the rules for its own token, and the rules that a site writes for Claude-User. A site that writes no group for Claude-User gets the rules for the crawler. The column “MCP server” of the tester shows this verdict.

To refuse the training crawlers and let the assistants in, use this robots.txt.

When the crawler cannot read robots.txt

The status of the robots.txt request has a meaning of its own:

Status	Meaning	What the crawler does
2xx	The file is there.	It obeys the rules of the file.
404 or 410	The file is not there.	It fetches all paths.
401 or 403	The file is not available to this crawler.	It fetches all paths, as RFC 9309 specifies. A 401 or a 403 on a page then stops the host for 1 hour.
A challenge page	The site checks for bots.	It stops the host for 24 hours and fetches nothing more.
429	The site limits the rate.	It obeys a good copy of the file from the last 30 days. Without one, it fetches nothing from the site. It tries again after Retry-After, or after 5 minutes to 1 hour.
Another 4xx	The file is not available.	It fetches all paths, as RFC 9309 specifies.
5xx, or no response	The site is unreachable.	It obeys a good copy of the file from the last 30 days. Without one, it fetches nothing from the site. It tries again after 5 minutes to 1 hour.

A crawler that cannot read the rules does not guess them. polite-crawler records the reason, for example a dead domain or a broken certificate.

Content signals

A robots.txt can also state how a site permits the use of its content. Cloudflare publishes the Content-Signal line, for example Content-Signal: search=yes, ai-train=no, ai-input=no. When a page refuses AI input, polite-crawler keeps the page but holds back its text from the AI reader. Test the example “Crawl delay and content signals” above to see the result.