Who lets AI agents in?
By Mark Russell. The crawler read the files on . The data is under CC BY 4.0.
On 12 October 2026, polite-crawler read the robots.txt file of each of the top 1,000 domains of the Tranco list. This page shows which sites let 15 agents in, from GPTBot and ClaudeBot to Claude-User and Googlebot. For each verdict, it also shows the rule that decided it.
Of the 1,000 domains, 559 answered with a robots.txt file or with no file. 77 sites blocked the request at the edge, and 364 domains were unreachable. The shares on this page count only the 559 sites that answered.
The findings
Section titled “The findings”- Approximately one site in five refuses the training crawlers. 116 of the 559 sites (20.8%) refuse GPTBot, and 116 sites refuse ClaudeBot. Of the 510 sites with a robots.txt file, 22.7% refuse GPTBot. CCBot and Bytespider get the most refusals: 23.6% and 24%.
- Fewer sites refuse an agent that acts for a person. Of the sites, 14.3% refuse ChatGPT-User and 14% refuse Claude-User. 42 sites refuse GPTBot but let ChatGPT-User in.
- Most refusals name the agent. Of the 116 sites that refuse GPTBot, 84 write a group for GPTBot. The other 32 refuse all bots with the group
*. - 77 sites blocked the request before the file. They gave a 403, a challenge page or a different error. A block does not tell the rules of a site for each agent, so the shares do not count these sites.
- Few sites refuse Googlebot. Only 23 sites (4.1%) refuse it. A site that refuses Googlebot goes out of Google Search.
- Content signals are rare. Only 15 sites write a
Content-Signalline.
- Refused by name84
- Refused, with all bots32
- Permitted by name44
- Permitted399
- Blocked at the edge77
- Unreachable364
GPTBot: 116 of the 559 sites that answered refuse it (20.8%), and 84 of them name it. The share does not count the sites that blocked the request or were unreachable.
The refusals for each agent
Section titled “The refusals for each agent”A site can refuse an agent in two ways. It can write a group for the agent, for example User-agent: GPTBot with Disallow: /. It can also refuse all bots with the group *. Then each agent without a group of its own gets that refusal.
- Refused by name: a group names the agent
- Refused with all bots: the group
*
Show the numbers as a table
| Agent | Company | Refused by name | Refused with all bots | Permitted by name | Of 559 that answered | Of 510 with a file |
|---|---|---|---|---|---|---|
GPTBot | OpenAI | 84 | 32 | 44 | 20.8% | 22.7% |
ClaudeBot | Anthropic | 82 | 34 | 39 | 20.8% | 22.7% |
Google-Extended | 72 | 31 | 40 | 18.4% | 20.2% | |
CCBot | Common Crawl | 92 | 40 | 21 | 23.6% | 25.9% |
Applebot-Extended | Apple | 63 | 35 | 21 | 17.5% | 19.2% |
Bytespider | ByteDance | 94 | 40 | 11 | 24.0% | 26.3% |
meta-externalagent | Meta | 69 | 40 | 13 | 19.5% | 21.4% |
OAI-SearchBot | OpenAI | 37 | 34 | 51 | 12.7% | 13.9% |
Claude-SearchBot | Anthropic | 38 | 38 | 27 | 13.6% | 14.9% |
PerplexityBot | Perplexity | 63 | 31 | 52 | 16.8% | 18.4% |
ChatGPT-User | OpenAI | 46 | 34 | 49 | 14.3% | 15.7% |
Claude-User | Anthropic | 41 | 37 | 24 | 14.0% | 15.3% |
Perplexity-User | Perplexity | 37 | 44 | 21 | 14.5% | 15.9% |
Googlebot | 6 | 17 | 76 | 4.1% | 4.5% | |
PoliteCrawler | polite-crawler | 0 | 47 | 0 | 8.4% | 9.2% |
The training crawlers get the most refusals. 59 sites refuse all seven of them. Google-Extended and Applebot-Extended are not crawlers: they are tokens. Googlebot and Applebot fetch the pages, and a group for the token tells Google or Apple not to use the pages to train its models.
The crawlers for AI search get fewer refusals than the training crawlers. A site that refuses a search crawler goes out of the answers of that search.
PoliteCrawler is a crawler that no site names. It thus gets the rules of the group *: 47 sites (8.4%) refuse all bots that they do not name.
One company, three agents
Section titled “One company, three agents”OpenAI and Anthropic each have three agents. One crawler collects training data, and one crawler collects pages for search. The third agent fetches a page when a person asks for it. Perplexity has a search crawler and an agent for a person.
- Training crawler
- Search crawler
- Acts for a person
Show the numbers as a table
| Company | Training crawler | Search crawler | Acts for a person |
|---|---|---|---|
| OpenAI | GPTBot 20.8% | OAI-SearchBot 12.7% | ChatGPT-User 14.3% |
| Anthropic | ClaudeBot 20.8% | Claude-SearchBot 13.6% | Claude-User 14.0% |
| Perplexity | – | PerplexityBot 16.8% | Perplexity-User 14.5% |
For each company, the sites refuse the agent for a person less than the training crawler. 42 sites refuse GPTBot and let ChatGPT-User in. 42 sites refuse ClaudeBot and let Claude-User in. These sites refuse the use of their pages for training, but they let an assistant read a page for a person.
The MCP server of polite-crawler obeys the rules for Claude-User when it fetches a page for an assistant. Read how. To write such rules for your own site, use the recipe.
What the 1,000 domains answered
Section titled “What the 1,000 domains answered”The Tranco list ranks domains, not websites. Many of the top domains serve a content delivery network, the DNS or an API, and have no website. 227 of the domains have no address in the DNS.
- A robots.txt file510
- The verdicts come from its rules.
- No file49
- A 404 or a 410, or an HTML page in place of the file. The site states no rules, so each agent is permitted.
- Blocked at the edge77
- The server answered with a refusal, a challenge or an error in place of the file. The rules of the site are not known.
- Unreachable364
- The crawler got no file that it could read: no address in the DNS, a TLS error, a timeout, a redirect loop or a body that it could not decode.
The study puts each answer into one of four states:
- A robots.txt file (510 sites). The verdicts come from the rules of the file.
- No file (49 sites). These sites gave a 404 or a 410, or an empty file. They state no rules, so each agent is permitted.
- Blocked at the edge (77 sites). The server answered with a refusal, a challenge or an error in place of the file. 55 sites gave a 403, and 23 gave a challenge, for example a CAPTCHA page. The other sites gave a different 4xx, a 429 or a 5xx.
- Unreachable (364 domains). The crawler got no file that it could read. 227 domains have no address in the DNS. In place of the file, 50 sites gave an HTML page (a home page or a sign-in page), so the crawler fetches nothing there. The other domains gave a TLS error, a timeout or a closed connection. 4 sent more than 5 redirects, and 2 sent a body that the crawler could not decode.
The shares on this page count the files and the sites with no file: 559 sites. They do not count the blocked or the unreachable domains.
A 4xx and a block are two different questions
Section titled “A 4xx and a block are two different questions”RFC 9309 tells a crawler what to do when it cannot read robots.txt. A 4xx means that the file is not available, so a crawler can fetch each path. A 5xx or no answer means that the site is unreachable, so a crawler fetches nothing.
This study asks a different question: what does a site tell each AI agent? A 403 to the request of the study does not answer that question. The edge of the site refused one request before it gave the file, so the rules for GPTBot stay unknown. Thus the study counts only a 404 and a 410 as “no file”. It counts each other 4xx as a block, not as a site that permits each agent.
The crawler itself also stops at a 403. After a 403 or a 401, it sends no more requests to the site for 1 hour, and after a challenge for 24 hours. The stop stays when the crawler starts again.
Content signals
Section titled “Content signals”A robots.txt file can also tell how the site permits the use of its content. Cloudflare published the Content-Signal line in September 2025, for example Content-Signal: search=yes, ai-train=no. Only 15 of the 559 sites write it:
| Signal | yes | no | Different in two groups |
|---|---|---|---|
search: show the site in search results |
13 | 0 | 2 |
ai-input: use the pages as input to an AI answer |
10 | 2 | 2 |
ai-train: use the pages to train a model |
7 | 6 | 2 |
When a page refuses AI input, polite-crawler keeps the page but holds back its text from the AI reader. Test a robots.txt with signals.
The method
Section titled “The method”- The list. The study used the top 1,000 domains of the Tranco list
4672X, from 2026-10-11. Tranco makes the list from the ranks of five sources across 30 days. - The requests. The crawler sent one request for
https://<domain>/robots.txtto each domain, and followed a maximum of 5 redirects. It did not try again, and it did not trywww.<domain>. All requests went out from one computer on 12 October 2026, with a new, empty store. - The User-Agent. Each request sent
PoliteCrawler/0.1.0 (+https://mark1russell7.github.io/crawler/bot/), with no contact. The address in it goes to the page for site owners. - The verdicts. The function
decideof polite-crawler decided each agent alone, as RFC 9309 specifies. The path was/polite-crawler-study/a-page, a page path that no site names. Thus a refusal of this path is a refusal of the pages of the site. - Redirects. RFC 9309 lets a crawler read more than 5 redirects as “no file”. The study, as the crawler, reads them as unreachable.
- One file for two domains. Two domains can redirect to the same file. The crawler then asks for the file a second time with its
ETag, and the server can answer304 Not Modified. The study uses the stored copy of the file, as the crawler does.
The limits
Section titled “The limits”- Edge blocking. A block can depend on the details of the request, for example the TLS connection of the client, and not only on the User-Agent. A different client can get the file. Some unreachable domains closed the connection, and this can also be a block.
- One User-Agent. Some sites give a different robots.txt to a different User-Agent. The study sent only the User-Agent of polite-crawler, not the User-Agent of each agent.
- One try. The crawler sent one request to each domain and did not try again. A timeout or a closed connection on that day puts a domain in “Unreachable”.
- Content signals. A file can write a different
Content-Signalline for each group. The table counts such a site in “Different in two groups”, not by its first line. - Domains, not websites. The list includes domains for content delivery networks, APIs and the DNS. Such a domain with no file counts as a site that states no rules.
- A snapshot. A site can change its robots.txt at any time, and some sites give a different file to different countries.
- A request only. A robots.txt does not stop a crawler that ignores it.
Get the data
Section titled “Get the data”- The dataset as JSON: each domain, its state, its verdicts and the lines that decided them.
- The dataset as CSV: each domain, its state and its verdicts, with one column for each agent.
- The script that sent the requests and wrote the dataset.
The data and the text of this study are under the CC BY 4.0 license. You can use them for any purpose when you name the source. The code of polite-crawler and the script are under the MIT license.
How to cite
Section titled “How to cite”Mark Russell. “Who lets AI agents in? The robots.txt rules of the top 1,000 domains.” polite-crawler field notes, 12 October 2026. https://mark1russell7.github.io/crawler/notes/robots-txt-ai-agents/. Data: CC BY 4.0.
Find a site
Section titled “Find a site”1,000 domains. Select a domain to see the lines that decided.
| Rank | Domain | GPTBot | ClaudeBot | Google-Extended | CCBot | Applebot-Extended | Bytespider | meta-externalagent | OAI-SearchBot | Claude-SearchBot | PerplexityBot | ChatGPT-User | Claude-User | Perplexity-User | Googlebot | PoliteCrawler |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | |
| 2 | ✓Permitted by name | ·Permitted | ✓Permitted by name | ✓Permitted by name | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ✓Permitted by name | ✓Permitted by name | ·Permitted | ·Permitted | ·Permitted | ·Permitted | |
| 3 | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | |
| 4 | ✓Permitted by name | ✓Permitted by name | ✓Permitted by name | ✕Refused, with all bots | ✓Permitted by name | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✓Permitted by name | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✓Permitted by name | ✕Refused, with all bots | |
| 5 | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | |
| 6 | Unreachable: no answer | |||||||||||||||
| 7 | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | |
| 8 | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | |
| 9 | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | |
| 10 | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | |
| 11 | ✓Permitted by name | ✓Permitted by name | ✓Permitted by name | ✕Refused, with all bots | ✓Permitted by name | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✓Permitted by name | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✓Permitted by name | ✕Refused, with all bots | |
| 12 | ✕Refused by name | ✕Refused by name | ✕Refused by name | ✕Refused, with all bots | ✕Refused by name | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused by name | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused by name | ✕Refused, with all bots | |
| 13 | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✓Permitted by name | ✕Refused, with all bots | |
| 14 | Unreachable: no answer | |||||||||||||||
| 15 | Unreachable: no answer | |||||||||||||||
| 16 | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused by name | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused by name | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✕Refused, with all bots | ✓Permitted by name | ✕Refused, with all bots | |
| 17 | Unreachable: no answer | |||||||||||||||
| 18 | Blocked at the edge: the rules are not known | |||||||||||||||
| 19 | Unreachable: no answer | |||||||||||||||
| 20 | Unreachable: no answer | |||||||||||||||
| 21 | Unreachable: no answer | |||||||||||||||
| 22 | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | ·Permitted | |
| 23 | Unreachable: no answer | |||||||||||||||
| 24 | Unreachable: no answer | |||||||||||||||
| 25 | Unreachable: no answer | |||||||||||||||