AI crawler checker
Does your robots.txt keep AI search crawlers away from your page?
- Checks your page
- Reads:robots.txt
- 1 rule
- Last updated
What we keep
The result appears here
Enter your page’s address and press “Check the page”. This is what the tool runs on it:
- robots.txt blocks AI search crawlersnot started
What it checks
- Whether robots.txt blocks the page for AI search crawlers:
OAI-SearchBotfrom OpenAI,Claude-SearchBotfrom Anthropic andPerplexityBotfrom Perplexity. - Whether the training and user-request crawlers of these companies, and
Google-Extended, are allowed or blocked, as information rather than a problem. - Whether robots.txt answers with a server error (5xx) or cannot be reached, which crawlers that follow RFC 9309 treat as blocking the whole site; we treat 429 the same way, as Google does.
Example
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: PerplexityBot Disallow: /
User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: /
How to fix
Allow the search crawlers, and block the training crawlers if you want to:
A group that names a crawler takes precedence over the * group, so this works even when the file also has User-agent: * with Disallow: /.
# Let AI search cite the site
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# Keep the content out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /FAQ
Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot collect the pages that AI search answers can cite, with a link to the source. GPTBot and ClaudeBot collect content to train models, and Google-Extended is a robots.txt name that controls whether Google uses your content to train its models; it does not affect your presence in Google Search. Blocking training is a legitimate choice that we do not count as a problem.
Your pages may be left out of those services' answers: OpenAI documents that sites blocking OAI-SearchBot are not shown in ChatGPT search answers, and Anthropic and Perplexity document a similar effect on visibility in their results.
No. The tool reads robots.txt only, and a firewall or bot-protection service can block a crawler that robots.txt allows. We fetch the file as ArablyzerBot, so the result describes what we received.
What this tool runs
- robots.txt blocks AI search crawlersmoderate
robots-blocks-ai-search
How we measure
We fetch robots.txt from the page's origin as ArablyzerBot and read it following RFC 9309 with the leniency of Google's open-source parser, up to 500 KiB and following at most 5 redirects. For each crawler on our list we take the group that names it, or the * group when there is none, and the longest rule that matches the page's path decides. Crawler names and purposes come from each company's documentation, and we update them when they change.