Skip to content

robots.txt tester

Paste a robots.txt file and type a URL, and see whether Googlebot or AI crawlers may fetch it.

  • Paste and test
  • 2 rules
  • Last updated

robots.txt tester

Free, no sign-up, and it runs in your browser: nothing you type leaves it.

What it checks

  • The group that applies to the crawler: the one that names it, or the * group when no group does.
  • The rule that decides for the URL: the longest pattern that matches it, with Allow winning a tie, as Google's parser does.
  • Patterns with * and $: the star for any run of characters, and $ for the end of the URL.
  • The crawlers: Googlebot and Bingbot, AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and training crawlers such as GPTBot and ClaudeBot.

Example

Wrongrobots.txt
# Keep AI away from our content
User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /
Rightrobots.txt
# Keep our content out of AI training only
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /cart/

How to fix

If the file blocks a crawler you want, find the rule and its line as the tester names them, and change it. To exempt a path from a wider block, for one, add an Allow with a longer pattern than the block's, since the longer one wins:

  • To keep your content out of model training but not out of search, name the training crawlers in a group of their own, such as GPTBot, ClaudeBot and Google-Extended, rather than blocking everyone with the * group.
  • To let a crawler through where the * group blocks, write it a group by name: a crawler follows its own group alone and leaves *.
  • Test the URL again after each change, then put the file at your site's root, such as https://example.com/robots.txt.
robots.txt
User-agent: *
Disallow: /account/
Allow: /account/help/

FAQ

No. The tester runs in your browser, and nothing you paste leaves it.

The robots.txt checker fetches the file your site publishes and judges your page by it. This tester works on a file you paste, so you can try your changes before you publish them, on any URL and any crawler.

Not always. It blocks crawling, not indexing: Google may show the URL of a blocked page if it finds links to it, without a description. To keep a page out of the results, use noindex and leave crawling allowed.

Because under RFC 9309 a crawler follows the group that names it when there is one, and does not read the * group with it. So if you write it a group, put in it everything you want to keep from it.

How we measure

Last updated Methodology and scoring

The tester runs in your browser, with the code Arablyzer's robots.txt rules read files with, following Google's open-source parser and RFC 9309. We read the groups and their rules, pick the groups that name the crawler, whatever the letters' case, or the * groups when none does, and merge their rules. Then we match the URL's path and query against the rules' patterns, take the longest pattern that matches, and let Allow win a tie. The /robots.txt file itself is always allowed.