robots.txt checker
Does your robots.txt keep Google's crawler away from your page?
- Checks your page
- Reads:robots.txt
- 1 rule
- Last updated
What we keep
The result appears here
Enter your page’s address and press “Check the page”. This is what the tool runs on it:
- robots.txt blocks Googlebot from the pagenot started
What it checks
- Whether robots.txt blocks
Googlebotfrom the page: we read the group that names it, or the*group when there is none, and the longest rule that matches the page's path decides. - The rule that blocks it and its line in the file, such as a
Disallow: /left over from the development version after launch. - Whether robots.txt answers with a server error (5xx) or 429, or cannot be reached, which Google temporarily treats as blocking the whole site.
Example
User-agent: * Disallow: / Sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
How to fix
Remove the blocking rule, or narrow it to what you really want to block, and leave the rest of the site open to crawling:
- Make sure
/robots.txtanswers 200, or 404 if you do not need one, and never a 5xx error. - After the change, check the robots.txt report in Google Search Console.
# Blocks only the admin area
User-agent: *
Disallow: /admin/FAQ
No. robots.txt stops Googlebot from reading the page, but the page can still appear in results as a bare link without a description. To keep a page out of results, use noindex and leave the page open to crawling, because Google does not see a noindex on a page it may not crawl.
That is fine. When /robots.txt answers 404, or another 4xx error other than 429, there are no crawling restrictions, and the tool passes. A 5xx error or 429, on the other hand, makes Google temporarily stop crawling the whole site until it can read the file again.
No. When a group names Googlebot, it follows that group's rules alone and leaves the * group aside; when its group appears more than once in the file, their rules are merged. So Googlebot alone can be blocked while the * group looks fine.
What this tool runs
- robots.txt blocks Googlebot from the pagecritical
robots-blocks-googlebot
How we measure
We fetch the page as ArablyzerBot and follow its redirects, then fetch /robots.txt from the origin of its final URL, following at most 5 redirects and reading its first 500 KiB, as Google does. We parse it as RFC 9309 describes, with the tolerance Google documents, such as common misspellings of field names, then apply one rule: we take the Googlebot group or the * group, and the longest rule that matches the page's path decides; on a tie, Allow wins. A 4xx answer, such as 404, means no restrictions; a 5xx or 429 answer, or a failed connection, means the whole site is blocked. Because we do not fetch the file as Googlebot, if your server or firewall answers Googlebot differently, the result describes what we received. The same file gives the same result on every check.