Why it matters
- Googlebot is the crawler Google uses to read your pages before showing them in search results. When robots.txt blocks it from a page, Google cannot read the page's content; the page may appear as a bare link without a description, or not at all.
- A single line such as
Disallow: /blocks the whole site, and it can be left over from the development version after launch. - When robots.txt answers with a server error (5xx) or 429, Google temporarily stops crawling the whole site until it can read the file again.
- robots.txt is not a way to hide a page from search results; use
noindexfor that and leave the page open to crawling.
Example
# Keep Google out while the new store is being built User-agent: Googlebot Disallow: /
User-agent: * Disallow: /admin/
How to fix
Remove the blocking rule, or narrow it to what you really want to block:
text
# Before: blocks the whole site
User-agent: *
Disallow: /
# After: blocks only the admin area
User-agent: *
Disallow: /admin/- Make sure
/robots.txtanswers 200, or 404 if you do not need one, and never a 5xx error. - After the change, check the robots.txt report in Google Search Console.
How we detect
- We fetch
/robots.txtfrom the origin of the final URL after redirects, following up to 5 redirects and reading the first 500 KiB, as Google does. - We parse it as RFC 9309 describes, with the tolerance of Google's open-source parser for common misspellings of field names and a missing colon.
- We pick the Googlebot group, or the
*group when there is none for Googlebot, merge groups with the same name, and apply the longest matching rule; on a tie,Allowwins.*and$are supported. - A 5xx or 429 answer, or a failed connection, means the whole site is blocked. A 4xx answer, such as 404, means no restrictions.
We fetch robots.txt as ArablyzerBot, not as Googlebot; if your server or firewall answers Googlebot differently, the result reflects what we received.