Why it matters
- The
noindexrule makes Google drop the page from search results entirely, even when other sites link to it.nonemeansnoindextogether withnofollow. - It can be left over from the development version or come from a content-management setting, such as the "Search engine visibility" option in WordPress's reading settings, which adds
noindexto every page. - The
X-Robots-Tagheader does not show in the page's source, so it can go unnoticed, and it also applies to other files such as PDFs. - If you mean to hide the page, such as a "thank you for your order" page or internal search results, this finding needs no fix.
Example
<meta name="robots" content="noindex, follow" />
<meta name="robots" content="max-image-preview:large" />
How to fix
Remove noindex and none where they come from:
html
<!-- Remove this tag, or keep only its other rules -->
<meta name="robots" content="noindex, follow" />- When it comes from the header, look in the server configuration:
Header set X-Robots-Tagin Apache,add_header X-Robots-Tagin nginx, or your CDN's settings. - In WordPress, untick "Discourage search engines from indexing this site" under "Settings → Reading", and check the SEO plugin's settings for the page itself.
- For Google to see the change, the page must not be blocked in robots.txt, because Google does not read pages it may not crawl.
How we detect
- We check pages that answered with a 2xx status only, whether HTML or other files.
- In HTML we read
<meta>tags namedrobotsorgooglebot, in any letter case, split their rules on commas, and look fornoindexornone. - We read every
X-Robots-Tagheader in the response. A rule preceded by a crawler prefix, such asgooglebot: noindex, only counts when the crawler is Googlebot; rules for other crawlers, such asotherbot: noindex, are ignored. - Each source is its own finding, quoting the value as received.