Skip to content

robots.txt answers with an HTML page

robots-txt-not-textminor
On this page

Why it matters

  • Crawlers read robots.txt line by line for User-agent, Allow and Disallow. An HTML page holds none of them, so the site reads as having no rules at all.
  • This usually happens when the site's app answers every address it does not know with its home page, or when the file is missing and the platform sends a "not found" page with status 200.
  • Rules written for AI crawlers or search engines in a file that never arrives are applied by nobody.

How to fix

Put a real text file at the site's root, sent as text/plain:

text
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

In apps that answer every address with the same page (SPA), exclude /robots.txt from that route, or serve it as a static file before it.

How we detect

We read robots.txt once per scan, as we do for the other rules. When it answers 2xx with the type text/html or application/xhtml+xml, the rule fails. We do not judge other types: many servers send it with no type or a generic one, and crawlers read it all the same.