Why it matters
- Crawlers read robots.txt line by line for
User-agent,AllowandDisallow. An HTML page holds none of them, so the site reads as having no rules at all. - This usually happens when the site's app answers every address it does not know with its home page, or when the file is missing and the platform sends a "not found" page with status 200.
- Rules written for AI crawlers or search engines in a file that never arrives are applied by nobody.
How to fix
Put a real text file at the site's root, sent as text/plain:
text
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlIn apps that answer every address with the same page (SPA), exclude /robots.txt from that route, or serve it as a static file before it.
How we detect
We read robots.txt once per scan, as we do for the other rules. When it answers 2xx with the type text/html or application/xhtml+xml, the rule fails. We do not judge other types: many servers send it with no type or a generic one, and crawlers read it all the same.