Skip to content

Content signals in robots.txt cannot be read

content-signals-invalidminor
On this page

Why it matters

  • Content signals (Content-Signal) are robots.txt lines that tell whoever collects pages how they may be used: in search, as input to AI answers, or to train models. The IETF's AI preferences drafts define a similar line, Content-Usage.
  • A reader of these lines ignores what it does not understand: a misspelled key, a value other than the two, a line outside any group. Your preference is then as good as unwritten.
  • Allowing search for a group that is kept from the whole site is a contradiction that confuses whoever reads the file.

Example

Wrong
Content-Signal: search=yes, ai-trian=no
Right
Content-Signal: search=yes, ai-input=yes, ai-train=no

How to fix

Put the line inside a User-agent group, with the defined keys and values:

text
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
  • Content-Signal's keys: search, ai-input and ai-train, with the values yes or no.
  • Content-Usage's keys in the IETF draft: search, ai-use and train-ai, with the values y or n.
  • A path may come before the pairs: Content-Signal: /blog/ ai-train=no.

How we detect

We read the Content-Signal and Content-Usage lines of robots.txt with the group they are in, and check each key=value pair against its line's vocabulary. A line before the first User-agent is outside every group. We do not count ai-train=yes as contradicting a block of the site: training may use copies collected before.