Why it matters
- Content signals (Content-Signal) are robots.txt lines that tell whoever collects pages how they may be used: in search, as input to AI answers, or to train models. The IETF's AI preferences drafts define a similar line, Content-Usage.
- A reader of these lines ignores what it does not understand: a misspelled key, a value other than the two, a line outside any group. Your preference is then as good as unwritten.
- Allowing search for a group that is kept from the whole site is a contradiction that confuses whoever reads the file.
Example
Content-Signal: search=yes, ai-trian=no
Content-Signal: search=yes, ai-input=yes, ai-train=no
How to fix
Put the line inside a User-agent group, with the defined keys and values:
text
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /- Content-Signal's keys:
search,ai-inputandai-train, with the valuesyesorno. - Content-Usage's keys in the IETF draft:
search,ai-useandtrain-ai, with the valuesyorn. - A path may come before the pairs:
Content-Signal: /blog/ ai-train=no.
How we detect
We read the Content-Signal and Content-Usage lines of robots.txt with the group they are in, and check each key=value pair against its line's vocabulary. A line before the first User-agent is outside every group. We do not count ai-train=yes as contradicting a block of the site: training may use copies collected before.