robots.txt is a permission file with a yes/no shape: an agent is allowed a path or it is not. Content-Signal is an attempt to say something more specific inside the same file — whether a site wants its content used for search, for AI input, or for training — without pretending those are the same decision. It is a useful line to publish. It is not a gate, and treating it like one is the mistake this article is about.
The scanner carries a one-point rule for it, and the rule page is unusually direct about why: the convention is cheap, adoption is early, and no engine has committed to honouring it. That is a good description of the whole idea.
What the directive looks like
A single line in robots.txt:
Content-Signal: search=yes, ai-input=yes, ai-train=yes
The three names describe different uses. search is indexing and being shown in search results. ai-input is being read as part of an answer — grounding, retrieval, citation. ai-train is training a model on the content. A site can reasonably answer yes to some and no to others; that is the point of a separate line rather than another Disallow.
The scanner’s rule is content-signal, worth 1 point of the 16 in the AI crawler access dimension. It matches the directive case-insensitively at the start of a line and requires a value after the colon. The rule page says what it is reading: a stated position, not a signal an engine has committed to.
What it changes, and what it does not
| Question | Content-Signal | The enforcement |
|---|---|---|
| Can a crawler read it? | Only if its parser knows the directive; unknown lines are ignored by compliant parsers. | User-agent plus Disallow is the mechanism every crawler knows. |
| Does it block a request? | No. | A WAF rule or bot management can refuse at the edge; robots is cooperative. |
| Does it settle training rights? | It states a position; whether it is binding depends on the crawler, the terms and the law. | Terms of service, contracts and applicable law are the actual instruments. |
| Does it affect ranking? | No. | Not applicable — no engine has said it uses the signal for ranking. |
The most useful way to read the directive is as documentation: it tells a reader with the right parser what the site’s position is, in one line, where the crawl rules already live. That is worth something even if enforcement is somewhere else. It is not worth pretending the line is a wall.
Make the line agree with the file
The failure that matters is contradiction. A site that writes ai-train=no while its robots.txt allows GPTBot, CCBot and every other training crawler has created a file that says two different things. The signal is not wrong; the pair is.
This site resolves it in the other direction. Its robots.txt explicitly allows the AI crawlers, and it publishes Content-Signal: search=yes, ai-input=yes, ai-train=yes with a comment explaining the choice: being readable by AI is the product’s argument, and a signal that contradicted the rules would be worse than no signal at all. The consistency is the point, not the value.
If you want to disallow training, the coherent version is both: disallow the training tokens in robots.txt and state ai-train=no. If you want to allow citation but not training, say so and accept that enforcement is partial — different crawlers publish different tokens, and not all of them separate the two uses.
Common mistakes
- Putting it in a comment. A directive after
#is text, not a directive. The scanner matches the start of a line. - Empty value.
Content-Signal:with nothing after it is not a signal; it is a truncated line. - Contradicting the robots rules. The pair has to describe one policy.
- Expecting it to enforce. It does not block, throttle, or opt out by itself. If the content matters, the blocking belongs in robots or at the edge.
- Putting it in a meta tag or a response header. The convention is a robots.txt line; a meta tag with the same words is not read as this directive.
- Copying a value without deciding.
yesto everything is a position; so isno. The line should be the one the site means.
How to check a site's line
# 1. Is the directive present, and what does it say? curl -sS https://your-domain.com/robots.txt | grep -i '^Content-Signal:' # 2. Does the file allow the crawlers the signal talks about? curl -sS https://your-domain.com/robots.txt | grep -iE 'User-agent|Disallow|Allow' # 3. Is the line inside a comment anywhere? (It should not be.) curl -sS https://your-domain.com/robots.txt | grep -n 'Content-Signal' # 4. Follow the scanner's exact test: case-insensitive, start of line, value present curl -sS https://your-domain.com/robots.txt \ | grep -icE '^[[:space:]]*content-signal[[:space:]]*:[[:space:]]*[^[:space:]]'
The fourth command is the mechanical check. The second is the one that catches the contradiction: read the allowed tokens and ask whether the signal describes the same policy.
What the signal is worth
One point out of sixteen in one dimension, which is about right. It is a cheap way to state a position where crawlers already look, and it may become more useful as the convention and the tooling around it mature. It does not replace the rules that actually govern access, and it does not change what an engine does by itself.
The standard is the same as for every other signal in this series: publish only what the site means, make it agree with the rest of the file, and do not let a line of text stand in for enforcement that lives somewhere else.