Browse sections
What does this check look at?
robots.txt contains a Content-Signal directive with a value on the same line, for example `Content-Signal: search=yes, ai-input=yes, ai-train=no`. The directive is matched case-insensitively at the start of any line.
What does a failure mean?
No Content-Signal directive in robots.txt.
Weighted 1 of the 16 points in this dimension because adoption is early: the directive was proposed in 2025, and this scanner reads it as a stated position rather than as a signal any engine has committed to. It is scored because the convention is cheap to follow and a tool that recommends it should be able to show its own line, not because it moves a ranking.
How do I fix it?
- Add a line such as: Content-Signal: search=yes, ai-input=yes, ai-train=no. Early adoption and weighted low, but it states your position in the one file every crawler already reads.
Why is it worth 1 point?
Points are relative weights inside a dimension, and a dimension's weight decides how much of the total it can move. The reasoning for this dimension's weight is published rather than asserted:
| Dimension | AI Crawler Access — 16% of the total score |
| Points for this check | 1 of 23 in this dimension |
| Worth at most | 0.7 of the 100 points |
Highest weight of the twelve. A page that is blocked or unreadable has no path to being cited, so it dominates the total.
Which dimension does this belong to?
AI Crawler Access — 16% of the total score, decided by 7 published rules.
If a crawler cannot read the page, nothing else on this list can matter. This is the only dimension where a failure makes the rest of the score moot.
The fix for this rule is walked through step by step in the allow ai crawlers guide.
What else is measured in this dimension?
- robots.txt is served — 2 points
- No AI crawler is blocked — 6 points
- robots.txt declares a sitemap — 1 point
- Homepage returns HTTP 200 — 2 points
- Homepage is marked noindex — 2 points
- sitemap.xml is present and well-formed — 3 points
How do I see my own result?
Run a scan on your own domain. The report lists every failed check with the evidence that produced it, so you can tell a real failure from a check that does not apply.
Run a free GEO audit on your own site →
The full method, including the dimensions that are deliberately weighted low and the parts of a picture a single-URL scan cannot see, is on the methodology page.
Questions about this rule
What does this check look at?
robots.txt contains a Content-Signal directive with a value on the same line, for example `Content-Signal: search=yes, ai-input=yes, ai-train=no`. The directive is matched case-insensitively at the start of any line.
What does failing it mean?
No Content-Signal directive in robots.txt.
What else in this dimension should I check first?
No AI crawler is blocked: At least one AI crawler is disallowed from the site root, or the server refuses identified crawlers outright. sitemap.xml is present and well-formed: No sitemap, or a response that is not valid sitemap markup. robots.txt is served: No readable robots.txt at the domain root. Homepage returns HTTP 200: A non-200 homepage response. The access verdict itself is reported by robots-ai-allowed or robots-ai-blocked, which account for what the refusal was aimed at.
Why does this dimension carry 16% of the score?
A page that is blocked or unreadable has no path to being cited, so it dominates the total.
Evidence and sources
GEO can boost visibility by up to 40% overall in generative engine responses, and in the paper's Table 1 the best of the tested methods improve on the no-optimization baseline by 41% and 28%. — Generative Engine Optimization, KDD 2024
This rule is applied by the scanner and published in the rule set it is read from, so the condition on this page is the condition the scan tests rather than a summary of it.
Primary sources
- Generative Engine Optimization (KDD 2024) — the measured gains the citability weight follows
- What Generative Search Engines Like — what generative engines prefer in retrieved content
- What Gets Cited: Competitive GEO — which of two competing sources is cited first
- The rule set and its weights — every check, its weight and its pass condition