LLMention
← All 40 checksAI Context Files

A crawler policy is published

One of 40 checks the scanner runs. This one is worth 1 point inside the AI Context Files dimension, which carries 5% of the total score.

Browse sections

What does this check look at?

robots.txt is readable, so a crawler policy is published. Readable means a body with non-whitespace content: an HTTP 200 with an empty or whitespace-only body publishes no policy and fails this check, exactly as it fails robots-present.

This is the same file the robots-present check reads, scored a second time in a different dimension: once because crawler access is what the policy governs, and once because a machine-readable context file is published. Adding robots.txt therefore moves two checks in two dimensions, and because that makes this the second score for one file, a site that serves an empty one can no longer collect this point for it.

How do I fix it?

  1. Publish robots.txt stating which crawlers are welcome.

Why is it worth 1 point?

Points are relative weights inside a dimension, and a dimension's weight decides how much of the total it can move. The reasoning for this dimension's weight is published rather than asserted:

DimensionAI Context Files — 5% of the total score
Points for this check1 of 6 in this dimension
Worth at most0.8 of the 100 points

Deliberately low. The check is worth 4 of this dimension's 6 points and the dimension carries 5% of the total, so the file is capped at 3.3 of the 100. It is scored because the convention is cheap to follow, not because it moves a ranking.

Which dimension does this belong to?

AI Context Files — 5% of the total score, decided by 3 published rules.

A cheap and optional signal: one URL that lets a site state its own context for machines. Publishing it is easy; whether any engine reads it is not something this method measures.

The fix for this rule is walked through step by step in the llms txt deployment guide.

What else is measured in this dimension?

How do I see my own result?

Run a scan on your own domain. The report lists every failed check with the evidence that produced it, so you can tell a real failure from a check that does not apply.

Run a free GEO audit on your own site →

The full method, including the dimensions that are deliberately weighted low and the parts of a picture a single-URL scan cannot see, is on the methodology page.

Questions about this rule

What does this check look at?

robots.txt is readable, so a crawler policy is published. Readable means a body with non-whitespace content: an HTTP 200 with an empty or whitespace-only body publishes no policy and fails this check, exactly as it fails robots-present.

What else in this dimension should I check first?

A markdown alternate is declared: No markdown alternate is declared on the page.

Why does this dimension carry 5% of the score?

Deliberately low. The check is worth 4 of this dimension's 6 points and the dimension carries 5% of the total, so the file is capped at 3.3 of the 100. It is scored because the convention is cheap to follow, not because it moves a ranking.

Evidence and sources

GEO can boost visibility by up to 40% overall in generative engine responses, and in the paper's Table 1 the best of the tested methods improve on the no-optimization baseline by 41% and 28%. — Generative Engine Optimization, KDD 2024

This rule is applied by the scanner and published in the rule set it is read from, so the condition on this page is the condition the scan tests rather than a summary of it.

Primary sources