"The AI crawler is blocked" is three different findings wearing one sentence. A robots.txt rule says no. A server can refuse the request before the file is read. A response can fail for reasons that have nothing to do with either. The symptom — the engine does not cite the page — is identical; the fix is different in each case, and applying the wrong one is the most common waste in this category.
This article is about the vocabulary the scanner uses, why the distinctions survive into the score, and how to tell which one you are looking at before editing anything.
Three words, three findings
| Finding | What happened | Where it is fixed |
|---|---|---|
| Blocked | A robots.txt group disallows the token from /, or the server refuses the crawler. | The robots file for the policy case; the edge or application for the refusal case. |
| Refused | A crawler-shaped request is rejected while the same URL serves a browser-shaped request. | A WAF rule, bot management, a rate limit or a user-agent rule — never robots. |
| Aborted | The response is not a 200, so any score describes that response rather than the page. | Whatever produced the status: the origin, the edge, or the scan itself. |
The words overlap in casual use and do not overlap in the scanner. Blocked is the umbrella the score uses for "this crawler cannot read the site, by policy or by refusal". Refused is the narrower and more interesting case: the refusal happens before robots.txt is consulted, so no robots edit can change it, and it catches every engine because they all identify as bots. Aborted says the number is about an error response, not a page — the reason the 30-site study reports nine rows separately instead of counting them as zero.
What the 30-site sample found
The published dataset is small and specific: 30 homepages, one scan, 3 October 2026. In those rows, nine carry blocked: true. Of those nine, seven are robots.txt policy — a group that disallows an AI token — and two are server refusals, where the crawler-shaped request was rejected and the browser-shaped request to the same URL was served. Nine rows are also aborted, a non-200 response, and the two sets overlap.
The overlap is the lesson. A single row can be blocked by policy and aborted at the same time; the count of blocked rows is not the count of robots.txt problems, and the count of aborted rows is not the count of sites that hate crawlers. The dataset keeps the three booleans separate precisely so a reader can ask which mechanism produced the finding.
Why the difference changes the score
The access rule is robots-ai-allowed, worth 6 points. If a token is disallowed, the rule fails. If the server refuses identified crawlers while serving browsers, the rule fails under the companion name robots-ai-blocked, so the blocked agents appear in the finding. But if both request shapes are refused, the refusal is about the scanning address rather than the site, and the rule scores 2 of 6 as unverified instead of failing.
That third branch exists because the alternative is unfair: a scanner that cannot reach a site from its own network should not report the site as hostile to AI. The cost is that "unverified" is not a pass either — the page has not been shown to work, and a reader should treat it as an open question, not a green tick.
How the wrong diagnosis happens
| What you see | What it is often mistaken for | What it actually is |
|---|---|---|
| 403 to a crawler, 200 to a browser | "robots.txt is blocking us" | A refusal before robots is read; edit the WAF, not the file. |
| A robots Disallow, requests 200 | "the WAF is blocking AI" | A policy block; the server is happy to serve, the file says no. |
| Both shapes refused | "the site blocks AI" | Possibly a geo/IP or rate-limit policy; test from another network. |
| 200 with an empty body | "the crawler is blocked" | Client-rendered content; the response arrived, the text did not. |
The third row is the one that produces the 2-of-6 unverified verdict. The fourth is a different article: a 200 is not evidence that the page was read.
Telling them apart, in order
# 1. Read the policy curl -sS https://your-domain.com/robots.txt # 2. Crawler-shaped request curl -sS -o /dev/null -D - -A "GPTBot" https://your-domain.com/ # 3. Browser-shaped control curl -sS -o /dev/null -D - -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/126.0 Safari/537.36" https://your-domain.com/ # 4. If both are refused, repeat from another network before concluding anything
Read the results in this order. If the file disallows the token, the finding is policy, even if the request also happens to be refused. If the file allows it and the crawler is refused while the browser is served, the finding is a refusal. If both are refused, the finding is about the network until proven otherwise.
What not to conclude
A 403 is not evidence of a deliberate AI policy. A missing robots.txt is not permission. A passed robots check is not proof that an engine can read the page. The scanner publishes these sentences because the inverse mistakes are common and expensive: a team edits a robots file that was never consulted, or removes a WAF rule that was doing its job, because the finding was not read carefully.
The vocabulary is the whole point. Once the finding has a name, the fix has a location.