Nine of the 30 homepages in the study never produced a page. The scanner asked, and got 401, 403, 429, 202 or 520 back. Those rows are still in the published data, they still carry a score, and that score is a measurement of the error response rather than of the site.
The exclusion is the methodological decision behind the study’s headline number. Count the nine as zero and the average is 44; count them as unmeasured and it is 62. The second is the number this site publishes, and this piece is the argument for it — including the part where it makes the result look less dramatic. It is also a large correction: 30% of the sample never produced a page, and excluding them moves the mean by 18 points, or 29% of the published figure.
Which nine, and what did they answer?
Four status codes and one non-standard code account for all nine. Only two of the nine — 22% — refused the crawler while serving a browser.
| Homepage | Status | Same URL as a browser | What the code says |
|---|---|---|---|
| bmw.com | 520 | 520 | Edge error, not a standard code |
| nytimes.com | 403 | 403 | Forbidden to every client |
| reuters.com | 401 | 401 | Authentication required |
| stackoverflow.com | 403 | 403 | Forbidden to every client |
| perplexity.ai | 403 | 403 | Forbidden to every client |
| google.com | 429 | 429 | Too many requests |
| amazon.com | 202 | 202 | Accepted, no page returned |
| zeit.de | 403 | 200 | Refused the crawler only |
| openai.com | 403 | 200 | Refused the crawler only |
What does each status code actually mean here?
Four of the five codes describe a wall, and one describes a limit on the scanner. They are not interchangeable, and the fix is different for each. 403 is the dominant answer: five of the nine rows (56%) came back forbidden.
- 401 — authentication required. The server wants credentials before it will answer. A paywall or a private origin produces this.
- 403 — forbidden. The server understood the request and refused it. This is the ordinary shape of bot protection, and it is the code that most often hides a fixable configuration.
- 429 — too many requests. This one is about the scanner, not the site: the request rate was limited, so the answer is a limit rather than a judgement of the page. It is published as such rather than dropped.
- 202 — accepted, with no page. A 202 is an acknowledgement, not a document. There is nothing to score, which is why it is flagged as aborted rather than scored as thin content.
- 520 — a non-standard edge error. It is emitted by Cloudflare when an origin returns something the edge cannot use. It is a failure to answer, not an answer.
Why does a 403 get a score at all?
Because the report has to show what happened, and a missing number would hide it. The score is computed from whatever came back, is flagged as aborted, and is excluded from the mean.
This is the same principle as the study’s publish-everything rule. The nine rows are the most interesting part of the sample: they show that 30% of a list of well-known sites will not be read by a crawler on a first request, and that a tool which silently dropped them would report a tidy number about a smaller world.
What is the difference between refused and blocked?
Refused means the same URL served a browser-shaped request and turned away a crawler-shaped one. Blocked means the crawler was not admitted, by arobots.txt rule or by that refusal.
The distinction is worth keeping because it is provable. Zeit and OpenAI are the two rows where the scan can demonstrate it: the identical URL answered 200 to one user-agent and 403 to another. Every other refusal in the table is a wall that applies to everybody, and calling it a crawler block would be an assumption rather than an observation.
How does the exclusion change the study’s average?
It moves it from 44 to 62. Treating the nine as zero adds them to the denominator without adding the 142 points they scored, so the mean falls by 18.
Both numbers are arithmetic on the same table. Only one of them is a statement about websites. A 403 page has no word count, no headings and no entity markup, and scoring it zero says the absence of a document is a badly made document.
What should I do if my own scan comes back with an error?
Read the status before reading the score. A non-200 is a configuration question, and no amount of content work will move it.
- Check whether a bot-protection rule is returning 403 to the user-agent the scanner sends. The crawler guide lists the names.
- If the site is behind a login or a paywall, the scan cannot see it and the number is not about your content.
- If the status is 429, wait and re-run. That one is a limit, and re-running is the whole fix.
- Once a 200 is returned, the score becomes a list of things to change. Until then it is a report about a door.
Why publish the failures instead of only the wins?
Because a sample that only reports what it could read is a sample selected by accessibility, and the selection is the finding. The nine refusals are the evidence that crawler access is not a formality even for famous domains.
It also keeps the method honest in a way a reader can audit: every row, including the ones that make the result worse, is at the study data endpoint, and the rule set that produced the scores is on the methodology page.