Six of the 30 homepages in the study are news publishers — 20% of the sample — and every one of them turns at least one tracked AI crawler away. Three of the six disallow the crawlers in robots.txt while still serving the page; a fourth refuses the crawler at the edge while serving the same URL to a browser; and two answer every request, crawler or browser, with 401 or 403.
The news half is also the densest part of a wider finding: 9 of the 30 sites (30%) block at least one tracked crawler, and all six news sites are among them.
The interesting question is not whether they block — they clearly do — but what the block costs them in this measurement and whether that cost is a mistake. For a publisher negotiating a content licence, it may be the price of the negotiation. For a site that has never had that conversation, it is usually just a setting nobody looked at.
Which six publishers, and what did each one return?
Three answered with a 200 and a robots.txt that disallows a tracked crawler. One answered the crawler with a 403 and a browser with a 200. Two answered everything with 401 or 403.
| Homepage | Status returned | Tracked crawler admitted? | Score |
|---|---|---|---|
| bbc.com | 200 | No — robots.txt | 61 |
| spiegel.de | 200 | No — robots.txt | 56 |
| theguardian.com | 200 | No — robots.txt | 47 |
| zeit.de | 403 to the crawler, 200 to a browser | No — refused at the edge | 26 |
| nytimes.com | 403 | No — robots.txt | 12 |
| reuters.com | 401 | No — robots.txt | 12 |
Three of the six decline at the policy layer while still serving a page. Zeit declines at the edge, which is the harder kind: no robots.txt line can express it and no crawler can negotiate it. The last two do not serve a page to anybody, so their scores describe the refusal rather than a page.
What is the difference between a policy block and a refusal?
A policy block is a statement. The page is served, robots.txt says which agents may not use it, and a well-behaved crawler complies.
A refusal is a wall. The server returns 401, 403 or 429 to the request itself, so the crawler never sees the content and never gets to consider the policy. The scanner keeps them as two fields — blocked and refused — because they need opposite fixes. One is a line to edit; the other is a rule in a CDN or a WAF configuration, often in a different system owned by a different team.
Zeit is the clean example of the second kind: the same URL answered a browser-shaped request with a 200 and a crawler-shaped request with a 403. That is a block that happens before robots.txt matters, and it is the one configuration in the sample that can be proved from the outside by changing nothing but the user-agent.
What does blocking cost a publisher in this measurement?
The tracked-crawler rule lives inside AI crawler access, the heaviest dimension at 16% of the score. A site that turns away one tracked crawler cannot pass that dimension in full, so the ceiling on its total drops before content is considered at all. For scale: content depth is worth 11% and trust 10%, so a block costs more than the whole content dimension.
The measured effect is visible: the three publishers that served a page scored 61, 56 and 47, all below the study average of 62 for the sites that answered. That is not proof of causation — these are three large, complex homepages and they fail other checks too — but it is consistent with a penalty that is structural rather than editorial.
Is blocking the crawlers a mistake?
For a large publisher, probably not, and it is not this site’s place to call it one. Six of six is a pattern rather than six coincidences, but a single homepage scan cannot tell a licensing strategy apart from a paywall, a bot-protection rule or a robots file copied from a template.
The trade is legible. Being readable by ChatGPT, Claude, Perplexity and Doubao is how a publisher gets quoted, and a quote is a distribution channel. Turning the crawlers away gives up that channel in exchange for leverage in a licensing negotiation, or in exchange for nothing at all if no negotiation exists. Which of those two situations a given site is in is not something a scan can see.
The distinction that matters for everyone else: a small site that blocks the crawlers is usually not running a licensing strategy. It copied a template, or inherited a WAF rule, or added a blanket bot block during a scraping incident and never revisited the user-agent list. That is the case worth checking, and the robots.txt and WAF guide is the one that walks through it.
What should a site owner check first?
Read robots.txt for the crawler names, then ask whether the edge agrees with it. The two can disagree, and the disagreement is invisible in a browser.
- Look for
GPTBot,ClaudeBot,PerplexityBotandBytespiderinrobots.txt. ADisallow: /under any of them is a policy block. - Check whether a bot-protection rule returns 403 to these user-agents. The scan reports this as a refusal rather than a policy block, which is the only way to tell them apart from outside.
- Decide deliberately. A block you chose and can name is different from one you inherited and cannot explain.
How is this measured, and what can the number not tell you?
The scanner compares two requests to the same URL: one crawler-shaped, one browser-shaped. A difference between them is a refusal; a robots.txtmatch is a policy block; everything else is scored as a page.
What it cannot see is intent or a contract. It does not know whether a publisher has a licensing deal, and it does not fetch the crawler’s own logs to check whether a block was honoured. It measures the public state of the front door, which is the correct scope for a tool that has to work on any domain in one request.