What does a robots.txt disallow actually do?
It tells a named crawler not to fetch the page. That is the whole rule, and it is all this page reports: whether the site chose it deliberately, inherited it from a template, or got it from a CDN default is not recorded in robots.txt.
“This document specifies and extends the ‘Robots Exclusion Protocol’ … allowing a website owner to control how automated clients interact with the site.” — RFC 9309, Robots Exclusion Protocol
Nor does a disallow make a site invisible in AI answers. Engines can quote a page they read elsewhere, and a site can block one crawler while admitting another. What the rule removes is the direct path: the page cannot be the thing the model read for itself. The mechanics are the ones documented by Google’s crawler documentation and by each provider’s own bot pages.
One pattern is checkable in the table: the blocked sites have a median score of 46, against 53 for the benchmark as a whole, and 35 of 48 graded F. A disallow is an access failure, and access is the first dimension in the score - but a high score elsewhere does not cancel the failure.
Which AI-tool sites are on the list?
Every blocked domain in the 620-site sample is in the table below, with the score its homepage received. The list is short enough to read in full, and each domain links to a fresh scan of the same page.
How was this measured, and how do I correct a row?
One scan of each homepage on 2026-11-04, using the same 40 checks and the same robots.txt evaluation as the free scan. A domain is listed when the robots.txt at its root disallowed at least one of the five tracked crawlers.
The sample is the one described on the benchmark page: 1,000 homepages drawn from a public AI-tools directory with a fixed seed, one scan per domain on 2026-11-04, scored with the same 40 checks the free scan uses. “Blocked” means the robots.txt served at the root disallowed at least one of the five crawlers listed above, as evaluated by the robots-ai-blocked check. The raw data is the benchmark CSV.
A domain on this list that has since changed its robots.txt is a stale row rather than a false one: the measurement records what was served on the date above. Send the domain and the current rule through the contact page and it will be re-checked in the next run.