Cloudflare WAF and AI crawlers: what a 403 actually means

A 403 is the most misread status code in this category. It gets filed as "robots.txt blocked the AI crawler", the robots file gets edited, and nothing changes — because the request was refused before the file was ever read. The two failures look identical from the outside: the engine does not cite the page. They have different causes, different fixes, and different owners.

This article is about telling them apart when a CDN or WAF sits in front of the origin, with Cloudflare as the worked example because it is the common case for sites like this one. The method generalises: make the same request twice with different User-Agents, read the response headers, and only then decide what to change.

Where a 403 can come from

A request to a site behind a CDN passes through several layers before your application runs. Each layer can answer with 403, and each has a different fix.

The practical question is not "which layer is it" at first — it is "is the refusal aimed at the crawler, or at the request in general". That question has a one-command answer.

The one test that separates crawler-targeted from general refusal

Send the same request twice: once with the crawler’s User-Agent, once with a browser User-Agent. Keep everything else identical, including the path and any cookies you have.

# crawler-shaped request
curl -sS -o /dev/null -D /tmp/crawler.headers \
  -A "GPTBot" https://your-domain.com/

# browser-shaped control
curl -sS -o /dev/null -D /tmp/browser.headers \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
      (KHTML, like Gecko) Chrome/126.0 Safari/537.36" https://your-domain.com/

# compare
grep -iE "^HTTP/|^cf-mitigated:|^server:|^cf-ray:" /tmp/crawler.headers
grep -iE "^HTTP/|^cf-mitigated:|^server:|^cf-ray:" /tmp/browser.headers

Reading it:

• 200 / 200. The crawler is not refused. If it still is not cited, the cause is elsewhere — parsing, content, or the engine’s own choices — and this is the wrong file to edit.
• 403 / 200. The refusal is aimed at identified crawlers. That is a WAF, bot-management or user-agent rule, not robots.txt. The fix is an exception for the crawler tokens you want to admit, and it has a real security cost that should be weighed.
• 403 / 403. The request is refused regardless of its identity. The likely causes are a geo/IP policy, a site-wide WAF rule, or a rate limit that the test itself triggered. Test again from another network before changing anything.
• 429 in either. Rate limiting. Wait and retry; a crawler hitting the same limit is a capacity or rule problem, not a robots.txt one.
• 503 or a challenge HTML page. Bot management is challenging the request. A 200 with a JavaScript challenge body is not a readable page for a crawler that does not execute JavaScript, even though the status looks fine.

Reading the Cloudflare headers

Two headers carry most of the signal on Cloudflare. cf-ray identifies the edge request, which is useful when comparing what you saw with what the origin logged.cf-mitigated is set on responses produced by Cloudflare’s mitigation features; a challenge typically carries cf-mitigated: challenge, and other block responses are marked as blocked. Seeing it tells you the response was produced at the edge, not by your application.

The absence of cf-mitigated is not proof that the origin answered. Some WAF actions and custom rules produce plain 403s. That is why the browser-shaped control request matters more than any single header: it distinguishes identity-based refusal from a broader policy, which is the decision you actually have to make.

What the refusal does to a scan

The LLMention scanner treats this case explicitly rather than guessing. When the homepage refuses a crawler-shaped request, it repeats the URL with a browser-shaped User-Agent. If that second request is served, the refusal is aimed at identified crawlers and the AI crawler access check fails under the blocked outcome. If both requests are refused, the refusal is about the scanning address rather than the site, and the check scores 2 of 6 as unverified instead of failing a site for somebody else’s network policy.

The published rule states this in the same words: robots.txt that says nothing is not permission if the request is rejected before the file is read. That is why the two-probe test is part of the rule rather than a footnote in a blog post.

If the block is intentional

Blocking AI crawlers can be a legitimate decision — the sample of news publishers in the study does it deliberately, and the article on that sample describes the trade honestly. The problem is not the block; it is an accidental block that nobody chose, discovered only when a citation does not happen and attributed to the wrong cause.

If the block is deliberate, write it down: which tokens, at which layer, from when, and what you expect it to cost. A one-line comment in the repository next to the WAF change is enough. If it is accidental, the fix depends on the layer: bot-management modes sometimes need a different mode rather than an exception, WAF rules need a scope change, and user-agent allowlists are weak on their own because the header is trivially spoofed. Prefer verified-bot handling or IP-based identity where the CDN offers it, and keep the exception as narrow as the policy allows.

What to do next

1. Run the two-command probe above and record the status pairs.
2. If both are refused, test from another network; if the second network is served, the policy is about geography or address, not about your site.
3. If only the crawler is refused, decide whether that was intended. If not, narrow the exception to the tokens you want, and re-run the probe.
4. Re-scan the site and confirm the AI crawler access dimension moved from blocked or unverified to passed — or record why it is blocked on purpose.

A 403 is not a verdict about your content. It is a statement about the path the request took. Reading the right path is the difference between a five-minute fix and an afternoon spent editing a file that was never consulted.