Do AI crawlers execute JavaScript?

Some render pages, some do not, and none promise it. The safe engineering assumption is an HTML fetch: if the words you want cited are not in the server response, you are relying on a behaviour the crawler does not have to provide. Test the response, not the assumption.

Is a noscript block enough?

Only if it contains the content, which it usually does not. A noscript tag that says "enable JavaScript" is a status message, not a page. The check the site scores reads the raw response; a crawler reads the same thing.

What about Google rendering my page?

Rendering in a search engine is a second pass that not every fetch receives, and it is not the same promise as an AI answer engine retrieving and quoting a passage. A server-rendered baseline is what makes the page work in both cases; relying on rendering makes it work only sometimes.

What a crawler sees when your page needs JavaScript

Two people can look at your page and see different documents. The browser shows a rendered page with every section, comment and lazy-loaded table. A crawler that does not run JavaScript sees the bytes the server sent: the shell, the navigation, the empty mount point, and whatever text happened to be in the HTML. When a citation does not happen, this gap is one of the first things to rule out, and it is cheap to check.

This article is about the fetch a crawler makes, why a page that looks complete in a browser can be almost empty in that fetch, and how to verify the difference in three commands. It is not an argument that JavaScript is bad. It is an argument that the content you want quoted should survive without it.

What a crawler actually receives

A crawler is an HTTP client. It sends a request with its own User-Agent, receives a response, and — for the agents the scanner tests — works with the body it received. Whether a particular engine runs a rendering pass is a property of that engine, not of your page, and it can change. The engineering question is therefore not "which crawler renders" but "what does the response contain".

The answer is visible in the response body, not in the browser inspector. That distinction matters because the two tools show different trees: DevTools shows the DOM after JavaScript has run, and the page source shows what the server sent. When they disagree, the crawler sees the second one.

Three checks that take a minute

Run these against a real URL, with a crawler-shaped User-Agent, and compare the result with what you see in the browser. The exact host does not matter; the shape of the comparison does.

# 1. How large is the response a crawler gets?
curl -sS -A "GPTBot" https://your-domain.com/page/ | wc -c

# 2. Rough visible-word count after stripping tags.
curl -sS -A "GPTBot" https://your-domain.com/page/ \
  | sed 's/<[^>]*>/ /g' | tr -s ' ' | wc -w

# 3. Does the response contain the words you expect to be cited?
curl -sS -A "GPTBot" https://your-domain.com/page/ \
  | grep -i -c "the phrase you want quoted"

Then do the browser control: open DevTools, disable JavaScript for the site (DevTools settings → Debugger → Disable JavaScript), reload, and read the page. What remains is approximately what a non-rendering crawler can work with. If the article disappears and only the navigation remains, the page depends on JavaScript for its content, even though the browser user never notices.

A large response is not proof of content. Application shells ship their JavaScript, styles and route data in the HTML, so a single-page app can return hundreds of kilobytes and almost no prose. Compare the word count at step 2 with the word count of the rendered article, not with the byte size.

What usually breaks, and how it fails quietly

Client-side fetching for the main copy. The first render is a loading state; the text arrives after a request. A response-level reader gets the loading state.
Content below the fold, loaded on scroll. Scroll-triggered fetching is a real user experience decision, but a crawler does not scroll. Everything behind an intersection observer is absent from the fetch.
Consent or region gates rendered by JavaScript. A modal that must be dismissed, or a redirect decided in the browser, can leave the server response with no article at all.
JSON-LD injected client-side. Structured data that exists only after hydration is missing from the body a crawler parses, which is the same failure as missing prose but harder to notice visually.
Client-side routing with no server fallback. A deep link that returns a shell unless the router runs means the URL a citation would use has no content behind it on a direct fetch.

What these have in common is that they pass every manual browser test. The page looks correct, the console is clean, and the only signal is a fetch you did not make.

What the scanner scores, and what it does not

The rule is no-js-dependency, worth 2 points in the semantic structure dimension. Its condition is deliberately modest: the raw server response must contain at least 100 words of visible text without executing JavaScript.

The note on the rule page is the part worth reading: the hundred words are counted across the whole response, so navigation, footer and cookie text are included. A page whose article is assembled by JavaScript can clear the threshold on its chrome alone. The check therefore proves the response is not empty; it does not prove the article is in it. That is an honest limit of a cheap check, and it is why the manual comparison above is still worth doing on a page that matters.

Fixing it without rewriting the product

The fix is usually not "remove the framework". It is to move the content that should be citable into the server-rendered path, while leaving interactivity where it belongs.

Server-render the article, not the shell. In a React framework that means server components or static generation for the prose; interactive widgets can stay client-side around it.
Fetch on the server for the first paint. If the page needs data, load it before rendering the HTML, or prerender it. A loading skeleton is a user-interface choice; it should not be the only thing a crawler receives.
Keep structured data in the response. JSON-LD belongs in the HTML the server returns, not in an effect that runs after hydration.
Test the deployed response, not localhost. CDNs, consent tools and bot rules can change the body. Run the curl check against production after a deploy.

A useful CI gate is the same idea: fetch the built page, strip tags, count words, and fail if the count is implausibly low for the page type. That is a blunt instrument, but it catches the regression that matters — the day a refactor moves the article into a client component and nobody notices because the browser still looks right.

What this does not solve

Server-rendering does not guarantee a citation. It guarantees that the content is present in the fetch, which is a precondition rather than the outcome. A page can be fully server-rendered and still not be quoted because the answer is buried, the structure is unextractable, or the engine chooses another source. This article is about removing one specific, verifiable cause of failure; it is not a claim about what happens next.

The practical rule: if you would be surprised to see your article missing from a printout of the response body, the crawler would be too.