What AI Crawlers Actually See on Your Website

A website can rank on the first page of Google and, at the same moment, be a blank page to ChatGPT, Claude and Perplexity. Not because the business lacks authority, and not because the writing is thin. The AI crawler asked for the page, the server sent it, and there were no words in it. The words were supposed to be painted in later by a script, and the crawler does not run scripts. This is the single most common reason we find for a well-built site being invisible in AI answers, and almost nobody who owns one knows it is happening.

Two Ways to Build a Web Page

When a browser asks a server for a page, the server sends back a document. For most of the web’s history that document contained the page: the headings, the paragraphs, the prices, the phone number, all present in the text the server returned. Anything that read the document, a browser or a crawler, read the words.

Over the last decade a second way became common. The server sends a nearly empty shell and a bundle of JavaScript, and the browser runs that script to fetch the content and draw the page on the visitor’s screen. The frameworks behind most modern sites (React, Vue, Angular) default to this, and many page builders and app platforms do it without telling the owner. To a person in a browser, the two kinds of page look identical. To anything that does not run the script, the second kind is a header, a footer and an empty box.

What the Crawlers Actually Do

The best public data on this comes from Vercel and MERJ, who measured crawler traffic across Vercel’s network and nextjs.org in late 2024. In one month, OpenAI’s GPTBot made 569 million requests and Anthropic’s Claude 370 million, together about a fifth of Googlebot’s 4.5 billion. Both crawlers downloaded JavaScript files, GPTBot in about 11.5% of its requests and ClaudeBot in 23.84%, and neither executed any of it. They fetch the raw HTML the server returns, take what is there, and move on. PerplexityBot, Meta’s crawler and Bytespider behaved the same way.

The exception is Google. Googlebot renders pages with a headless Chromium browser, executes the JavaScript and indexes what results, and Google’s own documentation describes the process and its cost: rendering goes into a queue, so JavaScript-dependent pages are indexed later and at greater expense than plain HTML. Gemini and Google’s AI features inherit that infrastructure, so a client-rendered page can be visible to Google’s AI and invisible to everyone else’s at the same time. That study is now nearly two years old, no AI vendor has documented adding JavaScript rendering to its crawler since, and the independent re-runs published through 2026 report the same behaviour.

What This Looks Like From the Crawler’s Side

You can see the problem yourself in about two minutes. Open your homepage, view the page source (not the developer tools, which show the page after scripts have run), and search for a sentence from your own copy. If it is there, a crawler without JavaScript can read it. If the source is a few hundred lines of script tags and an empty container, the sentence you are looking for does not exist until a browser builds it, and to GPTBot, ClaudeBot and PerplexityBot it does not exist at all.

The free JavaScript Rendering Check on this site does the same test with numbers. It loads one page twice on our own browser, once with JavaScript disabled and once with it enabled, counts the words a reader sees against the words a non-rendering crawler sees, reports the ratio, and shows you the JavaScript-off text so you can read exactly what such a crawler reads. When we pointed it at our own homepage it read 669 words with JavaScript and 638 without, a retention of 0.954, because this site was engineered to serve its words in the document and we measure it.

The ratio is read against the same bands the deep audit uses. Above 0.80 the page is fully or mostly visible to a crawler. Between 0.50 and 0.80 it is partly hidden. Below 0.50 most of what a person sees is invisible to the machines now answering questions about your industry. There is also a floor: a page with fewer than about 150 words is thin whatever its ratio, because a crawler that reads all of nothing has still read nothing. The deep audit runs this measurement on five sampled pages per site, the homepage and two each of service and content pages, and scores it inside the GEO/AEO pillar next to structured data, content clarity and trust signals. When a site’s firewall or bot challenge answers the browser instead of the page, the audit reports that it could not measure, rather than scoring the wall as if it were the site.

What Is True, and What Is Overstated

• •True: no major AI crawler executes JavaScript, and none of the vendors claims otherwise. Content that exists only after a script runs is not in the page the AI reads.
• •True: content in the initial document counts even if it is not visible prose. Structured data, inline JSON and server-rendered component payloads are all in the response and all readable. What is missed is specifically what the browser builds after loading.
• •Overstated: "Google cannot index JavaScript sites." It can. It does so later and at greater cost, through a rendering queue, and its documentation says so. That documentation is about Googlebot, and it says nothing about anyone else’s crawler.
• •Overstated: "AI crawlers are now most of your traffic." Requests are not visitors. A crawler fetching a script file is a line in a server log, not a customer; the volume figures describe machine attention, not revenue.
• •Overstated: "install a prerendering service and you are done." Google has called dynamic rendering, serving a different rendered copy to crawlers, a workaround rather than a solution, and every workaround of that kind has to keep two versions of the site telling the truth about each other. It can be the right answer. It is not a checkbox.
• •Often confused: which crawler you are blocking. OpenAI, Anthropic and Perplexity each run separate agents for training, for search indexing and for user-triggered visits. A robots.txt rule written to keep your content out of a training set can also keep it out of the search index that answers a customer’s question. The AI Search Visibility check on this site reads a robots.txt and names which is which.

Why the Fix Is an Engineering Decision, Not a Setting

Once a site is client-rendered, there is no toggle that makes its words appear in the document. The remedies have names (server-side rendering, static generation, prerendering at build time) and each is a decision about how the site is built and hosted, made by the people who build it. Each also pulls on the other things that decide visibility. Server rendering changes how fast the first paint arrives and what Core Web Vitals report. Static generation changes how content updates reach the page. Prerendering changes what is cached, for how long, and what a crawler and a customer might see differently. Get one of those wrong and a site that was invisible to AI becomes slow for people, or serves stale prices, or fails an accessibility scan it used to pass.

That is why the measurement above is only the diagnosis. The prescription depends on the framework the site runs, the platform it is hosted on, how often its content changes, what its performance and accessibility scores already are, and what it will cost to change any of that. Those are the questions the deep audit is built to answer in one pass, across all four pillars, on your actual pages: not whether the problem exists, which the free check settles in a few seconds, but which fix is right for this site and what it will take.