Most AI crawlers — GPTBot, ClaudeBot, PerplexityBot among them — fetch a page's raw HTML response and stop there. No JavaScript executes. Nothing your browser would render dynamically is dynamic to a crawler; it's either already present in the HTML the server sent, or it doesn't exist as far as the crawler is concerned. This is the single most common real failure in AI visibility, because it's invisible if you only ever look at your site in a browser — a browser runs the JavaScript, so it shows you the finished result regardless of whether a crawler would ever see it.
What "raw HTML" means concretely
When you view a page in a browser, you're looking at the DOM after the browser has parsed the HTML, run any JavaScript, and applied any changes that JavaScript made to the page. A crawler that doesn't execute JavaScript sees only the first part of that: whatever the server actually sent, before any of that runs. If your content is added to the page by JavaScript — common in some client-rendered frameworks and page builders — a crawler can receive a page that's technically valid HTML and functionally empty.
A real example: this site's own homepage
Running this site's own scanner against crawlpod.com measures exactly this — the actual text content present in the raw server response, before any script runs. On the date this was written, the result was:
- Raw HTML response size: 51,143 characters
- Extractable text content: 3,839 characters
- Text density: ~7.5% of the response
That's a real number from a real scan, not a hypothetical — and it's a reasonable result specifically because this site is server-rendered (Next.js App Router, statically generated where possible), so the meaningful content is already in the initial HTML rather than injected afterward. A client-rendered page with the same visible content could return a raw response with only a few hundred characters of text — an empty <div id="root"></div> and a pile of script tags — and a crawler that doesn't run JavaScript would see something close to nothing.
How to check what your own page actually sends
You don't need to be able to read raw HTML to find out. The scanner measures this directly: it fetches a URL's raw server response (the same way a non-JavaScript crawler would) and reports the actual text content found, flagging it when there's too little to be useful. It's one of eleven checks, weighted the highest of any single check in the published scoring — per the brief this project was built from, it's the single most common real failure, and the numbers above are why: the gap between what a browser shows you and what a crawler actually receives can be enormous, and it's completely invisible until you check the raw response directly.
Related
Content without JavaScript is check #3 in the full scoring breakdown · Agentic Browsing · GEO vs. SEO