CrawlPodScan your site

This site runs on what it sells

crawlpod.com uses the ai-visibility npm package for its own robots.txt, llms.txt, structured data, and crawler detection — not a separate internal implementation. Below is exactly which features are used, and this site's own live score from the same scanner at /scan, run against itself.

What this site actually uses

RobotsGenerator

ai-visibility/generators

Generates /robots.txt, explicitly allowing GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot, CCBot, and Applebot-Extended.

LLMSTextGenerator

ai-visibility/generators

Generates /llms.txt and /llms-full.txt from this site's actual content collections at build time, not a hand-maintained file.

SchemaBuilder

ai-visibility/schema

Builds Organization, WebSite, Article, FAQPage, BreadcrumbList, and SoftwareApplication JSON-LD used across the site.

detectBot

ai-visibility/detector

Detects AI crawlers in this site's edge proxy (src/proxy.ts) to power the homepage's live crawler-log panel — the zero-dependency, edge-safe detector added in 0.3.0.

The scanner you're looking at

this site's own /lib/scanner

Built for /scan, and run against this very page below — the same engine, no special-casing.

This site, scanned live

https://crawlpod.com/

100

out of 100

How this score is calculated

Each check contributes points out of 100. A pass earns full points, a warning earns half, a fail earns none. Checks we couldn't determine are excluded entirely rather than penalized. The weighting is published here, not hidden — a black-box score from a company selling the fix isn't trustworthy.

Crawler access

  • AI crawler access (robots.txt)15 pts
  • Bot vs. browser parity15 pts

Content readability

  • Content without JavaScript20 pts
  • Heading hierarchy5 pts
  • Direct answer near the top10 pts

Structured data

  • Structured data (JSON-LD)10 pts
  • Title, description, canonical, OG tags5 pts

Discovery files

  • sitemap.xml5 pts
  • llms.txt3 pts

Agentic readiness

  • Agentic readiness (accessibility tree)10 pts

Performance

  • Server response time2 pts

Crawler access

AI crawler access (robots.txt)

15 ptsPass

7/7 AI crawlers explicitly allowed, none blocked.

Why it matters

An AI crawler that's blocked in robots.txt won't index or cite the page at all, regardless of how good the content is. This is the single most binary check in the report.

Bot vs. browser parity

15 ptsPass

Bot-UA response has 100% of the text content the browser-UA response has.

Why it matters

If a site serves meaningfully less content to a known AI crawler than to a regular browser, that crawler can't see (and can't cite) what a human visitor sees. This can happen by accident — a bot-detection or security tool treating unfamiliar user agents as suspicious.

Content readability

Content without JavaScript

20 ptsPass

3839 characters of real text content present in the raw HTML response.

Why it matters

Many AI crawlers fetch raw HTML and never execute JavaScript. If the meaningful content only appears after client-side rendering, those crawlers see an empty shell — this is the single most common real failure in AI visibility.

Heading hierarchy

5 ptsPass

One <h1>, 13 headings total, no skipped levels.

Why it matters

AI crawlers and answer engines use heading structure to understand how a page is organized and what each section is about. Multiple or missing <h1>s and skipped levels make that structure ambiguous.

Direct answer near the top

10 ptsPass

Found a 335-character paragraph near the top of the content.

Why it matters

AI answer engines favor content that states the answer plainly before elaborating. A page that opens with navigation, a hero banner, or throat-clearing before any substantive text is less likely to be quoted directly.

Structured data

Structured data (JSON-LD)

10 ptsPass

3 schema type(s) found: Organization, WebSite, FAQPage.

Why it matters

JSON-LD tells an AI engine explicitly what a page is — an article, a product, an FAQ — rather than making it infer that from prose. It's the clearest signal a page can send about its own content.

Title, description, canonical, OG tags

5 ptsPass

Title, description, canonical, and Open Graph tags all present.

Why it matters

These tags are the fastest way for any crawler — search or AI — to understand what a page is about without parsing the full body. Missing or duplicate tags create ambiguity about which URL is authoritative.

Discovery files

sitemap.xml

5 ptsPass

sitemap.xml reachable and well-formed, 35 URL(s) listed.

Why it matters

A sitemap gives crawlers a direct list of pages to check, rather than relying entirely on discovering links by crawling. It's low-cost and widely supported.

llms.txt

3 ptsPass

llms.txt present and structurally well-formed.

Why it matters

llms.txt is a low-cost, forward-looking signal, not a guarantee of anything. No major AI engine has confirmed it drives citations, and Google's own guidance is that site owners don't need special files for AI Search. It's worth having as part of a bundle — it costs little — but the load-bearing work is crawler access, server-rendered content, and structured data, not this file.

Agentic readiness

Agentic readiness (accessibility tree)

10 ptsPass

All 28 interactive element(s) have accessible names, <main> landmark present.

Why it matters

AI agents (and screen readers) don't see pixels — they read the accessibility tree: element roles, names, and states. An unlabeled icon button or an input with no associated label is invisible to them even though a sighted human can infer its purpose. This is exactly what Google's Lighthouse Agentic Browsing category audits, and for a product about AI readability specifically, failing it undermines the pitch.

Performance

Server response time

2 ptsPass

Initial response took 74ms.

Why it matters

Slow responses cost crawl budget — crawlers (AI or otherwise) fetch fewer pages per visit when each one is slow, and some abandon slow requests entirely.

Frequently asked questions

Is this page's score computed differently than a normal /scan result?

No. This page calls the exact same scan engine used at /scan, against this site's own homepage. There's no separate or more lenient code path.

What if crawlpod.com doesn't score well?

Then that's a bug in this site, not something we'd hide — a company selling AI visibility tooling that fails its own scan has a credibility problem, not a marketing problem.

Does this page update automatically?

The result is refreshed on a rolling basis (at most an hour old), not computed fresh on every single page view.