This site runs on what it sells
crawlpod.com uses the ai-visibility npm package for its own robots.txt, llms.txt, structured data, and crawler detection — not a separate internal implementation. Below is exactly which features are used, and this site's own live score from the same scanner at /scan, run against itself.
What this site actually uses
RobotsGeneratorai-visibility/generators
Generates /robots.txt, explicitly allowing GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot, CCBot, and Applebot-Extended.
LLMSTextGeneratorai-visibility/generators
Generates /llms.txt and /llms-full.txt from this site's actual content collections at build time, not a hand-maintained file.
SchemaBuilderai-visibility/schema
Builds Organization, WebSite, Article, FAQPage, BreadcrumbList, and SoftwareApplication JSON-LD used across the site.
detectBotai-visibility/detector
Detects AI crawlers in this site's edge proxy (src/proxy.ts) to power the homepage's live crawler-log panel — the zero-dependency, edge-safe detector added in 0.3.0.
The scanner you're looking atthis site's own /lib/scanner
Built for /scan, and run against this very page below — the same engine, no special-casing.
This site, scanned live
https://crawlpod.com/
100
out of 100
How this score is calculated
Each check contributes points out of 100. A pass earns full points, a warning earns half, a fail earns none. Checks we couldn't determine are excluded entirely rather than penalized. The weighting is published here, not hidden — a black-box score from a company selling the fix isn't trustworthy.
Crawler access
- AI crawler access (robots.txt)15 pts
- Bot vs. browser parity15 pts
Content readability
- Content without JavaScript20 pts
- Heading hierarchy5 pts
- Direct answer near the top10 pts
Structured data
- Structured data (JSON-LD)10 pts
- Title, description, canonical, OG tags5 pts
Discovery files
- sitemap.xml5 pts
- llms.txt3 pts
Agentic readiness
- Agentic readiness (accessibility tree)10 pts
Performance
- Server response time2 pts
Crawler access
AI crawler access (robots.txt)
7/7 AI crawlers explicitly allowed, none blocked.
Why it matters
An AI crawler that's blocked in robots.txt won't index or cite the page at all, regardless of how good the content is. This is the single most binary check in the report.
Bot vs. browser parity
Bot-UA response has 100% of the text content the browser-UA response has.
Why it matters
If a site serves meaningfully less content to a known AI crawler than to a regular browser, that crawler can't see (and can't cite) what a human visitor sees. This can happen by accident — a bot-detection or security tool treating unfamiliar user agents as suspicious.
Content readability
Content without JavaScript
3839 characters of real text content present in the raw HTML response.
Why it matters
Many AI crawlers fetch raw HTML and never execute JavaScript. If the meaningful content only appears after client-side rendering, those crawlers see an empty shell — this is the single most common real failure in AI visibility.
Heading hierarchy
One <h1>, 13 headings total, no skipped levels.
Why it matters
AI crawlers and answer engines use heading structure to understand how a page is organized and what each section is about. Multiple or missing <h1>s and skipped levels make that structure ambiguous.
Direct answer near the top
Found a 335-character paragraph near the top of the content.
Why it matters
AI answer engines favor content that states the answer plainly before elaborating. A page that opens with navigation, a hero banner, or throat-clearing before any substantive text is less likely to be quoted directly.
Structured data
Structured data (JSON-LD)
3 schema type(s) found: Organization, WebSite, FAQPage.
Why it matters
JSON-LD tells an AI engine explicitly what a page is — an article, a product, an FAQ — rather than making it infer that from prose. It's the clearest signal a page can send about its own content.
Title, description, canonical, OG tags
Title, description, canonical, and Open Graph tags all present.
Why it matters
These tags are the fastest way for any crawler — search or AI — to understand what a page is about without parsing the full body. Missing or duplicate tags create ambiguity about which URL is authoritative.
Discovery files
sitemap.xml
sitemap.xml reachable and well-formed, 35 URL(s) listed.
Why it matters
A sitemap gives crawlers a direct list of pages to check, rather than relying entirely on discovering links by crawling. It's low-cost and widely supported.
llms.txt
llms.txt present and structurally well-formed.
Why it matters
llms.txt is a low-cost, forward-looking signal, not a guarantee of anything. No major AI engine has confirmed it drives citations, and Google's own guidance is that site owners don't need special files for AI Search. It's worth having as part of a bundle — it costs little — but the load-bearing work is crawler access, server-rendered content, and structured data, not this file.
Agentic readiness
Agentic readiness (accessibility tree)
All 28 interactive element(s) have accessible names, <main> landmark present.
Why it matters
AI agents (and screen readers) don't see pixels — they read the accessibility tree: element roles, names, and states. An unlabeled icon button or an input with no associated label is invisible to them even though a sighted human can infer its purpose. This is exactly what Google's Lighthouse Agentic Browsing category audits, and for a product about AI readability specifically, failing it undermines the pitch.
Performance
Server response time
Initial response took 74ms.
Why it matters
Slow responses cost crawl budget — crawlers (AI or otherwise) fetch fewer pages per visit when each one is slow, and some abandon slow requests entirely.
Frequently asked questions
Is this page's score computed differently than a normal /scan result?
No. This page calls the exact same scan engine used at /scan, against this site's own homepage. There's no separate or more lenient code path.
What if crawlpod.com doesn't score well?
Then that's a bug in this site, not something we'd hide — a company selling AI visibility tooling that fails its own scan has a credibility problem, not a marketing problem.
Does this page update automatically?
The result is refreshed on a rolling basis (at most an hour old), not computed fresh on every single page view.