Most websites aren't ignored by AI search engines because their content is bad. They are ignored because they are technically ineligible.
If an AI crawler is blocked by robots.txt, hits a wall of JavaScript, or finds no structured context (llms.txt, schema), the model cannot read the site. If it cannot read the site, it cannot cite it. Period.
We wanted to prove this. So we took 20 live production sites and applied only technical, on-page AI-SEO optimizations. We built zero backlinks. We did no off-page PR. We simply made these sites technically "eligible" to be read and understood by AI.
The Headline Finding: After implementing CrawlPod's technical eligibility checklist, the average AI crawler visit rate across the 20 sites increased by 340%. Consequently, the aggregate citation rate (the share of times these sites were named as a source by AI engines) jumped from 4.2% to 16.8% — a 4x increase in visibility, driven purely by technical readiness.
If you want to skip the write-up: reproduce the audit yourself.
The Core Thesis: Eligibility Precedes Citation
Before you can optimize for "Generative Engine Optimization" (GEO), you must first achieve AI Crawler Eligibility.
In this study, we defined a site as "Eligible" only when it met four baseline technical criteria. Our hypothesis was simple: fixing these four technical bottlenecks would remove the artificial ceiling on a site's AI citations.
Method: What We Actually Did
We selected 20 mid-market B2B and SaaS websites with existing content but low AI visibility.
The Rule: No new content was written, and no backlinks were built. We only applied technical, on-page remediation using CrawlPod's workflow.
The 4-Step "Eligibility" Checklist Applied:
- Crawler Access Unlocked: Updated
robots.txtto explicitlyAllowGPTBot, ClaudeBot, PerplexityBot, and Bytespider (where verified). - Renderability Fixed: Ensured clean, semantic HTML was served directly to verified AI crawler IP ranges, bypassing heavy JavaScript rendering bottlenecks.
- Context Provided: Generated and deployed
llms.txtandllms-full.txtat the root directory, giving crawlers a direct map of the site's most important content. - Structure Clarified: Implemented core JSON-LD (Article, FAQPage, Organization) and restructured leading paragraphs to answer core queries directly (Answer-First formatting).
Measurement Window
- Baseline: 14 days of tracking crawler logs and running a standardized set of 50 industry-specific prompts across ChatGPT (with Search) and Perplexity Pro.
- Gap: 14 days allowed for AI crawlers to re-index the newly eligible sites.
- Post-Change: 14 days of identical tracking and prompt testing.
Ready to optimize for AI?
Install the free CrawlPod plugin and see your WordPress site's AI visibility score in minutes.
Results
1. The Crawler Visit Explosion (Proof of Eligibility)
You can't be cited if you aren't visited. The first thing we measured was server-side crawler traffic (verified by IP, not just User-Agent).
| Metric | Baseline (14 Days) | Post-Change (14 Days) | Change |
|---|---|---|---|
| Avg. Verified AI Bot Hits / Site | 22 | 97 | +340% |
Takeaway: The technical changes worked immediately. The bots weren't ignoring the content; they were previously blocked or confused by it. Once the door was opened, they walked in.
2. The Citation Lift (Proof of Value)
With the sites now eligible and actively crawled, we measured how often they appeared as a cited source in our 50-prompt test suite (run 3 times per engine for consistency).
| Engine | Baseline Citation Rate | Post-Change Citation Rate | Lift |
|---|---|---|---|
| Perplexity Pro | 5.1% | 21.4% | +16.3 pp |
| ChatGPT (Search) | 3.3% | 12.2% | +8.9 pp |
| Aggregate Average | 4.2% | 16.8% | +12.6 pp |
(Note: "pp" = percentage points. A jump from 4.2% to 16.8% represents a 4x relative increase in citation probability).
What Moved and What Didn't
llms.txtis a cheat code for Perplexity. Sites that implementedllms-full.txtsaw a disproportionately high lift in Perplexity citations. Perplexity's architecture heavily favors direct, mapped text ingestion.- Answer-first formatting beats complex schema. While JSON-LD is important for entity recognition, we observed that simply moving the direct answer to the first 50 words of a page had a more immediate impact on ChatGPT's source selection than adding complex nested schema.
- Technical SEO is the foundation, not the ceiling. Making a site eligible doesn't guarantee it will be the top citation (that requires content quality and authority). But it does guarantee the site is now in the running. Before this, it wasn't even on the roster.
Limitations (Being Transparent)
We believe in radical transparency. Here is what this study does not prove:
- It does not isolate individual variables. Because we applied the 4-step checklist together, we cannot definitively say if
llms.txtor therobots.txtchange contributed more to the lift. They work as a system. - It does not measure long-term decay. AI models update constantly. This is a snapshot of a 30-day window. Continuous monitoring is required.
- No off-page factors were tested. This study strictly measures the impact of technical, on-page eligibility.
Reproduce This Audit
You don't have to take our word for it. You can run the same technical eligibility audit on your own site using the CrawlPod CLI.
# 1. Scan your site for AI crawler eligibility bottlenecks
npx ai-visibility discover --site https://yoursite.com
# 2. Generate the recommended technical fixes (llms.txt, robots rules, etc.)
npx ai-visibility fix --apply
# 3. Measure your baseline citation rate against competitors
npx ai-visibility measure --runs 3 --engines chatgpt,perplexity
For the underlying concepts behind this checklist, see what llms.txt actually does and the full GEO scoring methodology. For a broader, single-snapshot dataset across 46 sites, see our 50-site AI visibility audit.
See exactly where your own site lands against this data: run a free AI visibility scan — no signup required.