Most WordPress sites block AI crawlers without anyone deciding to. A caching plugin's aggressive bot-blocking default, a security plugin's "block bad bots" list that hasn't been updated to distinguish GPTBot from a scraper, or a hosting provider's platform-wide robots.txt — any one of these can make a site invisible to ChatGPT, Claude, and Perplexity, and there's often no error message telling you it happened.
This is a step-by-step guide to checking, fixing, and verifying AI crawler access and content readability on a WordPress site.
Why your WordPress site might be invisible to AI (and you don't know it)
Three causes account for most of it:
- JavaScript-dependent themes and page builders. Some AI crawlers don't execute client-side JavaScript. If a theme or builder renders core content only after JS runs, a crawler that doesn't run it sees an empty or near-empty page — same failure mode Googlebot solved for years ago with rendering, that some AI crawlers haven't matched yet.
- Security and firewall plugins with stale bot lists. A "block known bad bots" feature is only as good as its list. Plugins that haven't been updated recently may not distinguish GPTBot (OpenAI's crawler) from a malicious scraper, and block both under one rule.
- No structured data at all. Even a fully crawlable page gives an AI system little to work with if it can't tell, in machine-readable form, whether a page is a blog post, a product, or a business's contact page. That's what JSON-LD is for — more on this below.
Step 1 — Check if AI crawlers can actually access your site
Two ways to check, in order of effort:
The fast way: run a free scan of your site's URL. It checks robots.txt rules per AI crawler, whether the page serves the same content to a bot as to a browser, and reports the result in plain language — no signup.
The manual way: fetch your own site with an AI crawler's user-agent string and read what actually comes back:
curl -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://yoursite.com/
If the response is a 403, a redirect to an error page, or HTML with none of your actual content in it, something is blocking or breaking the request before your content ever renders.
Step 2 — Fix your robots.txt for AI crawlers
WordPress generates a virtual robots.txt by default if no physical robots.txt file exists in the site root, and most SEO plugins (Yoast, Rank Math) let you edit its contents from the plugin settings. What to check for:
- No blanket
Disallow: /rule that predates AI crawlers existing and was never revisited. - Explicit
Allow(or absence of aDisallow) forGPTBot,ClaudeBot,PerplexityBot, andGoogle-Extendedspecifically, since some hosting-provider default robots.txt files block "all bots except known search engines" as a security posture — which silently includes every AI crawler. See AI crawlers explained for the full current registry of crawler names and what each one actually does. - If you're deciding which crawlers to allow versus block deliberately (not by accident), see the same post's section on the actual tradeoff — allow versus block isn't a single obviously-correct answer for every crawler.
Step 3 — Add llms.txt to your WordPress site
llms.txt is a plain-text file at /llms.txt that gives an AI model a short, structured index of a site — see the full explainer for what it is, what it isn't, and why it's a low-cost addition rather than a guarantee of anything.
Two ways to add it on WordPress today:
- Manual: create a plain text file named
llms.txt, upload it to the site's document root via your host's file manager or SFTP, matching the format# Site name, a description line, and a list of key pages. - Plugin method: a small but growing number of WordPress SEO plugins have started adding llms.txt generation. Check whether your existing SEO plugin has added it before installing a second, single-purpose plugin just for this file.
Step 4 — Add JSON-LD schema markup
Structured data is one of the seven weighted dimensions in AI visibility scoring, and one of the more direct signals a page can offer — it states what a page is instead of making an AI system infer it from prose. On WordPress, the most common gaps:
- Article schema for blog posts — headline, author, published/modified dates. Most SEO plugins add this by default; verify it's actually present with a schema validator or the free scanner, don't assume.
- FAQPage schema for question-and-answer content — pairs well with an actual FAQ block if your theme or page builder supports one.
- Product schema for WooCommerce — name, price, availability, in the format an AI shopping assistant can parse reliably, not just a human-readable price string.
Step 5 — Structure content for AI, not just for scanning humans
Three habits matter more than most WordPress-specific settings:
- Answer-first writing. State the direct answer to a page's core question in the opening sentence or two, before background and context. This is the single highest-weighted dimension in AI visibility scoring for a reason — see the GEO guide for the full seven-dimension breakdown.
- Clean heading hierarchy. One H1, then H2s and H3s in order, no skipped levels. This is what lets a crawler segment a long page into individually citable sections instead of treating it as one undifferentiated block.
- Short, self-contained paragraphs. A paragraph that only makes sense with the three paragraphs before it is hard for an AI system to quote in isolation — each section should stand on its own.
Step 6 — CrawlPod for WordPress (in development)
Everything above can be done manually, in any theme, with any hosting. CrawlPod for WordPress is a plugin — currently in development, not yet published — meant to do the checking and fixing above from the WordPress admin dashboard, without editing theme code: crawler access control, llms.txt generation, structured data for posts and WooCommerce products, and a visibility score. Join the waitlist to be notified when it ships; in the meantime, the manual steps above work today on any WordPress install.
What to expect
None of this guarantees a citation — no tool can, since AI engines choose what to cite based on factors outside any single site's control. What it does is remove the accidental blockers: a crawler that couldn't previously reach the site now can, and content that previously gave an AI system nothing to parse now states its own structure explicitly. Track whether it's working the same way you'd track any referral channel — AI-driven traffic shows up in GA4 as referral traffic from chat.openai.com, perplexity.ai, and similar domains, distinct from organic search.
FAQ
Common questions about WordPress AI visibility are answered below.
See exactly where your WordPress site stands today: run a free AI visibility scan — no signup, real checks against your live URL.