An AI crawler is an automated bot operated by an AI company that fetches web pages — either to collect training data, to index content for an AI-powered search feature, or to retrieve a specific page a user directly asked about. GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot (Perplexity) are the most common, alongside Google-Extended, Bingbot, CCBot, and others.
Not one behavior
"AI crawler" covers bots with meaningfully different purposes: training crawlers (GPTBot, ClaudeBot, Google-Extended) collect data to train models; search/indexing crawlers (PerplexityBot, Bingbot) index content for AI-powered search features; user-triggered fetchers (ChatGPT-User, Claude-User) retrieve a specific page only when a person asks about it. A site can allow one category and block another — see the per-bot entries for how each behaves around robots.txt.
Why access matters more than anything else
An AI crawler that's blocked never sees a page's content at all, regardless of how well-structured or citable that content is otherwise. It's the single most binary factor in AI visibility — everything else (structured data, content quality, direct answers) is moot if the crawler can't get in.
On Shopify, WordPress, and Next.js
Check whether AI crawlers are blocked with a free scan — it reports each named crawler individually, since a site can allow some and block others without realizing it.
Related
GPTBot · ClaudeBot · PerplexityBot · GEO