Not every bot that hits a server is a search engine crawler anymore. GPTBot, ClaudeBot, PerplexityBot, and a growing list of others now crawl the web specifically for AI training or AI-powered search and answer generation — separate from, and in addition to, Googlebot and Bingbot. Knowing which is which matters, because they don't all do the same thing, and a robots.txt rule that blocks one doesn't automatically block another.
What are AI crawlers?
An AI crawler is an automated bot, operated by an AI company, that fetches web pages for one of three purposes:
- Training — collecting data to train a model. Runs on its own schedule, independent of any specific user request.
- Search/indexing — crawling to power an AI-driven search or answer feature, similar in spirit to how Googlebot indexes for classic search.
- User-triggered fetching — retrieving one specific page because a live user asked a question that made a chat assistant fetch it in real time.
The distinction matters practically: a site can allow the search/user-triggered categories (which can drive real referral traffic when an AI answer cites and links back) while declining the training category, or vice versa, or treat every crawler identically. There's no single correct answer — see the section below on making that call deliberately instead of by accident.
The AI crawler registry
Here are the crawlers most site owners actually need to know about. This list draws from the same registry ai-visibility ships and verifies against each vendor's own published documentation — see the full registry of all 21 crawlers across 13 vendors for the complete table, verification status, and source links per entry.
| Crawler | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training |
| ChatGPT-User | OpenAI | User-triggered |
| OAI-SearchBot | OpenAI | Search |
| ClaudeBot | Anthropic | Training |
| Claude-User | Anthropic | User-triggered |
| Claude-SearchBot | Anthropic | Search |
| PerplexityBot | Perplexity AI | Search |
| Perplexity-User | Perplexity AI | User-triggered |
| Google-Extended | Training | |
| CCBot | Common Crawl | Training |
| Amazonbot | Amazon | Training |
| meta-externalagent | Meta | Training |
| Applebot-Extended | Apple | Training |
| Bytespider | ByteDance | Training (no official documentation exists — the only entry in the registry with no vendor source to verify against) |
| cohere-ai | Cohere | Training |
Note what's not here: Googlebot and Bingbot are classic search-indexing crawlers, not AI-specific ones, even though Google and Microsoft both also operate AI-specific crawlers (Google-Extended) separately. And knowing a crawler's name is only half the picture — see what an AI crawler actually receives when it requests a page for what happens (and often doesn't render) once one of these actually fetches a URL.
Should you block or allow AI crawlers?
There's a real tradeoff, not an obviously correct universal answer:
The case for allowing. When an AI answer engine cites a page and links back, that referral traffic tends to convert unusually well — a Seer Interactive case study measured ChatGPT referral traffic converting at 15.9% against 1.76% for the same site's Google organic traffic, with several other AI engines also outperforming organic in the same dataset. Worth the honest caveat directly alongside that number: it's a single-client case study, and the absolute volume of AI referral traffic was still tiny (well under 1% of the site's total traffic) even with the higher conversion rate — a real, useful signal about traffic quality, not proof of a traffic-volume windfall.
The case for blocking, selectively. Training crawlers (GPTBot, ClaudeBot, Google-Extended) don't send a user to your site the way search/user-triggered crawlers do — their fetch feeds a model's training data, with no click-back event at all. A site that wants AI-driven referral traffic but doesn't want its content used for training can, in principle, allow the search/user-triggered crawlers from a given vendor while blocking that vendor's training crawler specifically — they're separate user-agent tokens, so robots.txt rules can target them independently.
A reasonable default: allow the search and user-triggered crawlers broadly, since they're the ones that can send citation-driven traffic back; decide on training crawlers deliberately, per vendor, rather than leaving the decision to whatever a hosting platform's default robots.txt happened to ship with.
How to control AI crawler access
robots.txt rules target specific crawlers by their user-agent token:
User-agent: GPTBot
Disallow: /
User-agent: PerplexityBot
Allow: /
Programmatically, in a Next.js or Node.js project, ai-visibility's RobotsGenerator builds this from a config object instead of hand-writing rules:
import { RobotsGenerator } from "ai-visibility/generators";
const robotsTxt = new RobotsGenerator({
allowAI: ["GPTBot", "ClaudeBot", "PerplexityBot", "Google-Extended"],
blockAI: [],
sitemapUrl: "https://example.com/sitemap.xml",
}).generate();
See the GEO guide for where this fits into the broader picture, and llms.txt explained for the complementary, unenforced file that helps allowed crawlers navigate a site more efficiently once they're in.
How to track AI crawler visits
Three levels of effort:
- Server logs. The most universal method — grep access logs for known AI crawler user-agent substrings. Works on any stack, no dependencies.
- ai-visibility's crawler-visit logger. The npm package's
AIVisitorLoggerand the Python package's analytics module both log matched crawler visits with timestamp, path, and response data, for surfaces that want this built in rather than hand-parsing logs. - CrawlPod for WordPress (in development). For WordPress specifically, a plugin-level analytics view is planned as part of CrawlPod for WordPress — join the waitlist for updates.
FAQ
Common questions about AI crawlers are answered below.
See which AI crawlers can actually reach your site today: run a free scan — it checks each one individually, since a site can allow some and block others without realizing it. For the full data behind claims like "most sites don't block AI crawlers evenly across categories," see the 50-website AI visibility study.