robots.txt for AI crawlers
AI crawlers identify themselves with distinct user-agent strings — GPTBot, ClaudeBot, PerplexityBot, and others — that you can allow or block individually in robots.txt. CrawlPod's free plugin tracks 16 known crawlers and lets you toggle each one from a dashboard instead of hand-editing the file.
Which AI bots exist
AI companies operate separate crawlers for separate purposes — training a model is a different bot from that same company answering a live user query. Some of the more common ones:
| Bot | Company | Purpose |
|---|---|---|
GPTBot | OpenAI | Training |
OAI-SearchBot | OpenAI | Search |
ChatGPT-User | OpenAI | Live browsing on user request |
ClaudeBot | Anthropic | Training |
PerplexityBot | Perplexity | Search / answers |
Google-Extended | AI training signal | |
Amazonbot | Amazon | Various |
Bytespider | ByteDance | Training |
This is a representative list, not exhaustive — CrawlPod's free tier tracks 16 known crawlers in total, covering training, search, and live-browsing bots across the major AI providers.
How to allow or block specific bots
robots.txt uses User-agent blocks to target a specific crawler by name, with Disallow and Allow rules underneath controlling which paths that crawler may fetch. A block that doesn't match any crawler's user-agent string has no effect on it.
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Bytespider
Disallow: /robots.txt is respected by convention, not enforced by any technical mechanism — well-behaved crawlers check it before fetching, but nothing stops a bot that ignores it. It's a signal, not a lock.
CrawlPod's crawler management UI
The free CrawlPod plugin's dashboard lists all 16 tracked AI crawlers with a per-bot toggle to allow or block each one, so you don't need to hand-edit robots.txt directly. Alongside each toggle, the dashboard shows analytics on which bots have actually visited your site and how often, so decisions are based on real traffic rather than guesswork.
Best practices
- Default to allowing AI crawlers unless you have a specific reason not to — blocking a crawler makes you invisible to that engine's answers, not just to its training data.
- Review your crawler analytics periodically rather than setting rules once and forgetting them; which bots are actually active changes over time.
- Understand that blocking a training crawler (for example, GPTBot) doesn't retroactively remove any content that crawler already used for training, and doesn't block that same company's separate live-browsing bot (for example, ChatGPT-User) — each user-agent is controlled independently.
See llms.txt setup guide for the complementary step of giving allowed crawlers a structured index of your content.