ClaudeBot is Anthropic's web crawler for collecting web content that may contribute to training Claude. It's one of three separate crawlers Anthropic operates, each with a distinct purpose and User-Agent, and — per Anthropic's own support documentation — all three respect robots.txt.
The three-bot family
- ClaudeBot — training data collection. Disallowing it excludes a site from training datasets.
- Claude-User — fetches a page when a Claude user asks a question that requires visiting it. Disallowing it means Claude can't retrieve that content in response to a user's question.
- Claude-SearchBot — indexes content to improve Claude's search result quality. Disallowing it reduces a site's visibility in Claude's search results.
Blocking one doesn't block the others — a site can, for instance, opt out of training (block ClaudeBot) while staying visible in user-directed queries (allow Claude-User).
On Shopify, WordPress, and Next.js
Same access mechanics as any other named crawler — explicit allow rules in robots.txt. Verify current status with a free scan.