AI crawler registry

The full registry

What "verified" means

verified: truemeans the token was checked directly against that vendor's own published documentation, with the source URL and check date recorded on the entry. Bytespider is explicitly verified: false— the opposite of "not yet checked". It means ByteDance was checked for official documentation and none exists at all; that's a real, permanent state, not a to-do item. The remaining unverified entries (YouBot, cohere-ai, Diffbot) predate the 0.4.0 verification audit and haven't been individually re-checked yet — an honest gap, not a claim that they're wrong.

Call getUnverifiedBots() from ai-visibility/detector to get this list programmatically — it currently returns all 4 of the entries above marked "No". (The package's own 0.4.0 CHANGELOG.mddescribes this as returning "currently just Bytespider"; that undercounts it. The package's README states the correct number, and it matches what the installed code actually returns — checked directly rather than trusted from either doc.)

crawlers.json for non-JS consumers

The same registry is published as plain JSON at dist/crawlers.jsonwith every npm release — generated from the compiled detector at build time, so it can't drift from the package's own AI_CRAWLERS. It isn't in the package's exports map, so it's not reachable via require()/import— it's meant to be fetched over a CDN mirror of the npm package, unauthenticated, by build tooling in languages that aren't JavaScript at all (a WordPress plugin's PHP build step, a Shopify app's Ruby/Liquid tooling):

# Pin an exact version — @latest is fine for eyeballing, not for a build script.
curl -sL https://cdn.jsdelivr.net/npm/ai-visibility@0.4.0/dist/crawlers.json -o crawlers.json

# Then transform it into whatever native structure that surface needs
# (a PHP array, a Liquid data file) and commit the *generated* result.

Fetch at build/release time, never at runtime.A plugin or app that fetched this URL on every page load would take on a live dependency on a third-party CDN being reachable, for data that changes a handful of times a year. Vendor a static copy instead, and check the response's schemaVersionfield before trusting its shape — fail loudly if it's higher than what your transform script was written against, rather than silently misreading a changed structure.

{
  "schemaVersion": 1,
  "packageVersion": "0.4.0",
  "generatedAt": "2026-08-03T22:35:26.441Z",
  "source": "https://github.com/Muhammadfaizanjanjua109/ai-visibility/blob/main/src/data/crawlers.ts",
  "crawlers": [
    { "name": "GPTBot", "company": "OpenAI", "userAgentPattern": "gptbot",
      "purpose": "training", "verified": true,
      "sourceUrl": "https://developers.openai.com/api/docs/bots",
      "lastChecked": "2026-08-03" }
    // ...one entry per crawler above, same shape as BotInfo
  ]
}

Re-verification cadence

The package re-checks every entry's sourceUrl quarterly (or before any release its maintainer is relying on), confirming the token is still current, still a stable substring rather than a version-pinned one (vendor UAs embed a version number, e.g. GPTBot/1.4 — that number is expected to change and must never be part of the match pattern), and that the purpose classification still holds. This is exactly the process that caught Claude-Web's deprecation for 0.4.0.

See AI crawler in the glossary for what these tokens are and why they matter, and the migration guideif you're updating code that still matches on Claude-Web.