CrawlPod

AI crawler detection and structured data for Node.js and Python

ai-visibility is available as an npm package for Node.js and a Python package on PyPI. Both are free, MIT-licensed, and share the same verified crawler registry and scoring weights. It detects AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and 18 others across 13 vendors), generates robots.txt and llms.txt, builds JSON-LD schema, and scores your content's AI readability. Any framework with a server — Next.js, Nuxt, React Router, Express, Django, Flask, FastAPI — gets full integration; Vue and React single-page apps get build-time file generation. Looking specifically for Python? See the Python developers page.

npm install ai-visibility

The npx and pip CLI commands above score a live URL for free with no install step. See the CLI reference for every command, including --fail-under as a CI gate.

Score your content from the command line

0.5.0 added audit and lint: a 7-dimension content-citability score (see the scoring methodology) for a live URL or a local build directory, with an opt-in CI gate.

npx ai-visibility audit https://yoursite.com
npx ai-visibility audit --dir ./out --fail-under 60   # CI gate, exits non-zero below 60
npx ai-visibility lint                                # shorthand: audit --dir . --fail-under 50

Zero-dependency, edge-safe subpaths

Importing from the package's root pulls in its full dependency graph (cheerio, chalk, commander). Subpath imports scope that down — this site imports ai-visibility/detector directly in its own edge proxy for exactly this reason.

ai-visibility/detector

Bot detection + detectAndOptimize(). Zero runtime dependencies, edge-safe, framework-agnostic.

ai-visibility/schema

JSON-LD builders for 11 schema.org types.

ai-visibility/generators

robots.txt and llms.txt generators.

ai-visibility/next

createNextMiddleware() for proxy.ts/middleware.ts.

The Python package also ships with zero required dependencies. Framework extras (pip install ai-visibility[django]) add only the framework itself as a dependency — see the Python developers page for details.

Framework support

ai-visibilitydetects crawlers from request headers and serves optimized HTML at request time — none of that exists in a client-only bundle. The question that decides what a framework can actually do isn't whether it has components — it's whether it has a server:

FrameworkHas a server?Integration
Node.js / ExpressYesFull — dedicated ai-visibility/express subpath
Next.js (App Router)YesFull — dedicated ai-visibility/next subpath
Nuxt (Nitro)YesFull — via the framework-agnostic detector/generators/schema exports
React Router (framework mode)YesFull — same pattern as Nuxt; Remix, Astro (server), TanStack Start too
Vue SPA (Vite, no server)NoBuild-time robots.txt/llms.txt + build-time JSON-LD only
React SPA (Vite/CRA, no server)NoSame as Vue SPA
Django (Python)YesFull — drop-in middleware, see the Python package
Flask (Python)YesFull — Flask extension, see the Python package
FastAPI (Python)YesFull — ASGI middleware, see the Python package

A client-rendered SPA with no server has a real AI-visibility gap this package can only partly close — the fix is SSR, prerendering, or a thin server layer in front of the static files, not a missing feature here. Full working recipes for all six rows: /docs/recipes.

A real example

Detect a crawler and record the visit without blocking the response. This site's own proxy runs a variant of exactly this — routed through an internal API endpoint instead of calling the database directly, so the code that runs on every page request stays free of any DB dependency, but the detect-and-log shape below is the same:

import { NextRequest, NextResponse, type NextFetchEvent } from "next/server";
import { detectBot } from "ai-visibility/detector";

export function proxy(req: NextRequest, event: NextFetchEvent) {
  const bot = detectBot(req.headers.get("user-agent") ?? "");
  if (bot) {
    event.waitUntil(logCrawlerVisit(bot, req.nextUrl.pathname));
  }
  return NextResponse.next();
}

The equivalent on Django — same detection, same registry:

# settings.py
MIDDLEWARE = [
    "ai_visibility.middleware.django.AIVisibilityMiddleware",
    # ...the rest of your middleware
]

AI_VISIBILITY = {
    "optimize": True,
    "inject_schemas": True,
    "log_visits": True,
}

Full Flask and FastAPI examples: the Python developers page.

Frequently asked questions

What does ai-visibility actually do?

It detects AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and others) by User-Agent, generates robots.txt and llms.txt, builds JSON-LD structured data, and scores HTML content for AI readability with specific, actionable fixes.

Does it work with the Next.js App Router?

Yes. ai-visibility/next exports createNextMiddleware() for middleware.ts (or Next.js 16's proxy.ts); its onDetect callback can be async and is safely kept alive via Next's own waitUntil(). detectAndOptimize() — HTML string and User-Agent in, optimized HTML out — lives in ai-visibility/detector, the framework-agnostic module, and is re-exported from /next for convenience.

Does it work outside Next.js — Nuxt, React Router, Vue, plain React?

Nuxt (Nitro) and React Router in framework mode get full integration through the same framework-agnostic detector/generators/schema exports Next.js uses under the hood — see the framework support table below. A Vue or React single-page app built with Vite has no server to run middleware on at all, so it only gets build-time robots.txt/llms.txt generation and build-time JSON-LD injection, not request-time bot detection.

Can I use it in Edge Middleware?

ai-visibility/detector is a standalone subpath with zero runtime dependencies, so it's safe to import in edge runtimes. The root package and other subpaths (which use cheerio, chalk, or commander) are not edge-safe — import the specific subpath you need rather than the root barrel if you're running on the edge.

Does it have a CLI, or is it library-only?

Both. `npx ai-visibility audit <url>` (or `--dir` for a local build) scores content citability with no install step; `lint` is a CI-friendly shorthand with a default fail threshold. See the CLI reference for every command.

Is this the same score as the WordPress plugin's audit?

No — this package's ContentAnalyzer scores content citability (can an AI system cite this page?) across 7 dimensions. The WordPress Pro audit engine scores agentic readiness (can an AI agent interact with this page?) across 9 checks. Same page, different questions, different numbers — see the scoring methodology page for the full comparison.

Is it free?

Yes, MIT-licensed, no paid tier for the package itself.

Where's the source code?

On GitHub — link below. Issues and contributions are welcome.

What's the difference between ai-visibility and CrawlPod?

ai-visibility is the open-source npm package. CrawlPod is the broader product built around it — this site, the free scanner, and (in development) no-code Shopify and WordPress versions of the same checks.