AI crawler detection and structured data for Next.js and Node.js
ai-visibility is a free, MIT-licensed npm package that detects AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and others), generates robots.txt and llms.txt, builds JSON-LD schema, and scores your content's AI readability. Install it in a Next.js or Node.js app with npm install ai-visibility. The detector and schema builders ship as dependency-free subpath exports for edge runtimes.
npm install ai-visibilityView on GitHub
Zero-dependency, edge-safe subpaths
Importing from the package's root pulls in its full dependency graph (cheerio, chalk, commander). Subpath imports scope that down — this site imports ai-visibility/detector directly in its own edge proxy for exactly this reason.
ai-visibility/detectorBot detection. Zero runtime dependencies, edge-safe.
ai-visibility/schemaJSON-LD builders for 9 schema.org types.
ai-visibility/generatorsrobots.txt and llms.txt generators.
ai-visibility/nextcreateNextMiddleware() and detectAndOptimize().
A real example
This is the actual middleware this site runs — detecting a crawler and recording the visit without blocking the response:
import { NextRequest, NextResponse, type NextFetchEvent } from "next/server";
import { detectBot } from "ai-visibility/detector";
export function proxy(req: NextRequest, event: NextFetchEvent) {
const bot = detectBot(req.headers.get("user-agent") ?? "");
if (bot) {
event.waitUntil(logCrawlerVisit(bot, req.nextUrl.pathname));
}
return NextResponse.next();
}Frequently asked questions
What does ai-visibility actually do?
It detects AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and others) by User-Agent, generates robots.txt and llms.txt, builds JSON-LD structured data, and scores HTML content for AI readability with specific, actionable fixes.
Does it work with the Next.js App Router?
Yes. ai-visibility/next exports createNextMiddleware() for middleware.ts (or Next.js 16's proxy.ts) and detectAndOptimize(), a framework-agnostic function that takes an HTML string and User-Agent and returns optimized output.
Can I use it in Edge Middleware?
ai-visibility/detector is a standalone subpath with zero runtime dependencies, so it's safe to import in edge runtimes. The root package and other subpaths (which use cheerio, chalk, or commander) are not edge-safe — import the specific subpath you need rather than the root barrel if you're running on the edge.
Is it free?
Yes, MIT-licensed, no paid tier for the package itself.
Where's the source code?
On GitHub — link below. Issues and contributions are welcome.
What's the difference between ai-visibility and CrawlPod?
ai-visibility is the open-source npm package. CrawlPod is the broader product built around it — this site, the free scanner, and (in development) no-code Shopify and WordPress versions of the same checks.