CrawlPod

Recipes

The first four recipes (App Router, JSON-LD, llms.txt, Express) are lifted directly from this site's own source or the package README — not illustrative pseudocode. The framework recipes further down (Nuxt, Vue, React, React Router) are adapted from the package's own verified examples/directory and framework integration guide — each was scaffolded, built, and exercised with real requests during that package's own development, since this site itself doesn't run those frameworks.

Server vs. no-server

ai-visibilitydetects crawlers from request headers and serves optimized HTML at request time — none of that exists in a client-only bundle. The question that decides what a given framework can actually do isn't "does it have components" — it's does it have a server:

FrameworkHas a server?What works
Node.js / ExpressYesFull integration — the Express recipe below.
Next.js (App Router)YesFull integration — native proxy.ts/middleware.ts support, the App Router recipe below.
Nuxt (Nitro)YesFull integration via the framework-agnostic exports — see the Nuxt recipe below.
React Router (framework mode)YesFull integration — same request/Response pattern as Nuxt. See the React Router recipe below.
Remix / Astro (server adapter) / TanStack StartYesNot covered by a dedicated recipe here, but the same pattern as React Router applies directly — all hand you a request object and let you return a Response.
Vue SPA (Vite, no server)NoBuild-time robots.txt/llms.txt generation and build-time JSON-LD injection only — see below.
React SPA (Vite/CRA, no server)NoSame constraints as Vue SPA.

A Vue or React SPA with no server genuinely can't run this package's middleware or bot detection — there's no request for it to run against. That's a real, only-partly-solved AI-visibility gap for a client-rendered SPA, not a missing feature of this package: the actual fix is SSR, prerendering, or putting a server (even a thin one — a Vercel/Netlify function, a Cloudflare Worker) in front of the static files. Run a free scan against a deployed SPA to see what a no-JS-execution crawler actually receives.

App Router setup

Detect crawlers in the edge proxy and log visits without blocking the response, using event.waitUntil() rather than createNextMiddleware's onDetect (which isn't awaited and doesn't expose waitUntil):

src/proxy.ts (the detect-and-log shape this site's own proxy is based on)
import { NextRequest, NextResponse, type NextFetchEvent } from "next/server";
import { detectBot } from "ai-visibility/detector";
import { logCrawlerVisit } from "@/lib/crawler-log";

export function proxy(req: NextRequest, event: NextFetchEvent) {
  const res = NextResponse.next();

  const bot = detectBot(req.headers.get("user-agent") ?? "");
  if (bot) {
    event.waitUntil(logCrawlerVisit(bot, req.nextUrl.pathname));
  }

  return res;
}

export const config = {
  matcher: "/((?!_next/static|_next/image|favicon.ico).*)",
};

Then generate robots.txt as a Route Handler:

src/app/robots.txt/route.ts (this site's actual file)
import { RobotsGenerator } from "ai-visibility/generators";

export async function GET() {
  const body = new RobotsGenerator({
    allowAI: ["GPTBot", "ClaudeBot", "PerplexityBot", "Google-Extended", "Bingbot", "CCBot", "Applebot-Extended"],
    blockAI: [],
    disallow: ["/api"],
    sitemapUrl: "https://example.com/sitemap.xml",
  }).generate();

  return new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8" } });
}

Adding JSON-LD to a page

Use the plain object SchemaBuilder returns — not toScriptTag(), which returns a full HTML string meant for raw template injection, not JSX:

app/blog/[slug]/page.tsx
import { SchemaBuilder } from "ai-visibility/schema";

function ArticleJsonLd({ headline, author, publishedDate, url }: {
  headline: string; author: string; publishedDate: string; url: string;
}) {
  const schema = SchemaBuilder.article({ headline, author, publishedDate, url });
  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{ __html: JSON.stringify(schema) }}
    />
  );
}

Generating llms.txt from a content directory

Build the page list from what actually exists in a content directory at request time, rather than hand-maintaining it — this is exactly how this site's own llms.txt and llms-full.txt are generated:

app/llms.txt/route.ts (this site's actual file, trimmed)
import { LLMSTextGenerator } from "ai-visibility/generators";
import { getAllPages } from "@/lib/pages-index"; // reads MDX frontmatter from content/

export const dynamic = "force-static";

export async function GET() {
  const body = LLMSTextGenerator.minimal({
    siteName: "Example",
    description: "Example site description.",
    baseUrl: "https://example.com",
    pages: getAllPages(), // built from content/*.mdx frontmatter, not hardcoded
  });

  return new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8" } });
}

Express bot optimization

Adapted from the package README, checked against ai-visibility/express's published types (this project has no Express server to compile it against directly):

server.ts
import express from "express";
import { createAIMiddleware, optimizeResponseForAI } from "ai-visibility/express";

const app = express();

app.use(createAIMiddleware({ verbose: true }));
app.use(optimizeResponseForAI({
  stripJs: true,
  removeAds: true,
  removeTracking: true,
}));

app.get("/", (req, res) => {
  // req.isAIBot / req.aiBotInfo are set by createAIMiddleware for any
  // downstream handler that wants to branch on it
  res.send("<html>...</html>");
});

Nuxt (Nitro)

Nitro's server gives you the real thing — request middleware, dynamic routes, all of it. Nitro uses H3's defineEventHandler, not Express signatures, so createAIMiddleware/optimizeResponseForAI from ai-visibility/express don't apply here — reach for the framework-agnostic AIBotDetector directly:

server/middleware/ai-detector.ts
import { AIBotDetector } from "ai-visibility/detector";

const detector = new AIBotDetector();

export default defineEventHandler((event) => {
  const bot = detector.detect(getHeader(event, "user-agent") ?? "");
  event.context.aiBot = bot;
  if (bot) setResponseHeader(event, "x-ai-crawler", bot.name);
});

Generated files are server routes returning plain text:

server/routes/robots.txt.ts
import { RobotsGenerator } from "ai-visibility/generators";

export default defineEventHandler((event) => {
  setResponseHeader(event, "content-type", "text/plain");
  return RobotsGenerator.allowAll({ sitemapUrl: "https://example.com/sitemap.xml" });
});

The gotcha that will actually bite you: nuxi init scaffolds a placeholder public/robots.txt. Nitro serves static files from public/ before dynamic server routes at the same path — that placeholder silently shadows server/routes/robots.txt.tsunless you delete it. The generated output can look identical between "shadowed" and "working" at a glance, since the scaffold's placeholder happens to also read User-Agent: * / Disallow:. Check for public/llms.txt too. See Troubleshooting for this and other framework gotchas.

Since Nuxt server-renders, JSON-LD injected via useHead() actually reaches non-JS crawlers (unlike the SPA cases below):

pages/index.vue
<script setup lang="ts">
import { SchemaBuilder } from "ai-visibility/schema";

useHead({
  script: [{ type: "application/ld+json", innerHTML: JSON.stringify(
    SchemaBuilder.website({ name: "My App", url: "https://example.com" })
  ) }],
});
</script>

detectAndOptimize()doesn't hook cleanly into Nuxt's own Vue SSR pipeline — it's meant for wherever you already have a raw HTML string in hand (a CMS-fetched page, a cached render, a static export you're serving dynamically), not for transforming Nuxt's own rendered output in-flight.

Vue (Vite SPA)

No server — see the table above. Build-time generation via a small Vite plugin, verified against Vite 8.2 / Vue 3.5 / @unhead/vue 3.2:

vite.config.ts
import { RobotsGenerator, LLMSTextGenerator } from "ai-visibility/generators";
import { SchemaBuilder } from "ai-visibility/schema";

function aiVisibilityStatic(): Plugin {
  return {
    name: "ai-visibility-static",
    transformIndexHtml(html) {
      const schema = SchemaBuilder.website({ name: "My App", url: "https://example.com" });
      return html.replace("</head>", `<script type="application/ld+json">${JSON.stringify(schema)}</script></head>`);
    },
    async closeBundle() {
      fs.writeFileSync("dist/robots.txt", RobotsGenerator.allowAll({ sitemapUrl: "..." }));
      fs.writeFileSync("dist/llms.txt", await new LLMSTextGenerator({ /* ... */ }).generate());
    },
  };
}

This is the recommendation, not just an option — injecting JSON-LD into the built index.htmlreaches crawlers that never execute JavaScript. Client-side injection below doesn't.

@unhead/vue v3 gotcha: createHead moved to the /client subpath (import { createHead }from "@unhead/vue/client") — the root-level export still exists but is deprecated and slated for removal in v4. useHead() itself is unchanged.

App.vue (client-side, JS-executing crawlers only)
<script setup lang="ts">
import { useHead } from "@unhead/vue";
import { SchemaBuilder } from "ai-visibility/schema";

useHead({ script: [{ type: "application/ld+json", innerHTML: JSON.stringify(
  SchemaBuilder.product({ name: "Example Product", price: 29 })
) }] });
</script>

React (Vite SPA)

Same shape as Vue, same no-server constraint, verified against Vite 8.2 / React 19.2 — the vite.config.ts plugin is identical in structure (swap @vitejs/plugin-vue for @vitejs/plugin-react). No head-management library is needed for client-side schema — a plain element works, since JSON-LD doesn't have to live in <head> (Google's structured-data docs explicitly allow it anywhere in the document):

ProductPage.tsx
<script
  type="application/ld+json"
  dangerouslySetInnerHTML={{ __html: JSON.stringify(schema) }}
/>

Same caveat as Vue: this only reaches JS-executing crawlers — the build-time transformIndexHtml injection above is what covers the rest.

React Router (framework mode)

React Router in framework mode (current as of React Router 8) hands you a request object server-side and lets you return a Response — the same pattern applies to Remix, Astro with a server adapter, TanStack Start, and Express + React SSR.

Bot detection in the root loader (runs for every request — there's no separate middleware layer in framework-mode React Router the way Express has one):

app/root.tsx
import { AIBotDetector } from "ai-visibility/detector";

const detector = new AIBotDetector();

export function loader({ request }: Route.LoaderArgs) {
  const bot = detector.detect(request.headers.get("user-agent") ?? "");
  return { isAIBot: Boolean(bot), botName: bot?.name ?? null };
}

Generated files as resource routes (a route module with only a loader, no default component, returning a plain Response):

app/routes/robots-txt.ts
import { RobotsGenerator } from "ai-visibility/generators";

export function loader() {
  return new Response(RobotsGenerator.allowAll({ sitemapUrl: "..." }), {
    headers: { "content-type": "text/plain" },
  });
}

A gotcha unrelated to ai-visibility: serving the production build without NODE_ENV=production set explicitly for both the build and serve steps produces TypeError: dispatcher.getOwner is not a function — a React dev/prod build mismatch in the server bundle. Set NODE_ENV=production for both react-router build and react-router-serve to fix it. See Troubleshooting.

Full definitions for every term above are in the GEO glossary, and the migration guide covers what changed getting here from 0.2.x/0.3.x.