FastAPI integration

AIVisibilityMiddleware from ai_visibility.middleware.fastapi is a Starlette BaseHTTPMiddleware subclass, added with app.add_middleware()— it works for any Starlette-based app, not just FastAPI, since it only depends on Starlette's request/response types.

Install

pip install ai-visibility[fastapi]

Adds FastAPI (≥0.68) and Starlette (≥0.19) as dependencies. The core package remains dependency-free otherwise.

Middleware setup

main.py

from fastapi import FastAPI
from ai_visibility.middleware.fastapi import AIVisibilityMiddleware

app = FastAPI()
app.add_middleware(AIVisibilityMiddleware, optimize=True, inject_schemas=True)

The middleware reads request.headers.get("user-agent") to detect a crawler and response.headers.get("content-type") to skip anything that isn't text/html. For a detected crawler on an HTML response, it drains the response's body_iterator, processes the full body, and returns a new Response with a corrected content-length — for every other request (non-crawler, or non-HTML), the original streaming response passes through unmodified.

Configuration options

AIVisibilityMiddleware(
    app: Starlette, *,
    optimize: bool = True,
    inject_schemas: bool = False,
    schemas: list[dict[str, Any]] | None = None,
    log_visits: bool = True,
    on_detect: Callable[[CrawlerInfo, str], None] | None = None,
    optimize_options: OptimizeOptions | None = None,
    analytics: CrawlerAnalytics | None = None,
)

Parameter	Type	Default	Description
`optimize`	`bool`	`True`	Serve an optimized HTML snapshot to detected crawlers.
`inject_schemas`	`bool`	`False`	Inject `schemas` as JSON-LD for detected crawlers.
`schemas`	`list[dict] \| None`	`None`	JSON-LD schema dicts to inject when `inject_schemas` is set. Static, set once at middleware construction — see below for per-route schemas.
`log_visits`	`bool`	`True`	Log detected crawler visits.
`on_detect`	`Callable[[CrawlerInfo, str], None] \| None`	`None`	Called with `(crawler, path)` on every detected visit — synchronous, not awaited (see async notes below).
`optimize_options`	`OptimizeOptions \| None`	`None`	Fine-grained stripping control — see the optimizer reference.
`analytics`	`CrawlerAnalytics \| None`	`None`	Pass a `CrawlerAnalytics` built with a custom `AnalyticsStore` to persist visits. Defaults to a fresh in-memory tracker.

Using schema builders with FastAPI dependency injection

Schema builders are plain functions, so they compose naturally with FastAPI's Depends()— build the schema as part of a dependency that also fetches the page's data, then inject it into an HTMLResponse:

routers/products.py

from fastapi import APIRouter, Depends, Request
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from ai_visibility import product_schema, inject_schemas

router = APIRouter()
templates = Jinja2Templates(directory="templates")

async def get_product_schema(product = Depends(get_product)) -> dict:
    return product_schema(
        name=product.name,
        description=product.description,
        price=product.price,
        price_currency="USD",
        availability="InStock" if product.in_stock else "OutOfStock",
    )

@router.get("/products/{slug}", response_class=HTMLResponse)
async def product_detail(request: Request, product = Depends(get_product),
                          schema: dict = Depends(get_product_schema)):
    html = templates.get_template("product_detail.html").render(
        {"request": request, "product": product}
    )
    return HTMLResponse(inject_schemas(html, [schema]))

Serving llms.txt and robots.txt

main.py

from fastapi.responses import PlainTextResponse
from ai_visibility import generate_llms_txt, generate_robots_txt
from ai_visibility.types import LlmsTxtConfig, LlmsTxtSection, LlmsTxtLink, RobotsTxtConfig

@app.get("/llms.txt", response_class=PlainTextResponse)
async def llms_txt():
    return generate_llms_txt(LlmsTxtConfig(
        title="Acme",
        summary="Acme makes widgets.",
        sections=[LlmsTxtSection(name="Docs", links=[
            LlmsTxtLink(title="Getting started", url="https://acme.com/docs"),
        ])],
    ))

@app.get("/robots.txt", response_class=PlainTextResponse)
async def robots_txt():
    return generate_robots_txt(RobotsTxtConfig(
        block_training_bots=True,
        sitemap_urls=["https://acme.com/sitemap.xml"],
    ))

PlainTextResponse as the route's response_class sets the correct Content-Type; returning the generated string directly is sufficient — no need to construct the response object by hand.

Async support notes

The 0.5.0 core library is entirely synchronous — optimize_html(), score_page(), analyze_content(), and every schema/ generator function are plain sync functions with no I/O of their own (they operate on an HTML string already in memory, not a network fetch), so there is no async variant to reach for and none needed. AIVisibilityMiddleware itself is async (async def dispatch(), per Starlette's BaseHTTPMiddleware contract) and calls the sync core.process_html()directly inline — for the HTML sizes this operates on, that's not a blocking-event-loop concern in practice, but if you're optimizing exceptionally large HTML documents (megabytes, not kilobytes) on a latency-sensitive event loop, run optimize_html() in a thread instead: await anyio.to_thread.run_sync(optimize_html, html). on_detect is also called synchronously and is not awaited even if you pass an async function — pass a sync callback, or fire-and-forget an async task from inside it yourself.

Testing with TestClient and AI crawler headers

test_main.py

from fastapi.testclient import TestClient
from main import app

client = TestClient(app)
GPTBOT_UA = "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2"

def test_strips_script_tags_for_gptbot():
    response = client.get("/", headers={"User-Agent": GPTBOT_UA})
    assert "<script>" not in response.text

def test_regular_visitor_untouched():
    response = client.get("/", headers={"User-Agent": "Mozilla/5.0 (normal browser)"})
    assert "<script>" in response.text

TestClientis synchronous and runs the app through Starlette's real ASGI stack, including AIVisibilityMiddleware's dispatch()— no separate async test setup is needed for this middleware specifically, even though it's implemented with async def.

See the middleware API reference for the shared AIVisibilityCore both Django and Flask adapters also wrap, and the Django guide / the Flask guide for the other two frameworks.