Do I need llms.txt AND robots.txt?

Yes, and they do different jobs. robots.txt is enforced by convention — well-behaved crawlers check it before fetching anything, and it controls access. llms.txt isn't enforced at all — nothing requires a crawler to read or respect it, and it's a content index, not an access-control file. One doesn't substitute for the other.

Does Google use llms.txt?

Google's own public guidance has stated that Google Search doesn't need a special file like llms.txt to understand a site's content — it already crawls and renders full pages. That guidance is specific to Google; it isn't a statement about how other AI vendors treat the file.

What's the difference between llms.txt and ai.txt?

llms.txt is a content index — pages, titles, summaries — meant to help an AI model navigate a site without crawling everything. "ai.txt" is a less settled idea: depending on which write-up you read, it describes either an access-permission file shaped like robots.txt, or an unrelated proposal. See the ai.txt section below for exactly where things stand and why that ambiguity matters.

llms.txt Explained: The robots.txt for AI — What It Is and How to Add It

Q: Is llms.txt an official standard?

No single body governs it, and adoption by AI vendors is partial and unconfirmed — it's a community proposal, not a W3C or IETF standard. Anthropic and Perplexity have shown informal, documented interest; no major vendor has published a firm commitment to always read it. Treat it as a low-cost, forward-looking addition, not a guarantee.

llms.txt is a plain-text file, published at a site's root (/llms.txt), that gives an AI model a short, Markdown-formatted index of a site's content — instead of making it discover everything by crawling link by link. It's easy to describe as "robots.txt for AI," but that comparison undersells how different the two files actually are in what they control.

What llms.txt actually is

The format is simple: a # Site name heading, a one-line description, then a list of pages grouped under headings, each with a title and optional link and summary. A minimal example:

# CrawlPod

> AI visibility toolkit — make websites discoverable and citable by AI answer engines.

## Documentation
- [Docs](https://crawlpod.com/docs): Install, quickstart, and API reference
- [CLI reference](https://crawlpod.com/docs/cli): audit, lint, and CI-gate commands

## Blog
- [What is GEO?](https://crawlpod.com/blog/what-is-generative-engine-optimization-geo)

Two common variants exist alongside the base file: a minimal version (just URLs and titles, no summaries — useful for large sites) and a full version with per-page summaries included directly in the file, so an AI model doesn't need to fetch every linked page just to know what it contains.

What it isn't

It isn't enforced. robots.txt works because well-behaved crawlers check it before fetching anything and respect what it says — that's a decades-old convention, not a law, but a strong one. Nothing equivalent obligates any crawler to read llms.txt at all, let alone act on it. It also isn't a ranking or citation mechanism by itself. The load-bearing work for whether an AI system can actually use a page — as covered in the full GEO guide — is crawler access, server-rendered content, and structured data. llms.txt is additive on top of those, not a substitute for any of them.

Why it matters in 2026 anyway

Even without universal vendor commitment, an index file that costs almost nothing to add and generate has a straightforward case for existing: AI crawlers that do use it get a faster, more structured path to a site's actual key pages instead of having to infer site structure from crawling alone. Anthropic and Perplexity have shown informal, documented interest in the format. Google's own public position is that Google Search doesn't need a special file like this — it already crawls and renders full pages — which is a real data point, but a Google-specific one, not a verdict on every AI vendor.

llms.txt vs. ai.txt — a real, unresolved naming collision

Worth being precise about, since a lot of SEO-blog content treats "ai.txt" as if it has one settled meaning: it currently doesn't. At least two incompatible things get called "ai.txt" — an informal convention shaped like robots.txt (User-agent: *, Allow/Disallow rules, optionally a contact/license line), and an unrelated proposal for a different kind of file entirely. No major AI vendor has confirmed reading either one. This is exactly why llms.txt — which has at least some documented, if partial, uptake — is the more defensible file to prioritize today. See the full reasoning for why the npm ai-visibility package specifically declined to ship an ai.txt generator until a single spec exists to generate against — worth flagging directly here: the sibling Python package's generators module does ship a generate_ai_txt() function, using the robots.txt-shaped interpretation specifically. The two packages currently disagree with each other on this point; if you're choosing between them for this one file, check each package's own docs rather than assuming parity.

How to add llms.txt to your site

Static file method (any site)

Create a plain text file named llms.txt following the format above and upload it to your site's document root. No framework or tooling required — this works identically on any host.

Next.js / Node.js

Using the npm ai-visibility package's LLMSTextGenerator:

// app/llms.txt/route.ts
import { LLMSTextGenerator } from "ai-visibility/generators";

export async function GET() {
  const body = await new LLMSTextGenerator({
    siteName: "Acme",
    description: "Acme makes widgets.",
    baseUrl: "https://acme.com",
    pages: [
      { url: "/", title: "Home", priority: "high" },
      { url: "/pricing", title: "Pricing", summary: "$29/month for Pro." },
    ],
  }).generate();

  return new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8" } });
}

Or, for a large site where per-page summaries aren't worth maintaining, LLMSTextGenerator.minimal() produces a URLs-and-titles-only version synchronously. See the full recipe this site itself uses — its own /llms.txt is generated from real MDX frontmatter at build time, not a hand-maintained list.

Django / Flask / FastAPI

Using the Python ai-visibility package's generate_llms_txt():

from ai_visibility.generators import generate_llms_txt
from ai_visibility.types import LlmsTxtConfig, LlmsTxtPage

content = generate_llms_txt(LlmsTxtConfig(
    site_name="Acme",
    description="Acme makes widgets.",
    pages=[LlmsTxtPage(title="Pricing", url="/pricing", content="$29/month for Pro.")],
))

See the Python docs for the Django, Flask, and FastAPI-specific ways to serve this at a real /llms.txt route.

WordPress

Manually: upload a plain-text llms.txt file to the site root via your host's file manager. A small number of WordPress SEO plugins have begun adding generation support — check whether your existing SEO plugin already covers it before adding a dedicated one. See the full WordPress AI-visibility guide for the rest of what a WordPress site needs beyond this one file.

Shopify

Worth reading before adding anything: since May 2026, Shopify serves a default /llms.txt platform-wide on every store, redirecting to a generic /agents.md. There's nothing to install to get the file — the actual work left is replacing the generic default with a real description of the store via a theme override.

How to verify your llms.txt

The direct way: visit yoursite.com/llms.txt in a browser and confirm it returns real content, not a 404 or your site's default catch-all page. The scored way: run a free scan — it checks llms.txt presence as part of the crawler-accessibility dimension, alongside robots.txt rules and structured data, in one report.

FAQ

Common questions about llms.txt are answered below.

Check whether your site has a working llms.txt and what else might be blocking AI crawlers: run a free scan — no signup required.

llms.txt Explained: The robots.txt for AI — What It Is and How to Add It

Frequently asked questions