robots.txt is an enforced, decades-old standard that tells crawlers which paths they're allowed to fetch — well-behaved crawlers (including every major AI crawler) check it before requesting anything. llms.txt is a newer, unenforced convention: a plain-text Markdown index of a site's content, meant to give an AI model a shortcut to understanding what a site contains without crawling every page. They do different jobs, and one doesn't substitute for the other.
Side by side
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Access control — what may a crawler fetch | Content index — a summary of what exists |
| Format | Line-based directives (User-agent, Disallow, Allow) | Markdown, starting with a # Site name heading |
| Enforcement | Followed by every major crawler (search and AI) as a matter of convention and, for most, policy | Not enforced by anything — a crawler can ignore it entirely |
| Adoption | Universal; has existed since 1994 | Partial and largely unconfirmed by AI vendors |
| Location | /robots.txt (fixed) | /llms.txt (proposed convention, same pattern) |
| What happens if it's missing | Crawlers default to fetching everything | Nothing breaks — it's optional |
The honest framing on llms.txt
No major AI engine has publicly confirmed that it reads llms.txt to decide what to cite, and Google's own guidance has been that site owners don't need special files for AI Search to work. That doesn't make it worthless — it costs little to add, and if adoption grows, having one already in place costs nothing extra. Treat it as low-cost and forward-looking, one part of a bundle, not a solution on its own. The load-bearing work for AI visibility is elsewhere: crawler access (which is enforced, via robots.txt), server-rendered content, and structured data.
Do you need both?
Yes, but for different reasons. robots.txt is the more consequential of the two — get it wrong (block AI crawlers you meant to allow) and nothing else in this comparison matters, because the crawler never fetches the page at all. llms.txt is a minutes-long addition once the actual content and structure are in good shape.
On Shopify, WordPress, and Next.js
robots.txt is editable on all three, though how varies: a robots.txt.liquid template on Shopify (theme-dependent), an SEO plugin's editor on WordPress, or a robots.ts/Route Handler on Next.js. llms.txt isn't generated automatically by Shopify today; a small number of WordPress SEO plugins have started adding it; on Next.js, npm install ai-visibility and use LLMSTextGenerator from ai-visibility/generators.
Related
See the Agentic Browsing glossary entry for how llms.txt shows up as one signal in Lighthouse's newer audit category, or run a free scan to check both files on a real URL.