CrawlPodScan your site

llms.txt setup guide

llms.txt is a plain-text file at your site's root that gives AI models a structured index of your key pages. CrawlPod's free WordPress plugin generates and serves one automatically from your published content — no manual file editing required — and lets you control which content types are included.

What is llms.txt

llms.txt is a proposed plain-text convention, served at /llms.txton a site's root domain, that gives AI models a structured, machine-readable index of a site's key content — a short site summary followed by grouped links with one-line descriptions. The idea is to make it easier for a language model to find and understand the most relevant pages on a site without having to crawl and infer structure from full HTML.

It's a community proposal, not a standard published or enforced by any governing body — see the llms.txt spec for the original proposal. Support among AI crawlers and answer engines varies and is still evolving, so treat it as a useful, low-cost addition rather than something every AI system is guaranteed to read.

How CrawlPod auto-generates it

The free CrawlPod plugin generates and serves llms.txt automatically once activated, built from your site's existing published content:

  1. Activate the free CrawlPod plugin on your WordPress site.
  2. CrawlPod scans your published posts, pages, and configured taxonomies to build a structured index.
  3. It writes a summary line for your site and groups links under section headings (for example, top-level pages, recent posts, key categories).
  4. The result is served live at yoursite.com/llms.txt, regenerated automatically as you publish, update, or unpublish content — you don't need to trigger a rebuild manually.

Customization options

From the plugin's llms.txt settings screen, you can choose which post types, taxonomies, and individual pages are included or excluded from the generated file — useful for keeping internal, archived, or low-value pages out of the index while making sure your key content is represented.

On the Business tier, the Dynamic AI Promo Codes feature adds an additional promotions section to your llms.txt output, giving AI systems a structured place to find current offers alongside your regular content links.

Best practices for content

  • Use clear, descriptive page and post titles — they become the link text AI models see, so vague titles produce a vague index.
  • Keep summaries to one line each; a short, accurate description is more useful than a long one that gets truncated or ignored.
  • Keep the file current by excluding outdated or superseded content rather than leaving stale pages indexed alongside their replacements.
  • Don't just dump your whole sitemap into it — a curated list of genuinely important pages is more useful to an AI model than an unfiltered copy of every URL on the site.
Example llms.txt
# Acme Coffee Roasters
> Small-batch coffee roaster shipping fresh-roasted beans nationwide, based in Portland, OR.

## Shop
- [Shop All Coffee](https://acmecoffee.example/shop): Browse our full lineup of single-origin and blended roasts.
- [Subscriptions](https://acmecoffee.example/subscriptions): Recurring coffee delivery, pause or cancel anytime.

## Guides
- [Brewing Guide](https://acmecoffee.example/guides/brewing): How to brew pour-over, French press, and espresso at home.
- [Origin Guide: Ethiopia](https://acmecoffee.example/guides/ethiopia): What makes Ethiopian beans distinct, and how we source them.

## About
- [Our Roasting Process](https://acmecoffee.example/about/roasting): How we roast, from green bean to bag.
- [Contact](https://acmecoffee.example/contact): Reach our team directly.

See robots.txt for AI crawlers for the related question of which bots are actually allowed to fetch your content in the first place.