Roadmap decisions

This page exists because publishing a reasoned "no" is rare and more useful than staying silent about it. A roadmap that only lists what shipped hides the deliberation behind what didn't — and "we looked at this and chose not to" is a different, more trustworthy claim than never having looked at all.

ai.txt generation — deferred (assessed 2026-08-05)

"ai.txt" currently describes at least two incompatible things: an informal convention shaped like robots.txt, and an unrelated 2025 domain-specific-language proposal. Most of what's written about either one online is SEO-blog content asserting a settled standard that doesn't actually exist. By contrast, llms.txt — which this package does generate — has at least informal, documented uptake from Anthropic and Perplexity; no major AI vendor has confirmed reading ai.txt in either shape.

This package has shipped incorrect output three times before (see the crawler-token and robots-default fixes in the migration guide) — releasing a generator for a format that doesn't have one settled shape yet is a good way to make a fourth. The two things being conflated under one name also means a generator would have to guess which one a caller wants, silently.

What would change this: a single specification emerging that both usages converge on, and at least one AI vendor publicly documenting that it reads the file. Until then, robots.txt and llms.txt— both demonstrated, both stable — are where this package's effort goes.

Crawler IP-range verification — deferred (assessed 2026-08-05)

User-Agent strings can be spoofed; verifying a crawler's source IP against a vendor-published range would close that gap for the vendors who publish one. Three problems stopped this from shipping:

• Incomplete coverage by design. Anthropic explicitly declines to publish IP ranges for its crawlers, naming robots.txt as the intended control surface instead. Any IP-verification system built today would cover only the vendors who do publish ranges — partial coverage that risks reading as complete.
• Rapid obsolescence. AI crawlers increasingly run on shared cloud infrastructure (AWS, GCP), where a given IP can belong to a completely different tenant next week. A shipped IP list goes stale faster than a User-Agent token does, and User-Agent tokens already require the quarterly re-verification cadence described in the crawler registry reference.
• Asymmetric failure cost.A false rejection — blocking a real crawler because its current IP isn't on a stale list — silently undermines the exact thing this package exists to help with. That failure mode is worse than the User-Agent spoofing it would guard against, which is a real but narrower risk.

What would change this:if reconsidered, IP ranges would ship as optional, explicitly-opt-in published data for the vendors who maintain stable ranges — never as a silent access denial for vendors who don't publish one.

Semantic HTML stripping, llms-full.txt/llms-small.txt generation, MDX/sitemap auto-discovery, and crawler-visit webhooks were assessed as in-scope but deferred to keep 0.5.0 focused on the CLI and the scoring redesignit depends on — those are scheduling decisions, not "no" decisions, and are targeted for a future release.