Roadmap decisions

This page exists because publishing a reasoned "no" is rare and more useful than staying silent about it. A roadmap that only lists what shipped hides the deliberation behind what didn't — and "we looked at this and chose not to" is a different, more trustworthy claim than never having looked at all.

ai.txt generation — deferred (assessed 2026-08-05)

"ai.txt" currently describes at least two incompatible things: an informal convention shaped like robots.txt, and an unrelated 2025 domain-specific-language proposal. Most of what's written about either one online is SEO-blog content asserting a settled standard that doesn't actually exist. By contrast, llms.txt — which this package does generate — has at least informal, documented uptake from Anthropic and Perplexity; no major AI vendor has confirmed reading ai.txt in either shape.

This package has shipped incorrect output three times before (see the crawler-token and robots-default fixes in the migration guide) — releasing a generator for a format that doesn't have one settled shape yet is a good way to make a fourth. The two things being conflated under one name also means a generator would have to guess which one a caller wants, silently.

What would change this: a single specification emerging that both usages converge on, and at least one AI vendor publicly documenting that it reads the file. Until then, robots.txt and llms.txt— both demonstrated, both stable — are where this package's effort goes.

Scope note: this decision covers the npm package specifically. The Python package (ai_visibility) does ship a working generate_ai_txt() in ai_visibility.generators— an intentional divergence between the two ecosystems' roadmaps, not an oversight on either side. If you're choosing between them and ai.txtgeneration specifically matters to you today, the Python package already has it; the reasoning above is why the npm package doesn't yet.

Crawler IP-range verification — deferred (assessed 2026-08-05)

User-Agent strings can be spoofed; verifying a crawler's source IP against a vendor-published range would close that gap for the vendors who publish one. Three problems stopped this from shipping:

• Incomplete coverage by design. Anthropic explicitly declines to publish IP ranges for its crawlers, naming robots.txt as the intended control surface instead. Any IP-verification system built today would cover only the vendors who do publish ranges — partial coverage that risks reading as complete.
• Rapid obsolescence. AI crawlers increasingly run on shared cloud infrastructure (AWS, GCP), where a given IP can belong to a completely different tenant next week. A shipped IP list goes stale faster than a User-Agent token does, and User-Agent tokens already require the quarterly re-verification cadence described in the crawler registry reference.
• Asymmetric failure cost.A false rejection — blocking a real crawler because its current IP isn't on a stale list — silently undermines the exact thing this package exists to help with. That failure mode is worse than the User-Agent spoofing it would guard against, which is a real but narrower risk.

What would change this:if reconsidered, IP ranges would ship as optional, explicitly-opt-in published data for the vendors who maintain stable ranges — never as a silent access denial for vendors who don't publish one.

Semantic HTML stripping, llms-full.txt/llms-small.txt generation, MDX/sitemap auto-discovery, and crawler-visit webhooks were assessed as in-scope but deferred to keep 0.5.0 focused on the CLI and the scoring redesignit depends on — those are scheduling decisions, not "no" decisions, and are targeted for a future release.