llms.txt, robots.txt and AI Crawlers: Controlling How Assistants Use Your Content
Publishing llms.txt is cheap and mildly useful. Controlling AI crawlers is neither, and it is the decision that actually affects your content.
Two questions arrive together in every content-heavy organisation right now. Should we publish an llms.txt file? And should we let AI companies crawl us at all? The first is a small technical decision that has been badly oversold. The second is a genuine commercial policy choice that most organisations are making by accident.
What llms.txt proposes
The llms.txt proposal suggests a Markdown file at your domain root that lists your most important pages with short descriptions, giving a model a curated route into your content rather than leaving it to infer structure from navigation. An accompanying convention offers expanded plain-text versions of key pages.
The reasoning is sound: HTML pages are noisy, and clean Markdown is cheaper to process. The problem is adoption. No major assistant has publicly confirmed that it uses llms.txt for retrieval or ranking. Treat it accordingly.
So should you publish one?
Yes, with correct expectations. It takes an hour, it cannot harm you, and the exercise of deciding which forty pages represent your organisation is valuable regardless of who reads the file. What it must not do is displace the work that demonstrably matters — server-rendered content, semantic structure, structured data and authority.
The control mechanism that does work
robots.txt, formalised as RFC 9309, remains the operative instrument. Reputable AI providers document distinct user agents, and — importantly — often separate them by purpose.
| Purpose | What it does | Blocking consequence |
|---|---|---|
| Training crawler | Collects content for model training | No effect on citations; reduces future model knowledge of you |
| Live retrieval agent | Fetches pages to answer a user's question now | You disappear from answers you would have been cited in |
| Search index crawler | Feeds a conventional index that AI features draw on | Severe organic and AI visibility loss |
Conflating these three is the most expensive mistake in this area. A blanket block applied by a nervous legal team frequently removes an organisation from AI answers while its competitors are cited weekly.
A defensible crawler policy in four decisions
- Public marketing and editorial content: allow retrieval agents. This content exists to be found; being quoted is distribution.
- Training crawlers: a legitimate business choice. Allow if you want long-term model familiarity with your brand; disallow if you object on rights grounds. Document the reason.
- Gated, licensed or paid content: disallow explicitly and enforce at the edge. Never rely on a convention file to protect revenue.
- Application surfaces, search results and faceted URLs: disallow. They generate infinite low-value crawl paths and leak nothing useful.
Enforcement, because robots.txt is voluntary
robots.txt expresses a request. Well-behaved crawlers honour it; scrapers do not. If your policy has commercial consequences, enforce it in infrastructure:
- Verify claimed identity — user agent strings are trivially forged; check published IP ranges or reverse DNS where providers offer them.
- Rate limit by verified identity and behavioural fingerprint rather than by string matching.
- Apply bot management at the CDN or WAF layer, distinguishing sanctioned agents from mass extraction.
- Log AI-agent traffic separately so you can see what is actually being fetched and how often.
The better alternative to blocking: structured generosity
Organisations that gain most from AI assistants tend to do the opposite of restricting. They make the good content trivially easy to consume — clean HTML, complete structured data, explicit answers, published data tables, named authors — while keeping genuinely proprietary material behind authentication and enforcing that boundary technically. The asymmetry is deliberate: maximise citation of the content that sells you, and protect only what customers pay for.
Implementation checklist
- Audit your current robots.txt for accidental blocks of retrieval agents.
- Write per-agent directives with a comment explaining each decision, so the file survives staff turnover.
- Publish llms.txt listing your canonical pages, and keep it current.
- Add edge-level bot management with verified allow-lists and rate limits.
- Segment AI-agent traffic in your logs and review it monthly.
- Record the policy in a governance document owned jointly by marketing and legal.
The goal is not maximum restriction or maximum openness. It is a deliberate, documented position that you chose — rather than one that a default configuration chose for you.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed convention for a Markdown file at the root of a site that gives large language models a curated, clean map of your most important content. It is a community proposal, not a standard adopted by major AI vendors.
Does llms.txt actually work today?
There is no confirmed evidence that major assistants use it for retrieval. It costs very little to publish and it forces a useful exercise in identifying your canonical content, but it should not be treated as a substitute for crawlability and structured data.
How do I control AI crawlers properly?
Through robots.txt directives targeting the documented user agents of each provider, reinforced by server-level rules. robots.txt is defined by RFC 9309 and is honoured by reputable crawlers only, so enforcement needs infrastructure.
Should we block AI crawlers entirely?
Rarely. Blocking retrieval crawlers removes you from answers where you would otherwise be cited. A common position is to allow live-retrieval agents and restrict training agents.
What about paywalled or licensed content?
Restrict it explicitly, enforce at the edge rather than by convention, and consider licensing arrangements. Convention files do not protect commercially sensitive content.