Your API Needs an llms.txt File — Here's How to Write One and Why Agents Will Read It
Source: Dev.to
What llms.txt Is
Standardized by Jeremy Howard (Answer.AI / fast.ai) at llmstxt.org. Two files are defined:
| File | Purpose |
|---|---|
/llms.txt | Concise index — what this site is, what’s available, where to find it |
/llms-full.txt | Complete documentation in one file — agents ingest this |
The concise version is the discovery layer. The full version is the consumption layer. Together they replace “agents try to browse your docs site and get lost.”
The Concise Index (llms.txt)
Minimal. Just tells agents what’s available and where:
# Stripe API
> Payment infrastructure for the internet.
> API: https://api.stripe.com
## API Reference
- [Authentication](https://docs.stripe.com/api/authentication.md)
- [Charges](https://docs.stripe.com/api/charges.md)
- [Webhooks](https://docs.stripe.com/api/webhooks.md)
## Guides
- [Quickstart](https://docs.stripe.com/quickstart.md)
- [Testing](https://docs.stripe.com/testing.md)
## SDKs
- [Python](https://docs.stripe.com/sdks/python.md)
- [Node.js](https://docs.stripe.com/sdks/node.md)That’s it. No styling, no navigation—just links to Markdown documents. Agents parse this and know exactly what’s on your site.
The Full Reference (llms-full.txt)
This is what agents actually read: all your documentation in one file, with a table of contents at the top.
# Stripe API — Full Reference
## Quickstart
[Full quickstart content...]
## Authentication
API key format, scopes, rotation...
## Endpoints
### Charges
POST /v1/charges — request/response examples...
### Customers
[Full customer API docs...]
## Error Handling
Error codes, retry patterns...
## SDK Examples
Python, Node, Ruby...Critical rule: The full file must be under ~50 KB characters. Agents have context limits; if it’s too long they’ll truncate or ignore it.
How to Serve It
Static site (recommended)
Drop llms.txt and llms-full.txt in your site root. They’re plain Markdown files. Serve them with the header:
Content-Type: text/markdownAPI (for dynamic content)
Generate the files programmatically from your API docs. Return them via an endpoint that respects the request header:
Accept: text/markdownAgents can request the format they need.
Why This Matters Now
Google, OpenAI, Anthropic, and others are training their agents to recognize llms.txt as a discovery mechanism. It’s the robots.txt moment for AI agents—early adopters get indexed first.
Adding llms.txt brings three benefits:
- Discovery – Agents find your content without human intervention.
- Efficiency – Agents read one structured file instead of crawling dozens of pages.
- Positioning – You join the agent discovery ecosystem before competitors do.
The “Works With Agents” Angle
A broader vision is a certification for agent‑compatible products.
- Works With Agents Ready – Your product has
llms.txt+ OpenAPI spec. Agents can discover and use it. - Works With Agents Certified – Your product has been tested with real agents; pitfalls are documented and skills exist.
The first tier is free and self‑serve—just add the files. The second tier is verified by dedicated infrastructure.
I built the Works With Agents infrastructure — FactBase, Skill Registry, Pitfall Registry — with llms.txt as the primary discovery mechanism. Every domain (workswithagents.com, .dev, .io) serves both llms.txt and llms-full.txt. If you’re building agent‑facing tools, do the same.