agents.txt — a proposed web standard for AI agents

Source: Dev.to

Introduction

The web has robots.txt, which has been around since 1994 and answers one question well: can you look at this?
AI agents, however, do more than just look. They book flights, submit forms, call APIs, authenticate as users, and transact on behalf of people. There is currently no standard for any of these actions.

To fill this gap, a proposal for agents.txt has been drafted. By placing a file at https://yourdomain.com/agents.txt, a site can tell agents what they can do, how to do it, and under what terms.

Sample agents.txt

Site-Name: ExampleShop
Site-Description: Online marketplace for sustainable home goods.

Allow-Training: no
Allow-RAG: yes
Allow-Actions: no
Preferred-Interface: rest
API-Docs: https://api.exampleshop.com/openapi.json
MCP-Server: https://mcp.exampleshop.com

[Agent: *]
Allow: /products/*
Allow: /search
Disallow: /checkout

[Agent: verified-purchasing-agent]
Allow: /checkout
Auth-Required: yes
Auth-Method: oauth2
Allow-Actions: yes

Why agents.txt?

They are complementary — sites should have both.

Draft and Feedback

• Draft version: v0.1
• License: CC BY 4.0
• Repository:

Feedback, pushback, and contributions are welcome.

Open Questions

1. Agent identity verification – How do you prove an agent is who it claims to be?
2. Capability vocabulary – Should capabilities use a controlled vocabulary, or free‑form strings?
3. Integration with MCP – As MCP matures, how tightly should this integrate with it?

What do you think?