Agent Policy vs robots.txt: What's the Difference?
Published November 7, 2026 · Reviewed November 2026 · By the Ingenious AI team
The short answer
robots.txt tells crawlers which paths they may fetch; an agent access policy tells AI agents which actions they may take. robots.txt has no per-agent commercial terms and no standard rate limit; agent-policy.json carries both on every rule, for named agents. The AGENT-POLICY v0.1 spec uses robots.txt as its analogy — and gives both files the same honest limit: neither enforces anything. Comparison against the v0.1 spec as published, November 2026.
How do they compare, row by row?
| Question | robots.txt | agent-policy.json |
|---|---|---|
| Who is it talking to? | Crawlers and bots that fetch pages | AI agents that act — browse, shop, book, call APIs |
| Where does it live? | /robots.txt | /.well-known/agent-policy.json |
| What is the unit of control? | Paths: which URLs may be fetched | Agents and actions: who may do what |
| How is the audience named? | User-agent strings, path patterns | Identifiers in vendor/agent form, e.g. openai/agent, plus generic/* |
| Commercial terms? | None — no such field | Per rule: free, paid-tier, or contact |
| Rate limit? | No standard field | Per rule: rate_limit.requests_per_minute |
| Data-use terms? | None | Plain-language data_use_terms: retention, resale, training |
| Does it enforce? | No — a declaration crawlers may ignore | No — a declaration agents may ignore; enforcement is out of scope |
Why isn't robots.txt enough for agents?
Because the questions changed. A crawler's decision is fetch or don't fetch, page by page — exactly what robots.txt expresses. An agent's decisions are transactional: may it put an item in a cart, complete a checkout, submit a form on a user's behalf, call your API, crawl the whole site, quote your article, train on your catalog? None of those are paths. The spec's store sample makes the gap concrete: one agent may shop but not checkout; another may quote only with a link back; everyone else must make contact first and stay under 10 requests per minute. There is no robots.txt syntax for any of that — the file was never given fields for terms, rates, or data use.
Do you still need robots.txt?
Yes, for its own job. The two files coexist: robots.txt keeps governing crawler fetching, and the agent policy governs agent operation — AgentGate's own site serves both, and its published policy (conditional by default; browse, read, search, summarize allowed; checkout, submit-form, and api-call not) does not touch what its robots.txt says to crawlers. If you are new to the format, read what an agent access policy is first, then how to tell agents what they may do for the five-step method. The wider Ingenious AI fleet is at getingeniousai.com.
Frequently asked questions
What is the difference between agent-policy.json and robots.txt?
robots.txt speaks to crawlers about paths: which parts of a site a crawler may fetch. An agent access policy speaks to AI agents about actions: which named agents may operate, which actions they may take (browse, shop, checkout, api-call), on what commercial terms, and at what requested rate limit. Different audience, different unit — paths versus agents and actions.
Does an agent policy replace robots.txt?
No. They answer different questions and a site can publish both: robots.txt for crawler fetching, agent-policy.json for agent operation. The AGENT-POLICY spec itself uses robots.txt only as an analogy — a machine-readable declaration of the site's wishes — and states the same honest limit for both: neither file enforces anything by itself.
Can robots.txt express commercial terms or rate limits per agent?
No. robots.txt has no field for commercial terms and no standard per-agent rate limit. An agent policy carries commercial_terms (free, paid-tier, or contact) and an optional rate_limit in requests per minute on every per-agent rule, plus plain-language data_use_terms covering retention, resale, and training.
How does each file identify who it is talking to?
robots.txt addresses crawler user-agent strings and path patterns. An agent policy addresses agent identifiers in vendor/agent form — the v0.1 registry lists openai/agent, meta/muse, xai/grok, google/gemini-agent, anthropic/claude-agent, and perplexity/agent, plus generic/* as the catch-all. The registry is non-exhaustive: an unknown identifier earns a validator warning, never an error.
Which one should I publish first?
Publish robots.txt if crawler fetching is your question; publish an agent policy if agent behaviour is. If AI agents can shop, book, or call APIs on your site, robots.txt cannot express your answer at all — generate a free agent-policy.json, validate it, and serve it at /.well-known/agent-policy.json.