Free plan

← All guides

Agent Policy vs Bot Blocking: Declaration Meets Enforcement

Published November 14, 2026 · Reviewed November 2026 · By the Ingenious AI team

The short answer

An agent policy and a bot blocker answer the same anxiety with opposite instruments. The policy declares your terms publicly, in advance, in a form good-faith agents honour voluntarily. Bot blocking enforces — throttles, challenges, and refuses traffic mechanically, indifferent to anyone's intentions. Sites that rely on only one get half a strategy: a declaration with no teeth for the agents who ignore it, or a wall with no terms for the agents who would happily have complied.

Generate your policy — free · Validate an existing one

What is the division of labour?

The policy is the terms layer: fetchable, standard, attributable — it tells every agent what is permitted, priced, paced, and refused, and it creates the public record those terms stand on. The blocking stack is the behaviour layer: server rate limits, bot management, firewall rules, authentication walls — machinery that acts on what traffic does, not on what anyone declared. The AGENT-POLICY spec draws the boundary itself, placing enforcement explicitly out of the format's scope. That is not the format being modest; it is the format refusing to let a document pretend to be a control — the pretence that leaves sites believing a JSON file stands between them and a hostile crawler. See also what happens when an agent ignores your policy.

What each layer sees that the other cannot

The policy sees relationships. It can distinguish the partner agent from the stranger, price the catalogue for one and fence it for another, and record a refusal against a named agent — discrimination no packet-level tool can express, because packets do not carry relationships. The blocking stack sees behaviour. It can count requests per minute as they happen, spot the mirror-in-progress, and stop the traffic the policy merely refuses. An architecture that asks either layer to do the other's seeing fails predictably: policies credited with stopping power they lack, and firewalls configured to guess at terms that were never published. Mature sites wire the two together loosely — the policy states the numbers, the stack enforces numbers slightly inside them — and keep the wiring documented where both teams can read it.

The declared cap and the enforced cap

Rate limits are where the layers touch most concretely. Your policy declares rate_limit per rule — a term, as the rate limits guide explains. Your infrastructure enforces its own ceilings for everyone, declared or not. The coherent relationship between them: the declared numbers govern identified, cooperating agents, who pace themselves beneath them voluntarily; the enforced ceiling protects the service against everyone else, set above the highest declared partner pace and below the pain threshold. When an identified agent breaches its declared cap, the first response is the relationship one — the terms were stated, the breach is documented, the conversation is short. The machinery is for the unidentified and the unrepentant.

Sequencing: publish first, tune the wall second

Owners often reach for blocking first because it feels like action. The better sequence inverts it. Publish the policy first — one afternoon's work — and observe: a meaningful share of agent traffic is good-faith traffic that complies the moment terms exist, which shrinks the behaviour problem to its genuinely hard core. Then tune the enforcement stack against what remains, with the policy as the specification of what the stack is defending: these paths, these paces, these refusals. Sites that block first and declare never tend to block their own customers' agents along with the harvesters, and to discover it from the customers. Declaration is also the cheaper instrument by an order of magnitude — a file, versus a tooling contract — which is reason enough to let it go first and measure.

Worked example: a publisher rights-sizes both layers

Fictional example. The Corvid Chronicle, a fictional news publisher, began with only a bot-management contract, tuned aggressively; its support queue filled with readers whose research agents were being challenged away. It then published a policy on the publisher pattern — reading open with attribution, crawl refused — and re-tuned the stack to enforce the policy's shape: generous passage for reading-shaped traffic, hard limits at mirror-shaped volumes. Ninety days later: challenge rates on identified reader agents fell to near zero, mirror attempts continued and were stopped mechanically at the same rate as before, and the publisher's licensing team gained 3 inbound enquiries that began with the policy. The wall did not shrink. It gained a door, a sign, and a specification.

The two-layer checklist

  1. Publish the terms first. The declaration sorts good-faith traffic before you spend anything on enforcement.
  2. Let each layer do its own seeing. Policy for relationships and terms; machinery for behaviour and volume.
  3. Set the caps coherently. Declared partner paces below the enforced ceiling; the ceiling below the pain threshold.
  4. Respond to breaches relationally first. Identified agents breaching stated terms get a conversation backed by a record — machinery is for the rest.
  5. Review both layers together. The quarterly policy review checks the stack still enforces the file's current shape, not last year's.

Keep reading

Who Should Own the Agent Policy on a Team? · Publishing Your First Agent Policy: a Launch Guide · Agent Policy vs robots.txt, Compared · Spec samples · Conformance · Pricing · All guides

Frequently asked questions

If I have good bot management, do I still need a policy?

Yes, for the traffic bot management cannot judge: the good-faith agents. A blocker treats your customers' research assistant and a catalogue harvester as identical request streams until one misbehaves. The policy separates them in advance — terms for the welcome, a record for the refusal — and gives the agents who would comply something to comply with.

Can bot blocking replace a block-posture policy?

Mechanically it refuses more than the policy ever can; declaratively it says nothing. A bank that only blocks has no public statement of its posture for operators, partners, or its own record — and no door for argued exceptions. The finance pattern pairs the strongest declaration (block) with the institution's real controls, each covering the other's silence.

What happens to agents that use my site within the policy but heavily?

Within-terms heaviness is a terms problem, and the terms have a field for it: the rate limit. Revise the number, negotiate the partner's tier, or add the cache sentence that turns volume into freshness. Machinery answers abuse; arithmetic answers enthusiasm. Confusing the two is how sites end up blocking their best channel.

Do agents identify themselves to the blocking stack?

Identification on the wire is the stack's own art — user agents, behaviour, fingerprints — and the policy deliberately does not standardise it in v0.1. What the policy contributes is the reference document: when the stack flags an agent, its declared rule (or its absence) tells your team what relationship, if any, that traffic was supposed to have.

Which layer should a small site build first?

The policy, without hesitation: an afternoon, no contract, immediate effect on every good-faith agent. Add enforcement proportionate to observed harm afterwards. Small sites that invert the order usually buy machinery sized for other people's problems while leaving their actual terms unstated to the agents already visiting.

Generate a policy — free · Validate a policy · What an agent access policy is

← Back