How to Welcome AI Agents on Your Site
Published November 14, 2026 · Reviewed November 2026 · By the Ingenious AI team
The short answer
Welcoming AI agents is a posture you publish: a default_policy of allow, rules that grant the reading verbs generously, and — this is what separates a welcome from an oversight — explicit fences around the actions and data uses you do not grant. The blog sample is the pattern in miniature: an open default, attribution expected in the data-use terms, and a single named agent switched off entirely. Openness with its boundaries written down is the version agents can safely accept.
What does a welcome consist of?
Three layers, all in the file. The posture: an allow default says the unnamed world may operate. The terms of the welcome: a generic/* rule giving strangers actual conditions — actions, pace, data-use terms — rather than a bare verdict, so the welcome has a shape. The fences: the disallowed verbs and after-life clauses that mark where the welcome ends. Owners sometimes fear the fences dilute the welcome; agents read them the opposite way. A policy with no fences makes a careful agent more cautious, because it cannot tell generosity from inattention — and inattention gets revised without notice, while fences imply an author who will still be there next quarter.
Who a welcome serves
Welcomes suit sites whose content gains by being worked with: publishers whose articles answer questions better with attribution than without readers, documentation and reference sites whose purpose is to be used, stores running the browse-not-checkout split at an open posture, public-interest and government information meant to travel. The common economics: the marginal cost of an agent's read is near zero, the marginal value of being the source agents cite is not, and the welcome is the instrument that collects the second while bounding the first. The blog sample's owner states the exchange plainly — agents are welcome that attribute and link back — which is a welcome with its price written in the data-use terms rather than the commercial ones.
Writing the generous rule
The reading verbs are the welcome's vocabulary: browse, read, search, summarize, quote. Grant them on free terms at a pace your site serves comfortably — the samples' 30 per minute for readers is a sane reference. Then the fences, chosen by asking what openness must not mean: usually crawl-full-site refused or priced (reading is welcome; mirroring is a different offer), checkout and submit-form refused where commitment and state-change live, and training addressed in data-use terms if your content's after-life matters to you. The result reads as hospitality with house rules — which is what a welcome that survives contact with reality looks like.
Maintaining a welcome
Welcomes decay in one direction: silently, into neglect. The counters are the standard maintenance habits with the emphasis flipped — instead of watching for abuse of a refusal, watch the value of the grant. Which agents actually come under the welcome? What do operators ask for that the file refuses by accident — a verb your journey needs that you never listed? Does the attribution you priced the welcome at actually arrive? A quarterly review that asks those questions keeps an open policy an instrument instead of a monument, and the cadence guide supplies the rhythm. Welcoming is a strategy, and strategies get reviewed.
Worked example: a documentation site opens its doors, with house rules
Fictional example. Fieldnotes Docs, a fictional documentation site for surveying software with 1,900 reference pages, publishes an allow default and a catch-all granting the five reading verbs free at 40 requests per minute, with attribution in the data-use terms and crawl-full-site disallowed (bulk access is a separate, priced conversation held at contact on one named rule for a partner's offline index). Six months in, its referral logs show agent-originated visits becoming a steady discovery channel for new survey firms; the support inbox shows 2 attribution disputes, both resolved by pointing at the published terms. The owner's only regret is sequencing: the bulk-access rule should have been written on day one, before the first mirror request arrived as a surprise instead of an enquiry.
The welcome checklist
- Publish the posture. An
allowdefault states the welcome at the top level, for every agent you do not name. - Give strangers conditions. A
generic/*rule with the reading verbs, a comfortable pace, and your attribution terms. - Write the fences in the same pass. Refuse or price the bulk and committing verbs —
crawl-full-site,checkout,submit-form— explicitly. - Price the after-life. Data-use terms carry the welcome's real exchange: attribution, retention limits, training position.
- Review the value quarterly. Who comes, what they ask for, whether your price arrives — the welcome is a strategy, reviewed like one.
Keep reading
Agent Policy for Price-Comparison Agents · How Often Should You Update Your Agent Policy? · How to Tell AI Agents What They May Do · Spec samples · Conformance · Pricing · All guides
Frequently asked questions
Does an allow default mean anything goes?
No. The default is a posture, not the whole policy. Rules bind named agents, a generic/* rule binds strangers to stated conditions, action lists fence the committing verbs, and data-use terms fence the after-life. A welcome with all four layers written is more controlled than a block nobody maintains — control lives in the detail, not the default alone.
Why would a site welcome agents at all?
Because agents are becoming a reading and shopping channel their users choose. A welcome puts the site inside that channel on its own terms: cited with attribution, compared with its prices stated, read at a pace it set. The alternative postures — conditional sorting, block refusal — suit other economics; the welcome suits content and catalogues that gain by being used.
What did the blog sample's welcome include?
An allow default, one rule switching a single named agent off entirely, and data-use terms welcoming agents that attribute and link back while refusing bulk scrapers that copy full articles. Three elements: openness, one argued exception, and the after-life priced in attribution. Most real welcomes are variations on exactly that trio.
Should a welcome include a rate limit?
Yes, for the same reason a house party has a capacity: the number protects the welcome itself. An unstated pace invites the fastest agent to define it. Set the catch-all at a comfortable working pace — the samples use 10 to 30 per minute for readers and strangers — and raise it for named partners as relationships justify.
Can a welcome be withdrawn?
It is your document: revise the default, tighten the rules, move last_updated, validate. Withdrawal is a commercial decision agents will read as such, so communicate it like one — operators whose agents built on your welcome deserve the same notice a pricing change would get. The file changes in a day; the reputation changes at its own speed.
Generate a policy — free · Validate a policy · What an agent access policy is