Blocking vs Allowing AI Agents Per Path: A Working Policy

An agent's journey through your site has three stages, and each stage is where a different kind of path decision gets made. For discovery, record whether the URL appeared in a sitemap, internal link, or search result. For the fetch, request the URL and record whether the server's path rule serves or withholds the content. Third, action: if the content is a page an agent can act on, such as a product page or a cart endpoint, the agent may do something beyond reading, like adding an item or advancing to checkout.

Cloudflare states that for all new domains on Cloudflare, AI training crawlers would be blocked by default unless domain owners chose otherwise ↗. Apply that statement only to the named platform default and crawler class. Check the actual rule for each agent and path instead of extending one default to unrelated answer-engine crawlers.

The practical consequence is that stage two, the fetch, is the only stage where a per-path rule actually operates. You cannot control discovery once a URL is public and linked; you cannot control what a permitted agent does with content once it has fetched it. What you can control is whether the fetch succeeds for a given agent on a given path, which is why every policy in this guide is built around that one lever.

A per-path policy written without groundwork tends to either block a path you needed open or leave open a path you meant to protect. Both mistakes are expensive to find later, because the symptom, a missing citation or an unexpected transaction, shows up weeks after the rule went live. Confirm each of the following before you touch a configuration file.

You need an inventory of paths worth differentiating, not a blanket domain rule. Group your site into a small number of categories: public informational content, gated or paywalled content, transactional endpoints like cart and checkout, and internal or admin paths that should never be crawled by anything. Most sites can do this in an afternoon using existing analytics; the categories rarely exceed five or six.

You also need visibility into which agents are actually requesting your paths today, before you decide what to do about any of them. Guessing at your current exposure is the single most common cause of a policy that breaks something on launch.

Related guide: How to Configure robots.txt for AI Crawlers in 2026-Without Guessing

This is a reproducible template, not a one-off fix: run it once per path category, and rerun it whenever you add a new category of path or a new class of agent shows up in your logs. The sequence below assumes you have already completed the prerequisites and have a categorized path list in hand.

The first three steps classify and configure; the last two confirm the configuration matches what you intended before you consider the rule live. Complete the confirmation again after the next site restructure and record the result beside the active rule.

Transactional paths deserve particular care in step two. Shopify's changelog on WebMCP support ↗ describes tools that let agents search a catalog, manage a shopper's cart, and go to checkout on the shopper's behalf, all in the same tab the shopper is using, and notes that everything an agent does happens on the shopper's live session. A path that supports that kind of interaction is not a candidate for a simple block; it is a candidate for a scoped allow that matches what the integration was built to do, with the risk boundary drawn around specific actions rather than around the whole path.

A rule that looks correct in a configuration file is not the same as a rule that behaves correctly under a real request. Verification means sending an actual request, or as close to one as you can construct, against each path and category, and checking the response against what you intended rather than what you assume happened.

Build a small test matrix: one row per path category, one column for the agent identity you are testing against, and a pass or fail outcome defined in advance. A pass means the response code and body match the intended posture for that agent and path. A failure needs a defined next action, not just a note that something went wrong; the table below gives a starting structure you can extend with your own path categories.

One commenter wrote: A decent LLM should be able to handle browsing the site, if needed it can use the sitemap. ↗

Treat that as one engineer's view, not evidence about the agent classes in your logs. Transactional and internal paths still need a rule you have tested yourself.

For every test, record the timestamp, declared agent identity, target URL, response status, relevant response headers, and a hash of the returned body. Keep that evidence beside the rule version and expected outcome so the next operator can reproduce the result without guessing what changed.

Rerun this verification matrix after any site restructure, any new crawl-control rule, and any change to a transactional integration like a cart or checkout flow, since any of the three can silently move a path from one category to another without anyone updating the rule that governs it.

Related guide: How to Use Server Logs to Detect Unannounced AI Crawlers

Access control and citation measurement answer different questions, and it is worth being precise about which approach gives you which answer. Blocking and allowing rules tell you what an agent can reach. They do not tell you whether an answer engine, once it has read an allowed path, actually names your brand or links back to the page. Confirming a path is open is necessary but not sufficient for confirming it is helping your citation rate.

The original paper presents Attributable to Identified Sources as a framework for assessing natural-language generation output when it pertains to the external world ↗. Use that as a separate source-support check after the access test; an access rule cannot establish whether an output is attributable to a specific source.

EdenRank records whether an answer names the brand and whether it links to the brand page, running an ongoing prompt panel and keeping a tracked run history so a team can see whether an access change on a given path actually moved citation behavior, rather than assuming it did because the rule deployed cleanly. That closes the loop the verification matrix leaves open: the matrix confirms an agent can reach a path, and the citation tracking layer confirms what happened after it did.

Cloudflare describes its own measurement release this way: We have discussed this challenge in smaller settings, and today we are excited to publish our findings as a new metric shown on the AI Insights page on Cloudflare Radar. ↗

The table below sets these approaches side by side. None of them replaces the others; a mature policy typically runs a mix of the first three plus an ongoing measurement layer, and picking one at the expense of the rest just moves the blind spot as opposed to closing it.