Should You Block AI Crawlers? Separate Search Visibility from Model Training

“Should we block AI crawlers?” sounds like a technical question. For a marketing team, it is really a distribution decision.

Imagine a buyer asking an assistant to compare insurance agencies, ecommerce suppliers, or software vendors. You may want the assistant to find your public service pages and describe the business accurately. You may also prefer that the same pages not be collected for future model training. Those are not necessarily the same permission.

My position: do not use one blanket “block AI” rule as a substitute for deciding what each public page is for. Separate search discovery, user-requested retrieval, and model training. Then decide which pages you actually want buyers to find.

Start with three different jobs, not a list of bot names

A search crawler discovers and indexes public material for answers or links. A user-directed fetch happens when someone asks an assistant to open a particular page. A training crawler gathers material that may help develop future models. The vendors do not implement these jobs identically, so a policy copied from another company's robots.txt is a poor starting point.

OpenAI's crawler documentation says its OAI-SearchBot and GPTBot settings are independent: a site can allow the search bot while disallowing the bot used for potential foundation-model training. It also distinguishes ChatGPT-User, which may fetch a page at a user's request and is not the automatic search crawler. OpenAI's publisher FAQ says sites that want their content included in ChatGPT search summaries and snippets should not block OAI-SearchBot.

Anthropic similarly documents separate ClaudeBot, Claude-SearchBot, and Claude-User roles. Its explanation says restricting Claude-SearchBot may reduce visibility and accuracy in user search results, while restricting ClaudeBot signals exclusion of future material from its training datasets. That is a different commercial tradeoff from “let every bot in” or “block every bot.”

Google makes the blanket-block mistake especially costly

Google's AI Overviews and AI Mode are part of Search. To be eligible as a supporting link, Google says a page must be indexed and eligible to appear in Search with a snippet; there are no extra technical requirements just for those AI features. Blocking Googlebot can therefore affect ordinary Search as well as AI-related exposure.

Google-Extended is a separate control for some other Google AI training and grounding uses. Google's crawler documentation explicitly says Google-Extended does not affect inclusion in Google Search or act as a Search ranking signal. Treating Googlebot and Google-Extended as interchangeable would confuse two separate decisions.

None of this means opening access guarantees an AI mention, citation, qualified visit, or sale. Google explicitly says meeting eligibility requirements does not guarantee crawling, indexing, or serving. Access is a prerequisite to inspect, not a performance promise.

A sensible policy starts with the buyer's page

Before anyone edits crawler rules, choose a small set of pages a real buyer would need. For a 30-person insurance firm, that might be a current services page, locations, the team and licensing information, and a clear explanation of how to request a quote. For an ecommerce business, it might be category pages, accurate product details, shipping and returns, and customer support. These are examples of public buyer information, not a recommendation to expose private records.

Ask three questions for each page:

1. Would we want a buyer to find and quote this page? If yes, do not accidentally make the page invisible to the search systems that matter to that buyer.
2. Is the information current and safe to make public? If no, fix the page or its access control first. A crawler rule is not a privacy system.
3. What use are we trying to limit? Search discovery, user-directed access, and model training may call for different controls.

This is also an ownership question. Marketing can define the buyer questions and pages. The site owner or developer should verify what the server actually returns to crawlers, including the robots file, page response, meta directives, and any CDN or security layer. Legal or policy owners should weigh training-use preferences where relevant. No one should silently change a sitewide rule from a marketing checklist.

Check access before diagnosing an “AI visibility” problem

A practical audit can be short:

• List five commercially important public URLs and the buyer question each should help answer.
• Read the live robots.txt for the actual host and review any page-level noindex or snippet controls.
• Check whether a login wall, blocked CDN rule, broken redirect, or error page prevents retrieval even when robots rules appear permissive.
• Use Google Search Console's URL Inspection for Google indexing questions; Google recommends it for diagnosing what Googlebot received.
• Record the current crawler policy and the reason for each exception before changing it.

Be precise about what the audit proves. A successful fetch proves access at that moment. It does not prove that an assistant indexed the page, used it in a particular answer, or will recommend the business. Conversely, one missing mention does not prove a crawler block. The page could be accessible but unhelpful, outdated, or simply not selected for that question.

Don't mistake a crawl rule for a confidentiality control

Google's Googlebot guidance draws an important line: robots.txt can tell a crawler not to request a page, but that alone does not necessarily prevent a URL from appearing in results. If material must not be public, use actual access controls or remove it from the public web. If a page should be public but not indexed, use an appropriate indexing control and make sure the relevant crawler can read it.

The same caution applies to AI search. An “AI-blocking” rule is not a substitute for securing customer data, private pricing, draft documents, or internal files.

Make the decision, then measure what happened

For most businesses trying to be found, the first move is not a new content factory. It is a clear policy for a few high-value pages: what is public, what search systems can access it, who owns the facts, and what use the business prefers to restrict. Then test real buyer questions and compare the answers with the pages those systems could reasonably reach.

That gives the team a useful sequence: permission, accurate page, observable answer, qualified visit, business outcome. Skip the first two and a low visibility score becomes a mystery. Stop at the third and it becomes a vanity metric.

Start with one observation: run a real buyer question through the free AI visibility checker. Treat the result as directional evidence, then inspect the relevant public page and crawler access with your site owner. One check cannot certify crawlability across every AI platform or guarantee future inclusion.