Robots.txt for AI Crawlers: Which Bots to Allow or Block (A Decision Guide for E-commerce Brands)
Learn how to manage robots.txt AI crawlers: distinguish search bots from training bots, decide whether to block AI bots, and verify the impact on visibility.

For most e-commerce brands, the robots.txt AI crawlers decision has a sensible default: allow AI search and retrieval crawlers, including OAI-SearchBot, Claude-SearchBot and PerplexityBot. These are the bots that decide whether your products can be cited in shopping answers. Training-focused crawlers such as GPTBot, ClaudeBot and Google-Extended deserve a separate, deliberate decision. Blocking everything may feel protective, but it is a common cause of accidental invisibility in AI search. Test every robots.txt rule, then monitor its impact on your actual AI visibility.
This guide covers how to make that decision, when to depart from the default, which robots.txt rules to use, and how to verify the result. It is a decision framework, not a blanket recommendation. A retailer selling commodity products has different needs from a brand with years of proprietary buying guides.
robots.txt AI crawlers: the trade-off between protection and discoverability 🦉
Picture a shopper typing "the best waterproof walking boots under £150" into ChatGPT, Perplexity or Gemini. The assistant returns a shortlist of four or five brands, and yours is not among them. Nothing in your analytics flags the loss, because nobody visited your site and nobody bounced. The sale simply went elsewhere.
One common cause is a robots.txt file written with good intentions. That single file now does two jobs that used to be one. It still governs search indexing, and it also governs whether your content can be fetched for live AI answers and, in some cases, used for model training. A rule that blocks "AI bots" in the abstract may switch off both at once.
The stakes are growing. Adobe reported that traffic from generative-AI sources to U.S. retail websites rose 1,300% year over year during the 2024 holiday season, and OpenAI said in February 2025 that ChatGPT had more than 400 million weekly active users. Neither figure proves that allowing a particular crawler produces recommendations, but together they show a large audience and a real referral channel. Pew Research Center's 2025 analysis of U.S. Google users found that people clicked a traditional result in 8% of visits when an AI summary appeared, against 15% without one, and clicked a link inside the summary in about 1% of visits. When fewer clicks flow through conventional results, being named inside the answer matters more.
First, a point about what robots.txt is and is not. It is the Robots Exclusion Protocol, formalised as RFC 9309, and it is a voluntary standard. The RFC states plainly that it is not a form of authorisation. Reputable crawlers honour it, but a determined scraper can ignore it, and the file itself is publicly readable. If you need to protect unpublished prices, customer data or unreleased products, use authentication and server-side access controls. Robots.txt is guidance for well-behaved crawlers, nothing more.
It still matters for generative engine optimisation, because the logic is simple: a page that cannot be fetched cannot be cited. Crawl access is necessary but not sufficient for AI visibility, and it is the one prerequisite entirely within your control.
How AI crawlers identify themselves in robots.txt
Crawlers announce themselves in the HTTP User-Agent header, and robots.txt rules match against a product token drawn from that identity. Under RFC 9309, matching on the token is case-insensitive, so GPTBot and gptbot are treated identically. In practice, use the exact documented token for each bot. A broad label like "AI bot" is one that no crawler recognises.

The table below summarises the main agents based on each operator's documentation. Treat it as a starting inventory, not a permanent reference.
| Operator | User-agent token | Primary purpose | Operator documents robots.txt handling? |
|---|---|---|---|
| OpenAI | GPTBot | Training (may improve foundation models) | Yes |
| OpenAI | OAI-SearchBot | Search index for ChatGPT search | Yes |
| OpenAI | ChatGPT-User | User-initiated fetch | Yes, but OpenAI says rules may not apply to user-initiated requests |
| Anthropic | ClaudeBot | Training and model development | Yes |
| Anthropic | Claude-SearchBot | Search features | Yes |
| Anthropic | Claude-User | User-initiated fetch | Yes; check current documentation |
| Perplexity | PerplexityBot | Discovery and indexing for search | Yes |
| Perplexity | Perplexity-User | User-triggered fetch, not used to build the index | Documented separately; do not assume robots.txt governs it |
| Googlebot | Google Search crawling | Yes | |
| Google-Extended | Control token for Gemini and Vertex AI use | Yes (a control token, not a crawler) | |
| Microsoft | Bingbot | Bing index, which underpins Copilot results | Yes |
| Apple | Applebot / Applebot-Extended | Search crawling / AI-use control token | Yes |
| Common Crawl | CCBot | Open web corpus widely used in training | Yes |
| Others | Meta-ExternalAgent, Amazonbot, Bytespider | Mixed; verify each operator's stated purpose | Verify individually |
Two clarifications save a great deal of confusion. First, Google-Extended and Applebot-Extended are control tokens, not separate crawlers. They govern how content already fetched by the main crawler may be used for AI purposes, and Google states that Google-Extended does not affect inclusion or ranking in Google Search. Second, the bots in the table change. Operators update names, purposes and published IP ranges, so re-verify them against official documentation and each operator's IP range lists before every material robots.txt change.
One more warning: anyone can claim to be GPTBot. A User-Agent header can be imitated, so it is not proof of origin. Imperva's measurement of broad bot traffic found that automated traffic accounted for 51% of all web traffic in 2024, with bad bots at 37%. That figure covers automated traffic generally rather than AI crawlers alone, but it explains why any serious bot policy needs server-side verification, either against published IP ranges or through the operator's documented reverse-DNS procedure.
Training bots vs search bots: which AI crawlers should you allow?
The useful question is not "is this an AI bot?" but "what is this bot for?" Three behaviours are worth separating:
- Training crawlers collect content that may be used to improve models. These include GPTBot, ClaudeBot, CCBot and the Google-Extended token.
- Search crawlers build an index used to retrieve and cite pages in AI answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot.
- User-initiated agents fetch a page when a person asks the assistant to look at it. ChatGPT-User, Claude-User and Perplexity-User fall here.

This separation is why blocking GPTBot does not by itself remove you from ChatGPT search. OpenAI documents OAI-SearchBot as a distinct agent for surfacing sites in ChatGPT search, and says site owners can manage the two independently. The reverse also holds: allowing OAI-SearchBot does not grant GPTBot permission. Anthropic draws the same line between ClaudeBot, Claude-SearchBot and Claude-User.
Why AI search access matters for e-commerce
Product pages, prices, availability and reviews are the raw material of shopping recommendations. If a retrieval crawler cannot read them, the assistant has to rely on third-party descriptions, which may be stale, incomplete or written by a competitor. For most retailers, the value of retrieval access far exceeds whatever protection blocking it would provide.
When should you block AI training crawlers?
There are legitimate grounds to restrict training crawlers: proprietary buying guides that took expert time to produce, original photography, licensed content you are contractually barred from redistributing, or a stated brand policy on AI use. Be realistic about the limits, though. A robots.txt block affects future crawling under the relevant rule. It does not remove information already collected, and it does not control content available through other sources.
Where the line between training and retrieval gets blurry
The line is not always clean, and operators revise their policies, so treat this as a decision to revisit rather than settle once. One consequence deserves special mention: Copilot answers draw on the Bing index, so blocking Bingbot has consequences well beyond traditional search.
Should you block GPTBot? A decision framework for e-commerce brands

Start by answering three questions honestly:
- How product-led is your traffic? If most visits land on product and category pages, retrieval access is central to your visibility.
- How valuable is your unique content? Think of original editorial, reviews and imagery that competitors cannot replicate.
- What is your legal exposure? Licensing terms, contractual restrictions and privacy obligations may limit what you can leave open.
Your answers will put you in one of four scenarios.
- Scenario 1, commodity or branded retail with little unique content: allow training and search bots, and put your effort into being cited. Blocking gains you little when your descriptions largely mirror manufacturer copy.
- Scenario 2, substantial original editorial, reviews or imagery: allow search and retrieval bots, but disallow training bots on specific directories such as
/blog/or/guides/. - Scenario 3, licensing or legal constraints: block training bots site-wide, and keep retrieval access to product and category pages so you remain eligible for shopping answers.
- Scenario 4, marketplace seller without control of robots.txt: you cannot edit the platform's file, so work through the platform's settings where they exist. Then strengthen your presence on the third-party pages that AI engines already cite, such as review sites, comparison articles and retailer listings.
I want to be candid about the evidence. There is limited public data on how much blocking training bots affects recommendations, and anyone claiming a precise figure is extrapolating. That uncertainty is exactly why measuring your own results matters, and I return to it below.
Sample robots.txt rules for AI crawlers

Before the snippets, one rule of specificity trips up more teams than any other. A crawler obeys the most specific matching group, and a named group overrides the wildcard (User-agent: *) group entirely rather than merging with it. If you create a group for OAI-SearchBot, it no longer inherits your wildcard Disallow lines for the cart and checkout, so you must repeat them. The examples below do this deliberately.
Configuration A: allow AI search and discovery
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /searchUser-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Allow: /
The first group applies to every bot without its own group, training crawlers included, and keeps transactional and internal-search URLs out of reach. The second group names the search and retrieval bots, repeats those restrictions, and explicitly allows everything else. The sitemap line tells crawlers where to find your canonical URLs.
Configuration B: allow retrieval, block training bots
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /searchUser-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
This keeps every retrieval path open while refusing the training-oriented agents site-wide. Because Google-Extended is a control token, the Disallow: / line restricts certain Gemini uses and leaves Googlebot, and therefore Search, untouched.
Configuration C: selectively block access to original content
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /blog/
Disallow: /guides/
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Here training bots may read product and category paths, but the directories holding original editorial are off limits. Pair this group with the search-bot group from Configuration A.
Robots.txt housekeeping
- Keep the
Sitemap:directive in place. - Do not block CSS, JavaScript or product feed endpoints that your pages need to render.
- Check every rule against current operator documentation before deployment, since tokens and behaviours change.
Common robots.txt mistakes that block AI bots
In my experience, accidental blocks do more damage than deliberate policy. The following checklist can be audited in under an hour:
- A leftover blanket rule, such as
User-agent: *withDisallow: /carried over from a staging site or migration. - Copy-pasted "block all AI bots" lists that include search and retrieval agents, which cuts off citation as well as training.
- CDN or WAF bot protection. Managed bot rules or rate limiting can return 403s or challenge pages to AI crawlers even though robots.txt permits them.
- Platform defaults. Cloudflare and similar services may apply AI-bot blocking by default, so the robots.txt file looks fine while the edge refuses the request. Check your dashboard rather than assuming.
- Client-side rendering. Product content built only in JavaScript may be invisible to crawlers that do not execute it.
- Forgotten subdomains, where separate robots.txt files on
shop.,checkout.orcdn.hosts have gone unreviewed. - Interstitials and walls. Login walls, geo-blocking or cookie-consent overlays can serve UK-only or consent-gated content to bots as an empty page.
- Conflicting directives. Noindex and nosnippet in meta tags or X-Robots-Tag headers can contradict your intended access policy. Google also advises using noindex, not robots.txt, to keep a page out of Search, and the crawler must be able to fetch the page to see that directive.
How to test whether AI crawlers can reach your pages

A correct-looking file proves little on its own. Work through these steps in order.
- Fetch
yourdomain.co.uk/robots.txtdirectly. Confirm it returns a 200 status, is served as plain text, and is not redirected or blocked. - Test specific combinations in a robots.txt parser. Check named user-agents against product, category and policy URLs, not just the homepage.
- Request key pages with curl using each bot's published user-agent string, and compare the status codes with a normal browser request. A 403 here, despite a permissive robots.txt, points to an edge-level block.
- Review server logs. Filter by user-agent and verify IP addresses against each operator's published ranges, so you can see which bots genuinely visit and which receive errors.
- Check CDN and firewall event logs for challenges or blocks aimed at AI crawlers.
- Run an AI legibility audit. MentionOwl's audit runs 16 technical checks covering crawlability, structured data and renderable content, which surfaces problems robots.txt cannot reveal, such as product copy that only appears after JavaScript runs.
- Record the date and the change made, so any later shift in visibility can be tied to a specific rule.
Will blocking AI bots hurt your recommendations? How to measure the impact
Access and visibility are related but not identical. Being crawlable is necessary, yet citations also depend on content quality, third-party mentions, reviews and authority. Equally, blocking a crawler does not guarantee you vanish from answers, since a service may draw on a search index, other sources, or information gathered before the block. So measure outcomes. A robots.txt change is not a visibility switch.

A before-and-after method for measuring AI visibility
First, fix a set of shopping questions that reflect real purchase decisions, such as category, price-band and comparison queries. Establish a baseline across them before touching robots.txt, then track the same questions daily for several weeks after the change. Where possible, compare pages whose rules changed against comparable pages that did not, and allow for seasonality, promotions and stock levels. Crawler requests alone do not demonstrate recommendation impact.
The AI visibility metrics that matter
- Query coverage: the share of your questions where your brand appears at all.
- Position-weighted ChatGPT citations: how prominently and how often your pages are cited.
- Share of voice against competitors in the same answers.
- Soft mentions, meaning brand references without a link.
- Sentiment: whether the way you are described helps or harms.
MentionOwl runs your auto-generated customer questions daily across ChatGPT, Claude, Gemini, Copilot and Perplexity, and condenses coverage, citations, share of voice and soft mentions into a 0-100 visibility score. Competitor tracking then shows whether rivals are being recommended in your place, which is often the first sign that an access rule is costing you. Cookieless AI traffic analytics add a second angle, showing whether referrals from assistants rise or fall after a change.
How to interpret the results
AI answers vary between runs, so a single response proves nothing. Look at trends across many queries over several weeks. Change one variable at a time and keep a change log, so cause and effect stay legible.
A practical rollout plan for robots.txt AI crawler rules
- Audit the current robots.txt, subdomains, CDN rules and rendering.
- Decide your position on training versus retrieval using the scenarios above.
- Deploy the revised file to staging first, then production, and confirm with log checks.
- Baseline your AI visibility and competitor share of voice before the change.
- Review weekly digest data for four to six weeks, then adjust.
- Schedule a quarterly review, since new crawlers and policy changes appear regularly.
- Keep perspective. Robots.txt is one lever within a wider AI SEO strategy that also includes structured product data, clear policies and strong third-party coverage.
Frequently asked questions about robots.txt and AI crawlers
Should I block GPTBot?
It depends on where you stand on training versus discovery. GPTBot is OpenAI's training crawler, and blocking it is a reasonable choice if you have proprietary content you do not want used for model training. OpenAI documents OAI-SearchBot separately for ChatGPT search, so you can block GPTBot while still allowing OAI-SearchBot. Whatever you choose, monitor your ChatGPT citations afterwards to confirm the change has had no unintended effect.
Will blocking AI bots hurt my AI recommendations?
Blocking search and retrieval crawlers can reduce your chances of being cited, because the assistant cannot fetch or index your pages. Blocking training-only crawlers has a less certain effect, and public evidence is limited, so measure it on your own queries. Tracking query coverage and share of voice before and after a change will give you a defensible answer.
Does robots.txt actually stop AI crawlers?
Only the ones that choose to respect it. Major operators such as OpenAI, Anthropic and Google document compliance, but robots.txt is a voluntary protocol, and some user-initiated agents or unscrupulous scrapers may not follow it. For stronger enforcement, you need server-level controls such as IP verification and bot management rules.
What is the difference between Google-Extended and Googlebot?
Googlebot is the crawler that fetches pages for Google Search. Google-Extended is a control token that lets you opt your content out of use in certain Google AI products, such as Gemini model training and grounding, without affecting how Googlebot crawls or ranks you in Search. Blocking Google-Extended does not remove you from search results.
How do I test whether AI bots can reach my pages?
Check that robots.txt loads correctly, test specific URLs against specific user-agents with a parser, request pages using each bot's user-agent string, and review server and CDN logs for blocked or challenged requests. An AI legibility audit can also flag crawlability and rendering problems that robots.txt alone will not reveal.
How often should I review my AI crawler rules?
At least quarterly, and whenever you migrate your site, change CDN settings, or hear that an operator has introduced a new crawler. Continuous monitoring of your AI visibility will alert you to sudden drops that may point to an accidental block.


