AI crawlers and robots.txt: which bots to allow and what you give up
GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest: what each is for, and how to allow answering bots while deciding separately on training access.
Short answer
Most AI companies run separate crawlers for training and for live retrieval, and blocking the wrong one removes you from AI answers without protecting anything. For a business whose site is marketing rather than product, the usual correct answer is to allow all of them. For publishers whose content is the product, the decision is genuinely harder.
The confusion here is structural. AI companies bundle two very different activities behind similar-looking user agents, and the advice circulating usually fails to separate them.
The two activities:
Training. Bulk crawling to build a corpus for model training. Your content becomes part of what a future model knows. No attribution, no traffic, no link.
Retrieval. Fetching pages live, in response to a user's question, to ground an answer. Usually cited, sometimes clicked, and this is the path that gets you recommended.
Blocking training while keeping retrieval is often what a business wants. Whether you can do that depends on the provider.
The agents worth knowing
GPTBot (OpenAI) crawls for training. OAI-SearchBot handles search indexing for ChatGPT's search feature. ChatGPT-User fetches a page when a user's question requires it. These are separate tokens, which means you can allow retrieval and disallow training.
ClaudeBot (Anthropic) is the general crawler. Claude-User covers user-initiated fetches. Claude-SearchBot handles search indexing.
PerplexityBot crawls for Perplexity's index. Perplexity-User covers user-initiated visits. Perplexity's whole product is retrieval with citation, which makes it the clearest case for allowing access.
Google-Extended is not a crawler at all. It is a control token: it governs whether content Googlebot has already fetched can be used for Gemini and AI answer grounding. Disallowing it leaves Search ranking untouched.
Applebot-Extended does the equivalent job for Apple's models. Applebot itself serves Siri and Spotlight.
Bytespider (ByteDance), Amazonbot, meta-externalagent and CCBot (Common Crawl) round out the set most sites see. CCBot is worth a thought: Common Crawl is a public dataset that many model builders draw on, so blocking it has broad downstream effect.
Deciding what to allow
The question is what your website is for.
If your site is marketing. A restaurant, hotel, café, tour operator, agency, most local businesses. Allow everything. The content is a description of a service people buy in the physical world. Training on it costs you nothing and being retrievable is the entire objective. Blocking here is self-harm.
If your content is the product. A publisher, a paid research site, a course business. The trade-off is real. Training use gives you nothing back. Retrieval gives you a citation, some traffic, and exposure. A defensible position is to allow retrieval agents, disallow training agents, and watch what happens to referrals.
If you hold sensitive material, none of this is the right tool. robots.txt is a request, not a control. Anything genuinely confidential belongs behind authentication.
A working file for a hospitality business
# Allow everything, including AI crawlers.
# One group, deliberately. Read the note below before adding named groups.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlThere is a trap here worth more than the rest of this section. A crawler obeys exactly one group, and Google states it plainly: "User agent specific groups and global groups (*) are not combined." So the moment you add User-agent: GPTBot with Allow: / under it, that agent stops reading the wildcard group, and the Disallow: /admin/ and Disallow: /api/ lines above no longer apply to it. An earlier version of this post recommended a file that did exactly that: it named eight agents and quietly exempted every one of them from both Disallow lines.
If you want to allow everyone, the wildcard group on its own is correct, and it is the safer file. Name an agent only when you want to give it different rules from everybody else, and when you do, repeat every Disallow line inside its group.
A file that separates training from retrieval
For a publisher who wants citations without contributing to training:
User-agent: *
Allow: /
# Training crawlers, disallowed.
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
# Retrieval and user-initiated fetches, allowed.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xmlThis is a reasonable position, and it is not airtight. The boundary between training and retrieval is defined by the providers and they can redraw it.
Things people get wrong
Blocking by accident. A staging Disallow: / that shipped to production. Check the live file, not the repo.
Assuming robots.txt hides anything. It is a public file listing the paths you care about. Compliant crawlers obey it; nothing else does.
Blocking AI crawlers and then buying GEO services. This happens more than it should. Check the file before anything else.
Never revisiting it. New agents appear every few months. Put a calendar reminder on it twice a year.
The short version
For almost every local and hospitality business the answer is: allow everything, list the agents explicitly, keep the sitemap line accurate, and spend the saved deliberation on making the pages worth retrieving.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
It stops your content being used to train OpenAI's foundation models, and on its own it does not remove you from ChatGPT's search results. OpenAI documents these as independent settings, so the effect depends on which agents you block. Blocking OAI-SearchBot is what keeps you out of ChatGPT search answers, though you can still appear as a navigational link. ChatGPT-User covers user-initiated fetches, and OpenAI says robots.txt rules may not apply to those.
What does Google-Extended control?
Google-Extended is not a crawler. It is a token in robots.txt that controls whether content already crawled by Googlebot may be used for Gemini and for grounding AI answers. Disallowing it does not affect ordinary Google Search ranking, which is the point of separating it.
Is robots.txt enforceable?
No. It is a voluntary convention. Well-behaved crawlers from major providers honour it. Scrapers ignore it. If you need enforcement you need server-side blocking by user agent and IP, or a WAF rule, not a text file.
Should a restaurant or hotel block AI crawlers?
Almost never. Your site exists to be found. Blocking the crawlers that feed the systems your customers now ask for recommendations is the opposite of what you want, and there is no content on a typical hospitality site worth protecting from training use.
Want to know how AI models currently describe your business?
We run a free visibility check across ChatGPT, Perplexity, Claude and Google AI Overviews, then show you exactly which signals are missing.
Book a visibility check