AI crawlers and robots.txt: which bots to allow and what you give up

The confusion here is structural. AI companies bundle two very different activities behind similar-looking user agents, and the advice circulating usually fails to separate them.

The two activities:

Training. Bulk crawling to build a corpus for model training. Your content becomes part of what a future model knows. No attribution, no traffic, no link.

Retrieval. Fetching pages live, in response to a user's question, to ground an answer. Usually cited, sometimes clicked, and this is the path that gets you recommended.

Blocking training while keeping retrieval is often what a business wants. Whether you can do that depends on the provider.

The agents worth knowing

GPTBot (OpenAI) crawls for training. OAI-SearchBot handles search indexing for ChatGPT's search feature. ChatGPT-User fetches a page when a user's question requires it. These are separate tokens, which means you can allow retrieval and disallow training. The exact user-agent strings and a robots.txt for each choice are in GPTBot, OAI-SearchBot and ChatGPT-User.

ClaudeBot (Anthropic) is the general crawler. Claude-User covers user-initiated fetches. Claude-SearchBot handles search indexing. Details in ClaudeBot, Claude-User and Claude-SearchBot.

PerplexityBot crawls for Perplexity's index. Perplexity-User covers user-initiated visits. Perplexity's whole product is retrieval with citation, which makes it the clearest case for allowing access. Details, and Cloudflare's stealth crawling report, in PerplexityBot and Perplexity-User. Whether any of these bots obeys the file is a separate question, answered in do AI crawlers respect robots.txt.

Google-Extended is not a crawler at all. It is a control token: it governs whether content Googlebot has already fetched can be used for Gemini and AI answer grounding. Disallowing it leaves Search ranking untouched.

A shared training token is on the way. On 30 October 2026 Google, Microsoft and Anthropic all published on User-Agent: __usage_ai-training, a token from an IETF draft that states a no-training preference across providers in one group. Anthropic says it has begun recognising the signal and is rolling it out over time. Microsoft starts on 1 February 2027. Google's crawler documentation says early 2027. At Google, using it without naming Google-Extended is planned to count as disallowing Google-Extended, so it costs more there than a training opt-out. Keep your named rules for now, because it is still a draft. What each provider has said, and what to write.

Applebot-Extended does the equivalent job for Apple's models. Applebot itself serves Siri and Spotlight.

Bytespider (ByteDance), Amazonbot, meta-externalagent and CCBot (Common Crawl) round out the set most sites see. CCBot is worth a thought: Common Crawl is a public dataset that many model builders draw on, so blocking it has broad downstream effect.

Deciding what to allow

The question is what your website is for.

If your site is marketing. A restaurant, hotel, café, tour operator, agency, most local businesses. Allow everything. The content is a description of a service people buy in the physical world. Training on it costs you nothing and being retrievable is the entire objective. Blocking here is self-harm.

If your content is the product. A publisher, a paid research site, a course business. The trade-off is real. Training use gives you nothing back. Retrieval gives you a citation, some traffic, and exposure. A defensible position is to allow retrieval agents, disallow training agents, and watch what happens to referrals.

If you hold sensitive material, none of this is the right tool. robots.txt is a request, not a control. Anything genuinely confidential belongs behind authentication.

A working file for a hospitality business

# Allow everything, including AI crawlers.
# One group, deliberately. Read the note below before adding named groups.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

There is a trap here worth more than the rest of this section. A crawler obeys exactly one group, and Google states it plainly: "User agent specific groups and global groups (*) are not combined." So the moment you add User-agent: GPTBot with Allow: / under it, that agent stops reading the wildcard group, and the Disallow: /admin/ and Disallow: /api/ lines above no longer apply to it. An earlier version of this post recommended a file that did exactly that: it named eight agents and quietly exempted every one of them from both Disallow lines.

If you want to allow everyone, the wildcard group on its own is correct, and it is the safer file. Name an agent only when you want to give it different rules from everybody else, and when you do, repeat every Disallow line inside its group.

The shortest allow-all file is two lines:

User-agent: *
Allow: /

Allow: / only says it out loud. Google's robots.txt guide, last updated 21 November 2025, says a crawler can fetch any page not blocked by a disallow rule.

A file that separates training from retrieval

For a publisher who wants citations without contributing to training:

User-agent: *
Allow: /

# Training crawlers, disallowed.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Not a crawler, and not training only. This also covers grounding
# in Gemini Apps and on Vertex AI. Read the note below before keeping it.
User-agent: Google-Extended
Disallow: /

# Retrieval and user-initiated fetches, allowed.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

The Google-Extended group costs more than the others. Google's crawler page, last updated 30 October 2026, says the token covers training future Gemini models and also grounding in Gemini Apps and on Vertex AI. So keeping those two lines refuses training and gives up grounding there, and nothing I read from Google separates the two. If being cited in Gemini matters more to you than refusing training there, delete them.

An earlier version of this sample filed Google-Extended under training crawlers with no warning, and left out ClaudeBot, the crawler Anthropic says to restrict for training. We corrected both on 6 November.

This is a reasonable position, and it is not airtight. The boundary between training and retrieval is defined by the providers and they can redraw it.

Things people get wrong

Blocking by accident. A staging Disallow: / that shipped to production. Check the live file, not the repo.

Assuming robots.txt hides anything. It is a public file listing the paths you care about. Compliant crawlers obey it; nothing else does.

Blocking AI crawlers and then buying GEO services. This happens more than it should. Check the file before anything else, and before paying for GEO at all.

Expecting llms.txt to do this job. An llms.txt file is a proposed list of your pages for AI systems. It allows and blocks nothing.

Never revisiting it. New agents appear every few months. Put a calendar reminder on it twice a year.

The short version

For almost every local and hospitality business the answer is: allow everything with one wildcard group, name an agent only when it needs different rules, keep the sitemap line accurate, and spend the saved deliberation on making the pages worth retrieving.