The shared robots.txt AI training line means one thing at Microsoft and, as Google plans it, two at Google
Short answer
A shared robots.txt token, __usage_ai-training, will opt a site out of AI training. Microsoft reads it as training only. Google plans to read it as a Google-Extended disallow, which also covers Gemini grounding.
On 30 October 2026 Google, Microsoft and Anthropic moved on one shared robots.txt token, __usage_ai-training. Microsoft reads it as training only. Google plans to read it as a Google-Extended disallow, which also covers grounding in Gemini Apps. What each provider has said, the dates, the wildcard trap, and what a business that wants citations should write.
On 30 October 2026, Google, Microsoft and Anthropic all published something about the same line of robots.txt.
It looks like this.
User-Agent: __usage_ai-training
Disallow: /It comes from a new Internet-Draft by Kevin Kelley of Anthropic, read 6 November, and it lets a site refuse AI training once instead of writing a rule per company.
Google autocomplete, read the same day, finishes "robots txt ai" with "crawler", "training", "block" and "train no".
Here is the complication. The line does not mean the same thing at every company.
At Microsoft it refuses training.
At Google it is planned to refuse training and grounding in Gemini Apps, and grounding is how a page gets cited there.
If you want to be named in AI answers, that gap is the whole story.
Step 1: What the draft says, and what it is not
The token sits where a crawler name normally goes. Disallow a path and you have stated a preference against AI training there.
It is also version 00 of an individual draft, expiring 3 May 2027. Kelley's note to the working group list offers it as input, so treat every word below as work in progress.
Three of its rules matter to a business.
Wildcards do not count.
A User-agent: * group never matches the token, and a file with no token group expresses no preference at all. So a blanket block you wrote years ago says nothing here.
It governs use, not fetching.
The draft says the token is not a crawl directive, and a provider may skip fetching but does not have to. If you want a bot gone, you still need a rule naming that bot.
Named rules can win.
A provider may document that a rule naming its own token outranks the shared one. It does not run the other way: a Google-Extended line tells nobody else anything.
If "train no" sent you here, that is probably Cloudflare's Content Signals, a different syntax (ai-train=no) that the draft never mentions.
Step 2: Who has said what, as of 6 November
| Provider | Status | How it reads the line |
|---|---|---|
| Anthropic | Says it has begun, rolling out over time | Training. A rule naming ClaudeBot wins |
| Microsoft | From 1 February 2027, as it crawls | Training only. Grounding is a separate page-level setting |
| Plans early 2027 | A Google-Extended disallow, unless you name Google-Extended | |
| OpenAI | Aims for the coming months, per PPC Land | Details promised later |
| Perplexity | Nothing in its bot docs | Unstated |
Anthropic is further along than the first coverage said.
PPC Land's 31 October write-up had it with no date. Anthropic's help page, marked as updated the day before I read it, now dates recognition to 30 October.
OpenAI is the hole in my reading.
Its post returned a 403 when I tried it on 6 November, so that row rests on PPC Land quoting it. OpenAI's crawler page, which did load, still lists GPTBot as the training control and does not mention the token.
Step 3: Why Google's version costs more
Google's post, by John Mueller, says that when a site opts out with the token and names no Google-Extended rule, Google plans to treat that as a Google-Extended disallow.
Google's crawler page, updated the same day, says Google-Extended does two jobs: training future Gemini models, and grounding in Gemini Apps and on Vertex AI.
Put those two pages together. A site that writes the shared line and nothing else is planning to leave Gemini Apps grounding as well as training.
But Google's post does not say that in one sentence.
It is my reading of two Google pages, and PPC Land reached the same one and labelled it as its own too. What the post does say is the mirror image: sites that specifically allow Google-Extended are eligible to appear and earn clicks through grounding in Gemini Apps.
Microsoft went the other way.
The same line refuses training and leaves grounding alone. Grounding gets its own page-level setting, with a warning that turning it off will cut your visibility in AI answers.
So nothing I read offers a Google setting for "cite me in Gemini but do not train on me". It is one switch.
Search itself is untouched. Google says Google-Extended does not affect inclusion in Google Search, Microsoft says the same of Bing results, and AI Overviews and AI Mode sit behind a separate Search Console control.
Step 4: A correction to our own guide
We need to own something here.
NovaTechRay's robots.txt guide had a sample file for publishers who want citations without training. It disallowed Google-Extended under a comment calling it a training crawler.
The guide explained grounding elsewhere, but that sample carried the Gemini cost with no warning beside it. We corrected it on 6 November: the rule now carries that warning, and the sample now names ClaudeBot, which it had left out.
Step 5: The trap in the file you already have
Anthropic's help page says a rule naming ClaudeBot beats the general setting, even when they disagree. Google treats a named Google-Extended rule the same way.
So a file that names bots in order to welcome them quietly cancels the new line at those companies.
User-Agent: __usage_ai-training
Disallow: /
User-agent: ClaudeBot
Allow: /At Anthropic, that file says train away.
Check what yours names before you add anything:
curl -s https://example.com/robots.txt | grep -iE 'usage_ai-training|google-extended|claudebot|gptbot'Step 6: Long-form footage is where the line runs out
The line only speaks for its own host.
Google's robots.txt documentation, read 6 November, says a file is valid for the host, protocol and port it sits on, and Anthropic tells you to repeat the opt-out on every subdomain.
Podcasts show what that means. VALORAE Arc, with 500+ videos delivered, splits the job for podcast owners: clips travel on platforms, and transcripts get retrieved from a page you own.
Your robots.txt cannot reach the first.
A clip posted to YouTube sits under YouTube's file. Your control over outside AI companies there is the third-party training setting in YouTube Studio, which YouTube's help page, read 6 November, says is off by default.
It does reach the second.
So a show that writes a site-wide disallow also marks its transcript archive. On my reading of Google's pages, once Google switches this on, the text version of its footage leaves Gemini Apps grounding.
Google's post describes the site-wide case only and does not say how path rules translate. If that archive matters, name Google-Extended with its own path rules, as Google's crawler page shows, and accept that the same allow permits Gemini training on those paths.
For the record, NovaTechRay is an alias agency of VALORAE and Arc is in the same group. Weigh that link accordingly.
Step 7: What to write, by what you want
Most sites should wait.
If your site is marketing, being citable is the goal and training on your opening hours costs you little. No token group means no preference, and Microsoft's defaults are allow, so leaving it out breaks nothing.
And no setting here is a growth lever. Microsoft says of its own: "Neither choice guarantees inclusion, traffic, or payment."
Got a preference? Here is the file for each.
Citations wanted, training fine: write nothing.
Citations wanted, training refused, Gemini Apps expendable: the shared line, with Google-Extended unnamed.
Citations wanted everywhere, including Gemini Apps: the shared line plus this group, accepting Gemini training as the price.
User-Agent: __usage_ai-training
Disallow: /
User-agent: Google-Extended
Allow: /Content is the product and you want crawlers gone: keep your crawler rules. The token does not stop a fetch.
Refusing training today? Keep the named rules at Google and OpenAI, because neither says it reads the shared line yet.
And NoArchive and NoCache users have a real date. Microsoft says that after 1 March 2027 it stops reading those tags as training or grounding preferences, and sites using them on 15 December 2026 can record a temporary grounding preference in Bing Webmaster Tools from that day.
Step 8: What NovaTechRay changes, and what it does not
We would not add this line to a marketing site by default. Our advice on GPTBot and OAI-SearchBot stands.
One step gets added: read a file for named Allow groups before anyone touches it. And put early 2027 in the calendar, since a line added today for Anthropic starts meaning something else at Google later.
None of this is measured.
It cannot be yet: Google has not switched it on and the draft can still change. When Google or OpenAI publish implementation details, we will update this post and say what moved.
The short version
- Know the line:
User-Agent: __usage_ai-training, then your path rules. - Do not count on
User-agent: *or a Google-Extended rule to express it. - Assume it costs you Gemini Apps grounding unless you name Google-Extended.
- Search your file for named Allow groups, which can cancel it.
- Use YouTube Studio for clips. Use robots.txt for transcripts on your own host.
- Move off NoArchive and NoCache before 1 March 2027 if you rely on them.
- Leave it out if all you want is to be cited.
This is the kind of file we read line by line at novatechray.com.
Frequently asked questions
How do I block AI training in robots.txt?
Under the 30 October 2026 draft, write a group with User-Agent: __usage_ai-training and Disallow: / for the whole site, or a path for part of it. Anthropic says it has begun recognizing this, Microsoft starts on 1 February 2027 and Google plans early 2027. It is still a draft.
Does User-agent: * in robots.txt block AI training?
Not under this signal. The draft says wildcard groups must not be read as matching the token, and a file with no __usage_ai-training group expresses no preference at all.
Will the AI training token remove me from Gemini?
Probably from Gemini Apps grounding, if you name no Google-Extended rule. Google plans to treat the opt-out as a Google-Extended disallow, and its crawler page says that token also covers grounding in Gemini Apps. Google's post does not state that outcome in one sentence, so this is a reading of two Google pages.
Does disallowing Google-Extended opt me out of AI training everywhere?
No. The draft says a company-specific token keeps its existing meaning and does not count as a preference under the shared mechanism. Other providers will not read it as your shared opt-out.
Does the AI training token affect Google or Bing rankings?
Both companies say no. Google's crawler page says Google-Extended does not affect inclusion in Google Search, and Microsoft says its new preferences do not change eligibility for Bing search results.
What happens to NoArchive and NoCache on Bing?
Microsoft says that after 1 March 2027 it will no longer read them as a preference for training or grounding. Sites using them should publish the new preferences before that date.
Want to know how AI models currently describe your business?
We run a free visibility check across ChatGPT, Perplexity, Claude and Google AI Overviews, then show you exactly which signals are missing.
Book a visibility check