Technical Readiness

One robots.txt rule to opt out of AI training: what __usage_ai-training does at Microsoft, Google, OpenAI, and Anthropic

Q: Do I need to change robots.txt to allow AI training?

No. A file without a __usage_ai-training group expresses no preference, and User-Agent: * rules don't count as one. Microsoft treats both training and grounding as Allow by default. You can add User-Agent: __usage_ai-training with Allow: / to state it explicitly.

On Oct. 30, 2026, Microsoft, Google, OpenAI, and Anthropic each backed one robots.txt rule for AI training: a group for the policy token __usage_ai-training. Microsoft starts on Feb. 1, 2027, Google plans support in 2027, OpenAI says in the coming months, and the draft's author says Anthropic has started implementing it. The rule states a training preference; it doesn't block crawling, and each company's own crawler rules keep working. Google is the case to check: when the rule opts a site out and Google-Extended isn't named, Google plans to read it as a Google-Extended disallow, and Google-Extended also controls grounding in the Gemini app. Microsoft adds a separate page-level grounding preference and stops reading NOARCHIVE and NOCACHE as AI controls after March 1, 2027. If you are fine with AI training, change nothing. If you opt out, decide Google-Extended explicitly, then rerun the prompts you track to confirm your citations held.

Updated 2026-10-30

Questions this guide answers

What is __usage_ai-training in robots.txt?
Does opting out of AI training remove my site from AI answers?
Does the shared AI training opt-out affect Gemini?
When will Microsoft, Google, OpenAI, and Anthropic honor the shared AI training signal?
What replaces Bing's NOARCHIVE and NOCACHE for AI?
What does DisallowExtended mean in Bing's AI grounding preference?
Do I need to change robots.txt to allow AI training?

Direct answer

__usage_ai-training is a robots.txt policy token for one decision: whether AI companies may train generative models on your content. On Oct. 30, 2026, Microsoft, Google, OpenAI, and Anthropic each backed it: Microsoft and OpenAI say they will honor it, Google plans to, and Anthropic has started implementing it. One group, User-Agent: __usage_ai-training followed by Disallow: /, states a training opt-out to all 4 at once.

It states a preference about use. It doesn't block crawling, and each company's rules for its own crawlers keep working, so search crawlers such as OAI-SearchBot, Claude-SearchBot, Googlebot, and Bingbot are still governed by their own lines. Google is the case to check. When the token opts a site out and Google-Extended isn't named, Google plans to treat it as a Google-Extended disallow, and Google-Extended also controls grounding in the Gemini app. The shortest opt-out can cost Gemini app citations.

For a brand that wants to be cited: if you are fine with AI training, change nothing, because a file without the token expresses no preference and Microsoft defaults to Allow. If you opt out, name Google-Extended explicitly, leave Bing's new grounding preference on its default, and rerun the prompts you track as each company starts honoring the rule.

What was announced on Oct. 30, 2026

4 companies backed the rule on the same day, on 4 timelines; Anthropic's commitment came from the draft's author on the IETF working group's mailing list. Only Microsoft has given a start date. The rule itself is an individual Internet-Draft submitted to the IETF's AI Preferences working group, not yet a standard; Microsoft calls the working group's progress significant enough that it "merits adoption" while that work continues.

Company	What it said	When	Its existing controls
Microsoft	Bing will honor the token in robots.txt for a site or path, add page-level training and grounding tags, and replace NOARCHIVE and NOCACHE as AI controls	From Feb. 1, 2027, as Bing crawls	A page-level training tag overrides robots.txt; grounding is a separate page-level setting
Google	Plans to read a token opt-out "as if the publisher had disallowed Google-Extended" when Google-Extended isn't named	"in 2027"; Google's crawler documentation says early 2027	Google-Extended keeps working "alongside this new joint signal"
OpenAI	"OpenAI will honor a common signal" for "foundational generative AI training"	"in the coming months," with implementation details to follow	"This new signal will not replace the controls we already provide through robots.txt"
Anthropic	The draft's author, Kevin Kelley of Anthropic, wrote that Anthropic "has already started implementing support"	"It'll take us some time to add support across our services"	"This complements our existing crawler controls rather than replacing them"

How the rule works

The draft, "Machine-Readable Training Preference Signal," adds no new syntax to robots.txt. The details that matter for a site owner are these.

It's an ordinary robots.txt group. The user-agent line names a policy token instead of a crawler. Disallow states a preference against training for those paths and Allow a preference for it, with the usual longest-match rules. The draft's example refuses training everywhere except /press-kit/.
No group, no preference. User-Agent: * groups never match the token, so existing wildcard rules aren't read as a training choice. In Kelley's words, a file that doesn't use the mechanism, "as is true of every one published today, expresses no preference at all."
It doesn't control crawling. The draft says the token "is not a directive for crawling behavior." Rules for a crawler's own user agent still decide what it fetches, though a crawler that collects only for training may skip pages that refuse training.
The specific rule should win. The draft says "Specific expressions SHOULD take precedence over general ones" and lets each company treat its own tokens, such as Google-Extended, as more specific. Google has said how it will apply that; OpenAI says its existing crawler controls "will continue to apply and be respected" and will publish its implementation details later.
"Training" has a narrow meaning. The draft uses the working group's definition: "Using an asset to modify the learned parameters of a generative AI model." Models used only for "classification, ranking, or scoring" are outside it, so the rule says nothing about search ranking.

The Google catch: the opt-out can cost Gemini app citations

Google's announcement is specific: "When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended."

Google-Extended isn't only a training control. Google's crawler documentation says it governs training of future Gemini models "and for grounding" in "Gemini Apps and Grounding with Google Search on Vertex AI." Google's announcement adds that "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Once Google supports the token, a site that adds only the 2-line opt-out refuses Gemini training and, by Google's reading, Gemini app grounding too.

Google Search is not affected. Google says Google-Extended "does not impact a site's inclusion in Google Search," and AI Overviews and AI Mode follow Googlebot and Search Console's Search generative AI control, as the UK opt-out guide explains.

There is still no Google setting that refuses Gemini training and keeps Gemini app grounding, and the token doesn't create one. What changes is how the choice gets made: without anyone naming Google-Extended, a generic training opt-out now makes the bundled choice for you. Decide Google-Extended on purpose, using the setups below.

3 robots.txt setups, and what each one does

Each setup assumes your search crawlers stay allowed. "Then" marks the next line of the same group; a new User-Agent line starts a new group.

Your goal	What to write in robots.txt	What it does
Allow AI training and keep every citation	Nothing. Optionally `User-Agent: __usage_ai-training` then `Allow: /`	No preference, or an explicit one in favor of training. Microsoft defaults to Allow, and Gemini app grounding is unchanged
Refuse training except Google's, and keep Gemini app citations	`User-Agent: __usage_ai-training` then `Disallow: /`, plus a group for `User-Agent: Google-Extended` then `Allow: /`	Refuses training at Microsoft, OpenAI, and Anthropic as each implements the token. At Google, Gemini training stays allowed, because Google-Extended is the same switch as Gemini app grounding
Refuse training wherever the token is honored, Google included	`User-Agent: __usage_ai-training` then `Disallow: /`	Refuses training at all 4 as each implements the token. Google reads it as a Google-Extended disallow, so Gemini app grounding goes too; AI Overviews and AI Mode are unaffected

Microsoft: training and grounding become separate choices

Microsoft goes further than the shared token. Its announcement covers content used for training and "to inform generated responses, including web-grounded responses in Bing, Copilot, and Microsoft 365," and it splits that use into 2 independent preferences. Both default to Allow, and setting one doesn't change the other.

AI Training. Use the shared token in robots.txt for a site or a path, or set it per page with <meta name="Bingbot" content="ai-training:disallow"> or the header X-Robots-Tag: Bingbot: ai-training:disallow. A page-level setting overrides robots.txt.
AI Grounding. Page level only, through the same meta tag or header with ai-grounding:disallow or ai-grounding:disallowextended; Allow is the default. Microsoft: "There is no site-wide robots.txt grounding preference."
DisallowExtended keeps a page available to "Microsoft-operated experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365," which are Microsoft's own products despite the names, and excludes "grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry." Microsoft launched Web IQ on June 2, 2026 as "a suite of AI-native grounding APIs" for AI systems and agents, "a search engine for AI systems."
Bing search is untouched. The preferences "do not change eligibility for traditional Bing search results," and the Bing search syndication API, which "returns links for traditional search scenarios," is not affected by the grounding preference.
Use, not access. "These settings govern use of content, not crawl access." An Allow setting doesn't override a robots.txt rule that blocks Bingbot, and a new setting takes effect only when Bing recrawls the page. Microsoft says IndexNow can "increase the chance of a recrawl" without guaranteeing a date; see how to get AI engines to see your updates.

What grounding restrictions cost

Microsoft states the trade plainly: "If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences." For a brand that wants to be cited, leave grounding on its default. DisallowExtended suits a publisher that wants Microsoft's AI products but not other companies' products built on Microsoft's grounding services. Microsoft doesn't name Web IQ's or Foundry's customers, so a brand can't see in advance which AI products DisallowExtended would remove.

If your site uses NOARCHIVE or NOCACHE

NOARCHIVE and NOCACHE are the AI controls Bing documented in 2023: NOARCHIVE keeps a page out of Copilot answers and training, and NOCACHE limits both to the URL, title, and snippet. Microsoft is retiring them as AI controls on this schedule.

Date	What happens	What to do
Dec. 15, 2026	Temporary grounding preferences open in Bing Webmaster Tools, only for sites that use NOARCHIVE or NOCACHE on that date	If you use these tags to keep pages out of AI answers, record a temporary grounding preference before March 1, 2027. Otherwise, nothing
Feb. 1, 2027	Bing starts honoring the new preferences as it crawls. On a site where it finds them, it disregards any remaining NOARCHIVE and NOCACHE	Publish the new training and grounding preferences you want, before March 1, 2027
March 1, 2027	NOARCHIVE and NOCACHE "will no longer be read as a preference for training or grounding." Pages that relied on them and have no recorded temporary preference "default to Allow for grounding" until Bing recrawls them	To keep pages out of grounding, publish the page-level tag and record the temporary preference. Training needs robots.txt or page tags, because there is no temporary training setting
Sept. 30, 2027	Temporary preferences expire, or earlier for each page Bing recrawls and finds a new grounding tag on	Make sure every page you want restricted carries the new tag

Check for tags nobody chose

For a brand, the retirement mostly helps. If NOARCHIVE sits on your pages by accident, or by a decision nobody remembers, it keeps them out of Copilot answers today, and from March 1, 2027 those pages default to Allow for grounding. Don't wait for that: search your templates, CMS settings, and response headers for noarchive and nocache now, and remove them from pages you want cited.

Before you edit robots.txt

The token makes a training opt-out easy to write and easy to get wrong. Run these checks first.

Decide training as a policy, then write it once. One __usage_ai-training group, plus an explicit Google-Extended group, states it to all 4 companies as each honors the token. Company-specific lines such as GPTBot, ClaudeBot, and Applebot-Extended still apply, and Apple and other AI companies haven't announced support for the token, so keep those lines consistent with it.
Leave the search crawlers alone. The token doesn't change what OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, or Bingbot may crawl, but a crawl block still removes you from answers. AI crawler access lists which bot does what.
Read the file you actually serve. If a CDN, plugin, or CMS writes robots.txt for you, fetch /robots.txt from your live site after every change, because that file is what each company reads.
Find old tags. Search for noarchive, nocache, and X-Robots-Tag rules before Bing's dates, and find out who added them.
Log the date. Record every robots.txt, meta tag, and header change, so the verification step has a before and an after.

Verify: did the change cost you citations?

A preference changes nothing until a company honors it and rereads your pages. Microsoft applies a page's setting when Bing recrawls it, and Google, OpenAI, and Anthropic will each start on their own schedule, so the effect arrives engine by engine. Before a change, capture which of your URLs each engine cites on the prompts you track. After each start date, rerun the same prompts on the same engines and compare.

For the Gemini app, where Google-Extended applies, ask your most important prompts in the app itself before and after Google starts honoring the token. Judge across several runs, because engines change what they cite on their own; how many runs you need covers the noise. For Copilot and Bing's AI summaries, Bing Webmaster Tools' AI Performance report counts citations of your pages; what the free reports show compares it with Search Console and GA4.

How SolCrys helps

SolCrys runs your prompt set on a schedule across ChatGPT, Google AI Overviews, Gemini, Perplexity, and Claude, depending on plan, and records every URL each engine cites. When a training or grounding preference changes, on your side or when a company starts honoring the token, the same prompts rerun on the same engines, so you can see across several runs whether each engine you track still cites your pages. The Content Audit's AI Bot Check reads robots.txt and requests a page as named crawlers, to flag a crawl block that a robots.txt edit introduced by mistake.

To see which sources ChatGPT cites for 10 of your buyer questions, Start Free: 10 ChatGPT prompts and 1 content audit, free, with email verification and no credit card. To verify a crawler-policy change across engines, talk to us.

Sources

FAQ

What is __usage_ai-training?

A robots.txt policy token for AI training preferences, defined in an IETF Internet-Draft published on Oct. 30, 2026. A group that starts User-Agent: __usage_ai-training and says Disallow: / states a preference against training generative AI models on your content; Allow states a preference for it. Microsoft and OpenAI have said they will honor it, Google plans to, and Anthropic has started implementing it. It doesn't block crawling, and it doesn't replace company-specific rules such as GPTBot or Google-Extended.

Does opting out of AI training remove my site from AI answers?

Not by itself at Microsoft, where grounding is a separate setting, and OpenAI and Anthropic describe it as a training signal that sits alongside their existing crawler controls. At Google it can: when Google-Extended isn't named, Google plans to read the opt-out as a Google-Extended disallow, and Google-Extended also controls grounding in the Gemini app. AI Overviews and AI Mode are not affected.

Does the shared AI training opt-out affect Gemini?

It can. Google-Extended covers Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and Google plans to treat a token opt-out as a Google-Extended disallow when Google-Extended isn't named. Adding a User-Agent: Google-Extended group with Allow: / keeps Gemini app grounding, and Gemini training with it.

When will Microsoft, Google, OpenAI, and Anthropic honor the shared AI training signal?

Microsoft from Feb. 1, 2027, as Bing crawls. Google in 2027; its crawler documentation says early 2027. OpenAI "in the coming months," with details to follow. Anthropic has started implementing support, and the draft's author says adding it across Anthropic's services will take time.

What replaces Bing's NOARCHIVE and NOCACHE for AI?

Separate AI Training and AI Grounding preferences. From Feb. 1, 2027, Bing honors the shared robots.txt training token and page-level tags such as ai-grounding:disallow. After March 1, 2027, NOARCHIVE and NOCACHE are no longer read as training or grounding preferences, and pages without a temporary preference recorded in Bing Webmaster Tools default to Allow for grounding until Bing recrawls them.

What does DisallowExtended mean in Bing's AI grounding preference?

It's 1 of 3 values for Bing's page-level grounding preference. It keeps a page available to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365, and excludes grounding services Microsoft provides to other companies, such as Web IQ and the Foundry grounding API. Microsoft says choosing it reduces brand visibility in AI experiences.

Do I need to change robots.txt to allow AI training?

No. A file without a __usage_ai-training group expresses no preference, and User-Agent: * rules don't count as one. Microsoft treats both training and grounding as Allow by default. You can add User-Agent: __usage_ai-training with Allow: / to state it explicitly.

Free ChatGPT visibility check

See where AI answers skip your brand — then fix it, free

Start a free workspace with your domain: 10 buyer-intent prompts through ChatGPT show where you are mentioned, cited, or skipped, and who gets recommended instead. A free content audit in the same workspace hands you the first fix to ship.

Start Free

Free · No credit card · About 5 minutes

Related guides

Technical Readiness

Search, Agent, or Training? How to Set AI Crawler Access in 2026 Without Blocking Your Own Citations

Which AI crawlers to allow, what Cloudflare's Sept. 15, 2026 rules changed, and how to block AI training without losing your AI citations.

Technical Readiness

Opting Out of Google AI Overviews: What the UK's New Rules Protect From Dec. 3, 2026

From Dec. 3, 2026, UK rules bar Google from downranking content that opts out of its AI answers in Search. What opting out removes, and who should opt out.

Attribution & ROI

The Free AI Reports: What Search Console, Bing, GA4, and Merchant Center Show, and What None of Them Do

Search Console, Bing Webmaster Tools, GA4, and Merchant Center now report AI search for free. What each one shows, what it misses, and how to cover the gaps.

Technical Readiness

How to Get AI Engines to See Your Updates Faster

An AI answer can't reflect your update until the engine refetches the page. How Google, ChatGPT, Perplexity, Claude, and Bing find changes, what IndexNow does, and how to check.

Measurement

How Many Runs Until Your AI-Visibility Number Is Trustworthy?

A single AI check is one sample, not a measurement. How many repeated runs and prompts you need before an AI-visibility number is trustworthy, with the published evidence on sample size and confidence intervals.

How SolCrys Works

What a SolCrys Content Audit Looks Like

What a SolCrys Content Audit checks on one URL, how the score and recoverable points work, and how a fix is promoted, published, and re-audited.

Strategy & Positioning

Why llms.txt Is Not a Strategy

llms.txt is a proposed standard for AI-friendly content delivery, but it is neither widely adopted by major AI engines nor a substitute for AEO fundamentals. This essay explains what llms.txt does, what it does not, and why brands should focus on the unsexy basics.