Shared robots.txt AI Training Control (__usage_ai-training): Separate Training, Crawling, and Grounding Rules

A conceptual illustration showing website path rules passing through a protocol gate to separate generative AI training, grounding citations, and search indexing. This is not a screenshot of an actual tool.

On October 30, 2026, Google, OpenAI, Microsoft, and Anthropic announced plans to support a shared robots.txt policy token—__usage_ai-training—so website operators can express generative AI training preferences without maintaining an ever-growing list of vendor-specific crawler rules.

The key takeaway: do not delete your existing GPTBot, Google-Extended, or ClaudeBot rules today and replace them with __usage_ai-training alone. The mechanism is currently defined in an individual IETF Internet-Draft rather than a finalized standard, and major providers target production rollout in early 2027. Furthermore, User-Agent: * wildcard groups never match this policy token, and crawling directives, generative model training preferences, and search or grounding visibility controls operate on separate layers.

This guide examines the protocol rules, provider rollout timelines, and precedence behavior based on the published IETF draft and official vendor documentation. It does not report live crawler measurements from production domains; the local Python checks below verify robots.txt precedence and fallback logic only.

1. Separate Crawling, Generative AI Training, and Search or Grounding Visibility

The technical specification comes from the individual Internet-Draft Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic on October 30, 2026, alongside the IETF AI Preferences (AIPREF) Working Group vocabulary (draft-ietf-aipref-vocab-08). Unlike traditional User-Agent lines that name a crawler, User-Agent: __usage_ai-training names a content usage purpose (policy token).

Dimension	Shared Training Token (`__usage_ai-training`)	Vendor Crawler & Policy Tokens (`Google-Extended`, `GPTBot`, `ClaudeBot`)	Search Indexing & AI Grounding Controls
Scope & Definition	States a preference on using retrieved assets to modify the learned parameters of a generative AI model	Controls crawler access (`GPTBot`, `ClaudeBot`) or vendor-specific training and grounding scope (`Google-Extended`)	Controls traditional search indexing (`Googlebot`, `noindex`) and AI answer citations (Search Console generative AI control, Bing `ai-grounding`)
Effect on Crawling	Not a crawling directive, though consumers that crawl solely for training may skip disallowed paths	`GPTBot` and `ClaudeBot` govern HTTP crawling; `Google-Extended` is a policy token without a separate HTTP user-agent string	Blocking `Googlebot` removes pages from ordinary Google Search as well as AI features and should not be used for training opt-outs
`User-Agent: *` Behavior	Never matches (MUST NOT match). A wildcard disallow expresses no preference on AI training	Standard crawlers fall back to `*` when unnamed, but policy tokens should be declared explicitly	A site-wide `Disallow: /` under `User-Agent: *` blocks search engine crawlers altogether
Exclusions	Excludes training models used exclusively for non-generative tasks such as classification, ranking, or scoring	Scoped to the specific products defined in each vendor's documentation	Training preferences and real-time grounding eligibility are decoupled differently by each provider

Sections 3 and 4 of the IETF draft establish three foundational rules:

Scoped strictly to generative AI parameter training: Following the AIPREF vocabulary, AI training means “using an asset to modify the learned parameters of a generative AI model.” Models used exclusively for classification, ranking, or scoring are excluded, even if the underlying architecture is capable of generation.

Wildcard groups (User-Agent: *) do not apply: Consumers must not interpret a User-Agent: * group as matching __usage_ai-training. If a robots.txt file has no __usage_ai-training group, the site expresses no preference under this specification.

Zero new grammar on top of RFC 9309: Path matching, group structure, and longest-match precedence work identically to RFC 9309 (Robots Exclusion Protocol). Existing parsers that do not recognize __usage_ai-training simply ignore the group without breaking legacy traffic rules.

2. Rollout Timelines and Vendor Precedence Rules Differ

Although four major AI providers backed the shared token on the same day, their implementation timelines and fallback rules are not identical. Cross-referencing the Google Search Central announcement, Google's common crawlers documentation, and the Microsoft Bing official blog reveals important operational nuances.

Provider	Target Rollout Timeline	Interaction with Existing Tokens & Precedence	Impact on Search & AI Grounding
Google	Targeted for early 2027	`Google-Extended` remains active. If a site opts out via `__usage_ai-training` and does not specify `Google-Extended`, Google plans to treat it as disallowing `Google-Extended`. An explicit `Google-Extended` rule overrides the shared token	`Google-Extended` does not affect Google Search inclusion or ranking. However, sites that explicitly allow `Google-Extended` remain eligible for grounding citations in Gemini Apps. Visibility in Search AI Overviews and AI Mode is managed separately via Search Console's “Search generative AI” setting
Microsoft (Bing)	Effective February 1, 2027 as crawled (transition deadline March 1, 2027)	Replaces legacy `NoArchive` and `NoCache` tags for AI training and grounding after March 1, 2027. Page-level training preferences override `robots.txt` rules	Separates AI Training from AI Grounding. Page-level grounding supports `Allow` (default), `DisallowExtended` (allows Bing, Copilot, and M365 grounding while excluding third-party Foundry API and Web IQ grounding), and `Disallow`, without affecting traditional Bing search rankings
OpenAI	Planned “in the coming months”	Will operate alongside existing `GPTBot` rules so publishers can express broad training preferences in one block	Separates search/browsing crawlers from training crawlers; keep explicit `GPTBot` rules in place until OpenAI confirms production activation
Anthropic	Implementation underway (author of the draft; exact completion date unannounced)	Described as a complement to existing crawler controls (such as `ClaudeBot`) rather than a replacement	Limiting crawler traffic itself still requires rules targeting the crawler's product token

Two vendor-specific behaviors deserve special attention before editing production files:

Google's implicit fallback and Gemini grounding coupling: When a site disallows __usage_ai-training and omits Google-Extended, Google plans to treat that as a Google-Extended disallow. Because Google-Extended governs both Gemini model training and grounding in Gemini Apps (as well as Grounding with Google Search on Vertex AI), a blanket shared opt-out will also opt the site out of Gemini Apps grounding unless Google-Extended is explicitly configured.

Microsoft's March 1, 2027 deprecation of NoArchive and NoCache for AI controls: Sites currently relying on NoArchive or NoCache to restrict Bing AI training or grounding must publish the new preferences before March 1, 2027. To preserve grounding restrictions while awaiting recrawl, Microsoft instructs publishers to both add on-page AI Grounding tags and record a temporary grounding preference in Bing Webmaster Tools (which expires on September 30, 2027 or upon recrawl).

3. Practical `robots.txt` Configuration Patterns

During the 2026–2027 transition period, the safest approach is a dual-stack configuration that adds __usage_ai-training while retaining existing vendor-specific blocks.

Pattern A: Opt Out of Generative AI Training Site-Wide Except `/press-kit/`

Combining Figure 1 from the IETF draft with current vendor tokens ensures immediate enforcement today and forward compatibility in 2027. Under RFC 9309 longest-match precedence, /press-kit/ (11 characters) overrides / (1 character).

Pattern B: Opt Out of General AI Training While Allowing Google-Extended

Following Section 4.5 of the draft (specific rules take precedence over general ones) and Google's announcement, a site can disallow __usage_ai-training broadly while explicitly setting Google-Extended if it chooses to remain eligible for Gemini Apps grounding.

4. Verifying Precedence and Catching Misconfigurations with Python

The three most common configuration mistakes are: (1) assuming User-Agent: * opts out of __usage_ai-training, (2) miscalculating path-length precedence on Allow exceptions, and (3) overlooking how omitting Google-Extended triggers Google's planned fallback behavior.

The dependency-free Python script below parses robots.txt groups and evaluates path decisions according to Sections 4.2 and 4.5 of draft-kelley-aipref-training-preference-signal-00 and Google's documented fallback rule.

Scope Verified Locally

Running the evaluation logic in a local Python environment confirmed three boundary cases:

Wildcard-only block (User-Agent: * with Disallow: /): Evaluating /blog/post returns shared_training_pref: "no_preference". A wildcard block removes the site from ordinary search crawlers without expressing any training preference under the new token.

Site-wide disallow with /press-kit/ exception: /articles/a resolves to disallow while /press-kit/logo.png resolves to allow, and effective_google_extended_2027 mirrors both when Google-Extended is omitted.

Shared token disallow with explicit Google-Extended allow: Evaluating /docs/api preserves shared_training_pref: "disallow" while overriding effective_google_extended_2027 to "allow".

This snippet checks prefix-based longest-match and token precedence; production validation should also account for wildcard characters (*, $) and CDN cache TTLs.

5. Pre-Deployment Checklist

Keep existing crawler rules active: Do not remove GPTBot, ClaudeBot, or Google-Extended rules while provider rollouts remain scheduled for late 2026 and early 2027.

Match each goal to the right control:

Generative AI training preference: __usage_ai-training and vendor training tokens in robots.txt

Google Search AI features (AI Overviews, AI Mode, Discover generative features): Search Console → Settings → Search generative AI

Microsoft Bing and Copilot grounding: Page-level ai-grounding preferences (Allow, DisallowExtended, Disallow) plus temporary Bing Webmaster Tools settings before March 1, 2027

Full search de-indexing: Page-level noindex meta tags or HTTP headers rather than blocking Googlebot

Track IETF draft revisions: draft-kelley-aipref-training-preference-signal-00 expires on May 3, 2027 unless updated. Review provider documentation again in early 2027 to confirm final token handling.