Google, OpenAI, Microsoft, and Anthropic each said on October 30, 2026 that they plan to honor one shared robots.txt preference for AI training. The signal sits in an individual Internet-Draft, draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley of Anthropic. It uses the policy token __usage_ai-training inside an ordinary robots.txt group.
For site owners and affiliate bloggers, this is a practical decision. You can state one training preference for every participating provider. You still have to weigh what that preference does to AI answer visibility, and you still keep the crawler rules you already run for GPTBot, Google-Extended, and the rest.
What The Individual Draft Defines
The IETF datatracker lists the document as an active Internet-Draft (individual). The datatracker states it is not endorsed by the IETF and has no formal standing in the IETF process. Treat it as a published proposal with company promises of future support, and stop short of calling it a finished rule that every crawler must obey today.
The draft adds no new robots.txt grammar. A site writes a normal group whose User-Agent line names the policy token. The figure in the draft uses the following block.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That reads as a preference against AI training for every path except /press-kit/, and a preference in favor of training for paths under that folder. Lookup follows the usual RFC 9309 path matching and longest-match rules. Parsers that do not know the token ignore the group the same way they ignore any unknown user agent.
The draft is clear that the token is a preference about use. It does not issue a crawl order. Ordinary rules for a crawler product token still govern fetching. Existing GPTBot and Google-Extended groups keep their meaning. The signal also cannot undo past scraping. Preferences apply to content retrieved while a given robots.txt file is in effect.
What Each Company Said It Will Do
Anthropic is furthest along in its own words. On the AI Preferences mailing list, Kelley wrote that Anthropic has already started implementing support for the draft, and that the work complements existing crawler controls and does not replace them.
OpenAI said it will honor the common robots.txt signal for foundational generative AI training, aims to implement it in the coming months, and will keep its existing crawler controls in place.
Google Search Central, in a post by John Mueller, said Google aims to support the shared control in 2027. Google’s crawler docs point to early 2027. When a site uses the shared token to opt out of training and Google-Extended is not set explicitly, Google plans to treat that as a Google-Extended disallow. Google-Extended still controls whether content may train future Gemini models and whether Gemini Apps can ground answers and send clicks. Sites that specifically allow Google-Extended stay eligible for those Gemini Apps responses and clicks. Google search ranking is a separate system. The shared token does not change how Googlebot ranks pages.
Microsoft Bing said it will honor the new AI Training preference from February 1, 2027 as it crawls. Microsoft also adds a separate page-level AI Grounding preference, so training and grounding can be chosen independently on Bing’s side. Microsoft states that these preferences do not establish licensing or payment arrangements, and that neither choice guarantees inclusion, traffic, or payment. Bing search eligibility stays subject to normal crawl and indexing rules. Older NoArchive and NoCache tags stop counting as training or grounding signals after March 1, 2027 for sites that have not moved to the new preferences.
The Tradeoff Site Owners Should Price First
A blanket training opt-out looks simple. At Google it also becomes a traffic choice. Tracking Desk warned that a blanket opt-out with no Google-Extended group also cuts Gemini grounding links and referral clicks, and that retailers in particular could lose AI-assistant traffic without noticing. Google’s own blog ties Google-Extended allowance to eligibility for Gemini Apps grounding clicks. There is no Google setting today that says “train no, cite yes” with a single shared line.
Microsoft’s grounding preference is separate, so Bing’s story is less bundled. Still, Microsoft warns that Disallow or DisallowExtended for grounding reduces brand visibility in AI experiences. If your affiliate pages earn useful AI-assistant referrals, decide the citation cost before you ship a sitewide Disallow for __usage_ai-training.
Our October 28 AEO and AI Overviews guide covers how Google separates AI Overviews and AI Mode from Google-Extended style training and grounding controls. Keep that map open while you edit robots.txt, and keep our Gemini 4 Carbon AI News brief nearby when you track how model providers treat gated versus open web sources.
A Practical Path For Affiliate And Niche Sites
- Inventory the GPTBot, Google-Extended, and other AI crawler groups you already publish. Leave them in place. The shared token does not replace them.
- Decide whether Gemini Apps citations and other AI-answer links matter to your traffic mix. If they do, plan an explicit Google-Extended group before you add a sitewide training Disallow.
- If you want a training preference for most of the site and still want a public kit available for training, use path Allows the way the draft example does for
/press-kit/. - On Bing, treat training and grounding as separate choices. Publish page-level grounding tags if you need that split, and watch Microsoft’s February 1 and March 1, 2027 dates if you still rely on NoArchive or NoCache.
- Log the robots.txt change in your analytics notes. A later drop in an AI Assistant referral channel is easier to explain when the date sits next to the file edit.
- For contract or copyright questions about training on your archives, talk to counsel.
JamoraquAI Take
One shared robots.txt group is useful once the big providers honor it. It remains a preference with staggered start dates. It does not act as a live switch that rewrites the past. Affiliate publishers should price AI-answer visibility next to training control, keep GPTBot and Google-Extended rules that already work, and wait for each company’s implementation notes before treating the token as fully in force.
FAQ
Is This An IETF Endorsed Rule?
No. Datatracker lists it as an individual Internet-Draft with no formal standing. Companies have promised future support on their own timelines.
Does The Token Stop Crawlers From Fetching Pages?
The draft treats it as a use preference without crawl-directive force. Crawler product-token rules still control fetching. Providers may choose to skip pages they cannot use for training, yet that is optional behavior.
Will Adding The Token Change Google Or Bing Rankings?
Google and Microsoft both frame the preference as separate from classic search ranking and inclusion. Do not expect a ranking boost or penalty from the token alone.
Can I Undo Training On Content Already Scraped?
No. The draft ties preferences to the robots.txt file in effect when content was retrieved. Past copies sit outside this signal.
Should I Remove GPTBot Or Google-Extended Rules After I Add The Token?
No. OpenAI and Google both say existing company-specific controls continue to apply. Keep them until a provider publishes a clear conflict rule you trust.
