🔍 Read the full analysis: What Four Major AI Companies’ Shared Opt-Out Does—and Doesn’t—Cover on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal asking them not to use a site’s content to train generative AI models. The signal is voluntary, does not block crawling or cover AI answers based on live web content, and will take effect on different schedules; Meta has voiced support but made no commitment to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will support a shared website signal asking them not to use a publisher’s content to train generative AI models. The proposed robots.txt rule could replace the need for separate training opt-outs for each participating company, but it is a voluntary request, not a technical block, and it does not cover content used to generate AI answers from the live web.
The signal is a group in a site’s robots.txt file: User-Agent: __usage_ai-training followed by Disallow: /. It appears in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley and published on October 30. An Internet-Draft is a proposal, not an adopted internet standard.
The companies have described different implementation schedules. Anthropic says it has started implementing support, though it expects rollout across its services to take time. OpenAI says it aims to implement the signal “in the coming months.” Google’s documentation points to early 2027. Microsoft says it will honor the training preference from February 1, 2027, as it crawls websites.
The shared rule would not immediately replace existing company-specific settings. OpenAI says its current crawler controls will continue to apply. Google says it plans to treat the shared signal as a Google-Extended disallow when a site has no separate Google-Extended rule. Microsoft says its new preferences will replace NoArchive and NoCache as training or grounding preferences after March 1, 2027. Its separate page-level setting for AI Grounding addresses web content used in answers from Bing, Copilot and Microsoft 365.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A Simpler Training Preference, Not a Block
For website operators, one shared line could make it easier to state a preference across several major AI providers. The current setup can require separate directives for crawlers such as GPTBot, Google-Extended and ClaudeBot, while Microsoft has used different page-level signals. Keeping the old rules during the staggered rollout may help avoid a gap before each company begins honoring the common one.
The practical limit is that robots.txt communicates a publisher’s preference; it does not enforce that preference. The companies have described honoring the signal, but the draft does not provide a technical mechanism that prevents a crawler from accessing a page. It also cannot remove content from models that have already been trained. For publishers, the signal is a clearer request, not a guarantee of control over past or future use.
The distinction between training and answers also matters. A site can express a preference against training while leaving other uses of its pages unaffected. Microsoft’s separate AI Grounding setting makes that distinction explicit. The shared training signal does not itself opt a page out of search results, block crawling, promise traffic, or create a payment right. Microsoft says neither choice guarantees “inclusion, traffic, or payment.”
Products Worth Considering
Apple MacBook Air 15-inch (2026) with M5 chip, 16GB memory, 512GB SSD, Silver
Jackery Explorer 500 v2 Portable Power Station, 512Wh, 500W
Samsung 27 Odyssey OLED G5 (G50SF) QHD QD-OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Four Rollouts, One Draft
The four announcements refer to the same proposal developed through the IETF’s AI Preferences working group. The Internet Engineering Task Force develops technical standards, but this draft has not been adopted as one. The draft’s existence and the companies’ statements are confirmed; its status as an official standard is not.
The companies have made distinct commitments rather than announcing a single launch date. Anthropic says implementation has begun, while OpenAI and Google have described future targets. Microsoft has set a date for its support and introduced a separate preference for AI grounding. Those differences mean a website’s settings may need to account for both the shared rule and company-specific controls during the changeover.
Meta has not made the same implementation commitment. Its communications chief, Andy Stone, said the company strongly supports publishers’ ability to express preferences and called a common signal an important step. That statement expresses support for the idea, but does not say Meta will honor the rule. The source report also found no statement on adoption from Apple, Amazon, xAI or Perplexity in the material it reviewed.
““has already started implementing support for this draft””
— Anthropic
Adoption and Enforcement Remain Open
Participation beyond the four companies is unsettled. Meta has voiced support without confirming it will follow the signal, and the source material reports no adoption statement from several other large AI or technology companies. The list of services covered by each company’s implementation is also not fully specified.
The proposal’s standards status is another open issue. It is an individual Internet-Draft and has no formal standing in the IETF standards process at present. At a working-group meeting on October 21, one of the chairs said, “We haven’t declared consensus on anything yet.” Whether the draft changes, gains group support or becomes a standard remains unknown.
It is also not clear how consistently the preference will be observed across services or how publishers could respond if a company does not honor it. The signal does not block access, and the announcements do not describe an enforcement or verification system. The announced controls concern the use of web content; they are not settings for a person’s own conversations with an AI assistant.
Watch the Staggered Rollout
Publishers can add the shared training preference if they want to communicate that request, while retaining existing company-specific rules until providers have implemented the new signal. They should distinguish training preferences from AI-answer preferences and check each provider’s documentation for the settings it recognizes.
The next dated milestones are Anthropic’s continuing rollout, OpenAI’s stated target of the coming months, Google’s target of early 2027 and Microsoft’s February 1, 2027 start. Microsoft says NoArchive and NoCache stop counting as training or grounding preferences after March 1, 2027. Further company guidance and any development in the IETF process will determine how broad and consistent the shared signal becomes.
Key Questions
What does the shared AI opt-out signal do?
It places a training preference in robots.txt, asking participating companies not to use a site’s content to train generative AI models. The proposed rule uses the user-agent name __usage_ai-training with Disallow: /.
Does the rule prevent AI companies from crawling a website?
No. The signal communicates a preference about training use; it is not a technical barrier to crawling. The announcements do not describe an enforcement system that prevents access.
Does it stop a website from appearing in AI-generated answers?
Not by itself. The shared rule concerns training, not content used to produce answers from the live web. Microsoft has announced a separate AI Grounding preference for that use in Bing, Copilot and Microsoft 365.
When will the companies honor the signal?
Dates vary. Anthropic says implementation has begun; OpenAI aims to support it in the coming months; Google points to early 2027; and Microsoft says its support starts on February 1, 2027. Meta has not confirmed adoption.
Is the signal an official internet standard?
No. It is an individual IETF Internet-Draft, not an adopted standard. The working group had not declared consensus on the proposal as of its October 21 meeting.
Source: ThorstenMeyerAI.com
