OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that lets publishers opt out of being scraped to train AI systems. The development was first reported by Search Engine Roundtable and covered by The Verge, and it consolidates what has so far been a company-by-company patchwork of controls.
The central change is consolidation. Instead of setting AI training preferences separately with each company, publishers can apply one set of preferences across several AI providers. The mechanism is built around a shared policy token signal, proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of an Internet Engineering Task Force effort.
The signal lives within a website's robots.txt file, which has long been the mechanism for telling crawlers what they may and may not fetch. Extending that familiar file to AI training preferences lowers the implementation burden for publishers who already manage crawler rules there.
Key details, including rollout timing, the full technical specification, and how compliance will be checked, are not clear from the reporting so far. Publishers should therefore treat the announcement as a direction of travel rather than a finished control, and should not yet remove their existing company-specific blocks in favour of the new signal alone.
The distinction between training and other uses also matters. A training opt-out governs whether content is used to change a model's learned parameters. It does not by itself decide whether an AI answer can cite or quote a page today, which is a separate question governed by separate controls.
The shared standard arrives alongside separate October 30 changes from Google and Microsoft. Google said it plans to honour a shared AI training opt-out in robots.txt, while Bing split AI training from AI grounding and set a March 1, 2027 deadline for anyone still relying on NoArchive as an AI signal.
Bing will honour its new training and grounding preferences from February 1, 2027. Taken together, the announcements turn one vague question, should I block AI, into three separate decisions: can your content train models, can AI answers quote and link it, and can Google's AI Overviews and AI Mode show it. Each has its own control.
First, audit what your current robots.txt and crawler rules actually say, and which AI crawlers you are currently blocking or allowing. Many sites have accumulated rules they no longer understand, and a shared standard is a good moment to clean them up.
Second, decide your position on each use separately. Opting out of training while staying visible in AI search answers is a coherent choice for many publishers, since training and citation have different costs and benefits. Using the right control for each purpose keeps visibility while protecting the asset you care about.
Training opt-outs reshape the content economy that link building depends on. If more publishers restrict training use, the value of original, citable content rises, because AI systems will still need fresh, trustworthy sources to ground their answers, and those sources earn links when cited.
For content creators, the practical move is to keep producing original work that is worth citing, and to use the new controls deliberately rather than blocking everything in panic. A page that is blocked from every AI use cannot be cited, and a page that cannot be cited cannot earn the links that AI-era visibility increasingly runs on.
An Internet Engineering Task Force draft is an early step, not a finished standard. Individual drafts can change substantially, stall, or be replaced, and adoption by the four named companies is a signal of intent rather than a guarantee of identical implementation. Publishers should expect differences in how each company honours the signal, at least at first.
That is normal for web standards. Robots.txt itself began as a convention before it became a formal specification, and it worked because the major crawlers honoured it consistently. The shared training signal will succeed or fail on the same test: whether publishers can set it once and trust that the major AI providers respect it the same way.
Sources: Search Engine Roundtable and The Verge, reported October 31, 2026.
Linkbaits helps you create content that earns citations and backlinks. Start with the free tools and build your first linkable asset this week.
Start free trial