Google, OpenAI, Microsoft and Anthropic back one robots.txt line for AI training opt-outs

On 30 October 2026, Microsoft, Anthropic, OpenAI and Google each said they will support a single, shared way for websites to tell AI companies whether their content may be used to train AI models. The signal is one rule group in robots.txt, the plain text file that sites already use to give instructions to crawlers. It comes from a new proposal at the IETF, the body that develops many core internet standards.

Until now, a site that wanted to opt out of AI training had to list each company's crawler by name (GPTBot, Google-Extended, ClaudeBot and so on) and keep that list up to date. The new signal is meant to replace that guesswork with one vendor-neutral line.

How it works

The proposal, draft-kelley-aipref-training-preference-signal, was posted to the IETF's ai-control mailing list by Kevin Kelley on 30 October. It uses existing robots.txt syntax with a special name in the User-Agent line. Instead of naming a crawler, that name, __usage_ai-training, names a use of the content. The example in the announcement looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Read under the draft, that says: please don't use anything on this site for AI training, except the press kit. A few details matter for site owners:

• Your current file says nothing yet. Wildcard rules (User-Agent: *) never match the new token, so existing robots.txt files express no training preference until you add the group.
• It is a preference, not a lock. The draft keeps preferences separate from crawl access. If you want to stop a crawler from fetching pages at all, your normal robots.txt blocks still do that job.
• It is still a draft. This is an individual Internet-Draft offered as input to the IETF's AI Preferences working group, not a finished standard. Details may change.

Where each company stands

Anthropic. Kelley's message says "Anthropic has already started implementing support for this draft", with support rolling out across its services over the coming months alongside its existing crawler controls.

Microsoft. The Bing blog post from 30 October gives the most detail. Bing will read the training preference from robots.txt for a whole site or a folder, or from a page-level meta tag or X-Robots-Tag header such as ai-training:disallow, which overrides robots.txt for that page. Microsoft is also adding a separate grounding preference. Grounding means using your page to inform an AI answer in real time, for example in Copilot, rather than to train a model. It is set per page only, with ai-grounding:disallow or ai-grounding:disallowextended. The "extended" option keeps your content usable in Microsoft's own products such as Copilot and Microsoft 365 but not in grounding services Microsoft sells to third parties. If you publish nothing, both default to allow.

OpenAI and Google. Both published posts backing the shared signal: OpenAI's "Making it easier for websites to express AI training preferences" and Google's "Supporting a shared AI training control for websites". We could not read the full text of either post today, so check them directly for each company's rollout timing and how the signal interacts with their existing controls such as GPTBot and Google-Extended.

Key dates from Microsoft

• 15 December 2026: sites that currently use NoArchive or NoCache can record a temporary grounding preference in Bing Webmaster Tools.
• 1 February 2027: Bing starts honouring the new preferences as it recrawls pages.
• 1 March 2027: after this, Bing stops treating NoArchive and NoCache as training or grounding preferences, so sites relying on them need to publish the new ones.
• 30 September 2027: temporary grounding preferences expire (earlier if the page is recrawled).

Limits and trade-offs

• It relies on crawlers choosing to comply. Four large companies have signed up. Others may not.
• Opting out has a cost for visibility. Microsoft says choosing Disallow or DisallowExtended for grounding can noticeably reduce how often your content appears in AI experiences, though it does not affect Bing search eligibility. It also says "Neither choice guarantees inclusion, traffic, or payment."
• Nothing is instant. A new preference takes effect only after the page is recrawled.
• Training and answers are different decisions. Blocking training does not by itself stop your pages being used to answer questions in AI search. Microsoft handles that with the separate grounding setting.

Who should care

Anyone whose website content is part of what they sell: publishers, course creators, research and data businesses, agencies with original guides. If you want AI assistants to recommend and cite you, you will probably leave training and grounding open. If your content is the product, this is the cleanest opt-out yet offered.

What to do this week

1. Decide your position on two separate questions: may AI companies train on your content, and may AI assistants use it to answer questions? Marketing pages and paid content often deserve different answers.
2. Look at your current robots.txt. Note which named AI crawlers you already block, and keep those rules for now. The new token adds to them rather than replacing them.
3. If you want to opt out of training, add a __usage_ai-training group, using Allow for anything you want models to learn from, such as a press kit or public docs.
4. If you use NoArchive or NoCache, put 1 March 2027 in the calendar and read Microsoft's post before then.
5. Watch the IETF draft. It may change before it becomes a standard.

If you'd like a second pair of eyes on how your site's content, search visibility and AI tooling fit together, that is the kind of work The DevDrift helps teams with.

Sources

• Microsoft Bing: Your content, your choice: clearer preferences for training and grounding (30 October 2026)
• IETF ai-control list: New draft: draft-kelley-aipref-training-preference-signal-00 (30 October 2026)
• IETF Datatracker: draft-kelley-aipref-training-preference-signal
• OpenAI: Making it easier for websites to express AI training preferences
• Google Search Central: Supporting a shared AI training control for websites