One robots.txt Line for AI Training: OpenAI, Microsoft, Google and Anthropic Back a Shared Signal

The syntax

The token is defined in an individual IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, dated October 30. Its sole author is from Anthropic. It is not a working-group document and is not endorsed by the IETF, but it draws on the AI Preferences working group's vocabulary. Microsoft's Bing post gives the examples:

User-Agent: __usage_ai-training
Disallow: /

User-Agent: __usage_ai-training
Disallow: /example-path/

Three rules in the draft matter for operators. First, "wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token", so an existing User-Agent: * block says nothing about training. Second, if no group names the token, "no preference has been expressed". Third, "This policy token is not a directive for crawling behavior." It describes how content may be used, not whether a crawler may fetch it. Existing per-crawler rules keep working; OpenAI says its current crawler controls "will continue to apply and be respected."

Microsoft separates training from grounding

Microsoft's post adds a second, Microsoft-only preference. AI Training covers changing a generative model's learned parameters. AI Grounding covers using a page "as input to help inform AI-generated responses", as in Copilot or AI answers in Bing. Grounding is set per page, with a meta tag such as <meta name="Bingbot" content="ai-grounding:disallow"> or an X-Robots-Tag header. There is no site-wide grounding rule in robots.txt. A third value, DisallowExtended, keeps grounding inside Microsoft-operated products and out of services Microsoft sells to third parties.

Both preferences default to Allow, and neither affects eligibility for Bing search results. The dates are firm: Microsoft starts honouring the new preferences on February 1, 2027, as it recrawls. Sites that rely on the older NoArchive or NoCache tags must publish the new preferences by March 1, 2027, after which the old tags are no longer read as training or grounding preferences. A temporary grounding setting opens in Bing Webmaster Tools on December 15 for those sites. Microsoft also warns that choosing Disallow for grounding "will significantly reduce your content's visibility in AI-generated experiences", and that neither choice guarantees "inclusion, traffic, or payment."

What remains uncertain. The draft is revision 00 and may change. OpenAI and Google have not published implementation details or dates. The draft covers future training, not models already trained, and Microsoft notes that a published setting does nothing until the page is recrawled.

Analysis: The useful change is that one line can now reach several providers, but only for training. Grounding, which decides whether an assistant quotes your page in an answer, is still provider-specific. Belgavi's Book Store could let assistants ground answers on its public return policy while refusing training on its book reviews. Jenya would need two changes for that: a __usage_ai-training group for /reviews/ in robots.txt, and no grounding tag on the policy pages. She should also not assume the existing User-Agent: * rules already cover training. Under the draft, they do not.

Sources and evidence

• Making it easier for websites to express AI training preferences (OpenAI, October 30)
• Your content, your choice: clearer preferences for training and grounding (Microsoft Bing Blog, October 30)
• Machine-Readable Training Preference Signal (IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00)
• Google, Anthropic and others to support new AI crawling opt-out standard (SiliconANGLE, October 30)