OpenAI, Google, Microsoft and Anthropic back one shared signal that lets websites opt out of AI training
OpenAI, Google, Microsoft and Anthropic have pledged to honour a single, shared signal that lets publishers and other websites opt out of having their content used to train generative AI models. Today site owners must block each company's crawler separately. Anthropic has already started supporting the new rule, Microsoft will do so from 1 February 2027, Google in 2027 and OpenAI in the coming months.
The signal is a proposal at the Internet Engineering Task Force (IETF) written by Kevin Kelley, a technical lead for standards at Anthropic. It lives in a website's robots.txt file as a special "policy token": a group starting with User-Agent: __usage_ai-training followed by Disallow: / tells all participating companies that the site does not want its content used for AI training, while Allow lines can make exceptions, for example for a press kit.
The rule covers only future training. It cannot undo content that has already been collected, and it does not stop normal crawling or remove sites from search results. Microsoft said websites will stay eligible for Bing regardless of their choice and that it will offer separate controls for training and for "grounding", when an AI tool looks up a verified source to answer a question. Google will treat the signal as a stop sign for Google-Extended, the crawler that gathers material for its Gemini models. Microsoft and Google warned that opting out could reduce the chance of a site's content appearing in AI apps.
The move comes as publishers fight a wave of bots scraping their articles. The New York Times has been suing OpenAI and Microsoft since December 2023 for using its articles without permission; both companies invoke fair use. Cloudflare has blocked AI bots by default on new customer sites since July 2025. Meta said it supports a common signal but did not say whether it would honour it.