Opt Out of AI Training on Your Website in Two Lines, Without Leaving Search

If you do not want AI companies to train their models on your website content, add two lines to your site's robots.txt file: User-Agent: __usage_ai-training and, below it, Disallow: /. On October 30, 2026, OpenAI, Google and Microsoft announced they will honor this shared signal: Microsoft starts on February 1, 2027, Google aims for 2027, and OpenAI says it will implement it in the coming months. The signal covers training only, so it does not remove your site from Google or Bing search results. Until each company switches it on, keep its existing controls in place. If you read our piece on Meta's new terms that let its bots crawl your website, this is the next step.

What exactly was announced?

Until now, a website owner who wanted to keep content out of model training had to know each company's crawler by name and write a separate rule for it: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and a new name every time a new company appeared. The new signal replaces that list with one shared name. It comes from an Internet-Draft filed with the Internet Engineering Task Force (IETF), the body behind many of the internet's technical standards, written by Kevin Kelley of Anthropic within the IETF's AI Preferences working group. The draft is dated October 30, 2026, expires on May 3, 2027, and is intended for the standards track, which means it is still a proposal, not a finished standard.

The idea is deliberately simple. The draft adds no new commands to robots.txt. It uses an ordinary group whose User-Agent line carries a special name, called a policy token, and the usual Allow and Disallow rules then say which parts of the site may or may not be used. For example, you can refuse training on the whole site but allow it on your press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The draft defines AI training as using content to modify the learned parameters of a generative AI model, meaning a model that produces text, images or other content. Models used only for classification, ranking or scoring are not covered.

Who will honor it, and when?

Two details in the announcements matter for any business with a website. First, Microsoft says it will stop reading the old NoArchive and NoCache tags as AI preferences after March 1, 2027; if your site relies on them, publish the new preferences before that date. Grounding transition settings open in Bing Webmaster Tools on December 15, 2026, only for sites already using those tags, and they expire on September 30, 2027. Second, the draft says a general rule written for all crawlers, User-Agent: *, does not count as a training preference. A blanket rule does not say «no training»; only the named token does.

Training is not the same as appearing in answers

This is the distinction most website owners miss. Training means your content helps build a future model. Answers, which Microsoft calls grounding, mean an AI assistant reads your page at the moment a customer asks a question and cites it with a link. The new token deals with training only. OpenAI keeps the two apart: GPTBot covers training, while OAI-SearchBot decides whether your site can appear in ChatGPT search answers. Microsoft goes further and adds a grounding preference set page by page, with three options: Allow; DisallowExtended, which limits use to Microsoft's own products such as Copilot; and Disallow. Microsoft warns that restricting grounding significantly reduces your visibility in AI answers, and that neither choice guarantees inclusion, traffic or payment.

Google is the exception to watch. Its Google-Extended control covers both training Gemini models and grounding in the Gemini apps, and Google says sites that explicitly allow Google-Extended are eligible to appear in Gemini app responses and receive clicks. So if you opt out through the shared token, Google will treat it as a Google-Extended disallow. You stay in Google Search, because Google-Extended does not affect inclusion or ranking there, but you give up grounding in the Gemini apps. If you want that visibility, set Google-Extended explicitly. For how AI answers pick their sources, see how to make your company show up in ChatGPT.

Should your company opt out?

There is no single right answer; it depends on what your website holds. Opting out makes sense if the content itself is the product: paid courses, original research, detailed price lists, or thousands of product descriptions your team wrote. Allowing it costs little if the site presents your services and you want your company name widely known. In both cases, keep the search and answer crawlers allowed, because they are what bring customers.

Five steps for this week

1. Read your current file. Open yourdomain.com/robots.txt in a browser. Look for a forgotten User-agent: * with Disallow: /, or a rule that blocks OAI-SearchBot, Googlebot or Bingbot.
2. Decide two things separately. Training: yes or no. Answers: yes or no. Write the decision down so the next developer does not undo it.
3. Add the shared token and keep the current rules. None of the three companies honors the token yet, so keep your GPTBot, ClaudeBot and Google-Extended rules alongside it until each one does. Anthropic's help page still documents ClaudeBot as the way to opt out.
4. Check your Bing tags. If your pages use NoArchive or NoCache, plan the move to the new preferences before March 1, 2027.
5. Put anything private behind a login. robots.txt is a public, voluntary signal: the draft lets private agreements override it, and a preference applies to what is fetched after you publish it, not to what was collected before.

The Origami view

For years, robots.txt was a file nobody at the company remembered, written once by whoever built the site. That is changing: it now decides whether your content trains AI models, whether your pages appear in AI answers, and soon how Microsoft treats each page. One shared name is real progress over a list of crawler names that grew every month, but it does not replace a decision. What we often see is a website where nobody knows who last edited robots.txt, or a content delivery network blocking crawlers the owner wanted to welcome. At Origami we build websites, platforms and custom systems, and we treat these settings as part of the build, written down and reviewed. If you want your robots.txt, CDN rules and AI visibility reviewed together, contact us.

Sources

• OpenAI: Making it easier for websites to express AI training preferences, October 30, 2026
• Google Search Central Blog: Supporting a shared AI training control for websites, October 30, 2026
• Microsoft Bing Blog: Your content, your choice: clearer preferences for training and grounding, October 30, 2026
• IETF: Internet-Draft draft-kelley-aipref-training-preference-signal-00
• Google for Developers: Google's common crawlers, Google-Extended section
• OpenAI: Overview of OpenAI crawlers
• Anthropic Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?