Opt out of AI training: the shared robots.txt rule explained

If you have a blog, a channel or a website, here's a question that affects you: are AIs allowed to read your pages for training? Until now, the answer was a headache, with a different instruction for every bot. In late October 2026, the big players started agreeing on a single line for saying no.

Heads up: it's real progress, but there are gaps. Let's sort it out.

Key takeaways

• A draft standard submitted to the IETF on October 30, 2026 proposes a single instruction in robots.txt: __usage_ai-training.
• It was written by Kevin Kelley, an engineer at Anthropic, and is backed by Google, OpenAI and Microsoft.
• It's not an official standard yet: it's a proposal under discussion.
• If you put nothing in your file, training stays allowed by default.
• This rule only covers training, not how your content shows up in AI answers.

What is robots.txt?

Every website can have a small text file at its root, robots.txt. It tells bots (Google's, Bing's, and so on) what they're allowed to visit. Think of it as house rules posted on the door: honest bots follow them, but nothing technically forces them to.

The problem with AI: each company had its own bot name to block (GPTBot for OpenAI, Google-Extended for Google...). You had to know the list and keep it up to date.

What's new: one line for everyone

The draft, called draft-kelley-aipref-training-preference-signal, is being discussed in the IETF's AIPREF working group, the body that standardizes part of the internet. The idea is simple: instead of a bot name, you use a shared "name," __usage_ai-training, in place of a bot in the User-agent line.

In practice:

User-agent: __usage_ai-training
Disallow: /

That means "don't use any page on this site to train a generative AI." Conversely, an Allow gives permission. Just like a regular robots.txt, you can go case by case: refuse everything except one folder, your press page for example.

Who's on board, and when?

According to Blog du Modérateur, a French tech news site, the timelines differ from company to company:

• Anthropic already applies the signal and is extending it to all its services.
• OpenAI plans to respect it "in the coming months."
• Microsoft: Bing will comply starting February 1, 2027.
• Google: support in 2027, with no exact date.

For Google, an opt-out through the shared rule blocks Google-Extended, unless you've set a specific rule for that bot.

What it doesn't fix

Here's what the hype often leaves out.

It's not a legal requirement. Like any robots.txt, it relies on companies' goodwill. A company that isn't part of the deal can ignore it.

It doesn't change the past. Content already used to train models isn't removed.

It doesn't control display. Opting out of training doesn't necessarily stop your site from appearing in AI answers, like Google's AI Overviews or Copilot: those settings are separate. Microsoft says your site stays eligible for Bing results whatever you choose.

Existing blocks stay separate. Your current rules for GPTBot or Google-Extended keep working, independently of the new line.

Why it matters now

It's no coincidence this is happening now. Publishers are getting fed up: on the same day, five French publishers' organizations filed a complaint against Google over its AI answers and their impact on traffic. And websites are increasingly shutting out bots, as we saw with AI agents blocked by Amazon or JPMorgan. The AI giants have every reason to offer a simple way to opt out before the law forces them to.

So, what changes?

If you publish content, it's good news: eventually, one line will be enough to state your choice, instead of a list of bots to keep track of. But nothing will really be in effect everywhere before 2027, and the standard hasn't been approved yet.

If you don't publish anything, it still concerns you: it's a sign that the question of the "right to train on our content" is moving from the courts toward shared technical rules.