Google Will Support a Shared AI Training Opt-Out in robots.txt, but Not Until 2027

Google has announced plans to support a shared AI training control in robots.txt. John Mueller posted the announcement on the Search Central blog on 30 October. The idea is that a site owner sets one preference and every participating AI provider reads it, instead of maintaining a separate rule for each crawler.

Nothing has shipped. Google says it aims to support the new control, "along with other AI providers in 2027," and will post updates as it goes.

How it would work

The control is a shared "policy token" placed in robots.txt. It comes from the AIPREF working group at the Internet Engineering Task Force (IETF), the body that standardises internet protocols. Google calls its own proposal a draft.

The one rule Google spells out is this. If a site uses the token to opt out of training, and does not mention Google-Extended anywhere in the file, Google plans to treat that as if the site had disallowed Google-Extended. Google-Extended is the AI training control Google has offered since 2023.

Three details from the post matter for anyone who edits robots.txt:

• Company-specific rules keep working. Google says existing controls "will continue to function alongside this new joint signal." You can set a broad choice with the token and still give Google its own instruction through Google-Extended.
• An explicit Google-Extended rule wins. The opt-out is only read as a Google-Extended disallow when Google-Extended "is not explicitly specified."
• Allowing Google-Extended has a stated upside. Google says sites that specifically allow it "are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

What Google hasn't said

The post doesn't give the token's name or syntax, so there is nothing to add to a robots.txt file yet. It also doesn't name the other AI providers that will adopt the control. It says only that "other AI providers can handle it similarly," and that Google wants support to spread across the industry.

Google also describes the proposal as consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which treats robots.txt as a main route for publishers to state machine-readable preferences. The post doesn't say what Google would do if the IETF vocabulary changes before the standard settles.

What to do now

There is no action that unlocks this feature, but there is groundwork worth doing:

1. Write down your actual position. Decide whether you want to be used for AI training, for AI answers, for both or for neither. Those are different choices, and Google's post only covers training.
2. Audit your current robots.txt. Check which AI user agents you already block or allow, and whether a CDN or firewall rule is overriding the file. Our look at Cloudflare's September change shows how often the edge and robots.txt disagree.
3. Keep Google-Extended rules explicit if you care about Google specifically. Under Google's description, an explicit rule is what stops the shared token from being applied on your behalf.
4. Don't expect it to fix everything. A robots.txt preference is a request. It does nothing about crawlers that ignore the file.

Access policy is one part of how a site gets cited in AI answers. We cover that choice, and the rest of the picture, in our AI search optimisation work.

Sources

• Google Search Central: Supporting a shared AI training control for websites (30 October 2026)