Before You Block AI Training in robots.txt, Decide What Your Site Is Actually Protecting
On October 30, 2026, Google’s John Mueller announced that Google plans to support a shared AI training control in robots.txt: a single “policy token” that lets a website express its training preference to many AI providers at once, instead of maintaining a separate instruction for each one. The mechanism comes from the AIPREF working group at the IETF. Google said it aims to support the control, together with other AI providers, in 2027, and that company-specific signals such as Google-Extended will keep working alongside it.
For most site owners this is good news, because the current situation is messy. Blocking training today means knowing every crawler’s name and keeping the list current. A shared token is simpler. It also makes the decision easier to take casually, and that is the problem this article is about. One line in robots.txt can quietly change how a site is discovered.

How the AI training controls fit together — original conceptual diagram based on Google’s announcement; no performance data.
What the announcement does and does not say
Three details matter. First, if a site uses the shared token to opt out and does not explicitly mention Google-Extended, Google plans to treat that as if the publisher had disallowed Google-Extended. Second, the announcement states that websites which specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. Third, Google describes the proposal as consistent with the EU Code of Practice for general-purpose AI model providers, which recognizes robots.txt as a primary way for publishers to express machine-readable preferences.
What the post does not say is just as important. It does not claim that opting out of training changes ordinary Google Search ranking, and it does not describe the shared token as a substitute for existing crawler rules. It is a preference about training, carried in a file most sites already have.
Separate three things that get bundled together
When a team says “block AI,” it usually means one of three different things. Sorting them out before editing robots.txt prevents an expensive mistake.
- Training. Whether a provider may use your pages to train models. This is what the shared token and Google-Extended address.
- Grounding and citation. Whether an assistant may fetch and cite your page when answering a user. Google’s post ties eligibility for grounding in Gemini Apps to allowing Google-Extended, so for Google the two are connected.
- Indexing for Search. Whether Googlebot may crawl and index the page for classic results. This is controlled by ordinary robots.txt rules and meta robots tags, not by the training token.
A publisher that objects to training but wants citations has a real trade-off to weigh for Google specifically. A business site that lives on product and service pages may care far more about being cited in an answer than about whether a model learned from its FAQ. A newsroom with licensing deals may decide the other way. Neither answer is wrong; the mistake is making the choice without naming it.

Three decisions that are often made with one line — original conceptual diagram; no performance data.
Inventory what the file already says
Before adding anything, read the current robots.txt and the meta robots tags on key templates. Many sites carry rules written years ago by a developer who has since left: a disallow on a folder that now holds the pricing page, a sitemap line pointing at a dead URL, or a blanket block that was meant for a staging server. Adding a new AI token on top of an unexamined file compounds whatever is already wrong.
Free checks are enough for this pass. The free crawler and audit tools on SEOtools can show which pages are blocked, which are indexable and which carry conflicting directives, so the team edits from a verified list rather than from memory. Export the result and keep it with the decision; it is the “before” picture if anything changes later.
Decide per section, not per site
The shared token is site-wide by design, but the business decision rarely is. A documentation area may be valuable training data and harmless to share. A library of original research may be the asset the company wants licensed, not scraped. Commercial landing pages are usually the pages where being cited matters most.
Write the decision down as a short table: section, what it contains, whether training is acceptable, whether citation is wanted, and who signed off. If the shared token cannot express the nuance a section needs, the company-specific controls still can, and Google said they will continue to function. The point of the table is that the robots.txt edit follows a decision somebody can defend, rather than the other way round.
Watch for the second-order effects
Changes to crawler access rarely show up as a single clean metric. Three things are worth watching for a few months after any edit.
- Impressions in generative AI features in Search Console, which Google began reporting separately in 2026, to see whether visibility inside AI answers moved.
- Classic Search clicks on the same sections, to confirm that a training preference did not accidentally become an indexing block.
- Referral traffic from assistant products, where your analytics can distinguish it, to see whether citations changed.
Record the edit date as an annotation. Without it, a drop six weeks later will be blamed on an algorithm update, a redesign or the weather, and the real cause will be a line nobody remembers adding.

A decision table the next person can read — original conceptual diagram; illustrative rows, no performance data.
Prepare for 2027 now
Google said support for the shared control is planned for 2027 and that updates will follow as the IETF work continues. That leaves time, and the useful way to spend it is not drafting the robots.txt line but settling the policy behind it. Companies with legal, editorial and marketing stakeholders rarely agree in a week.
If the site has no one who owns search visibility, this is also a sensible moment to bring in outside help, because the decision touches crawling, indexing and measurement at once. An English-language agency such as SEOquick would typically start by auditing the current crawl rules and the pages that earn commercial traffic, so the AI preference is set with knowledge of what is at stake rather than as a reflex. Whoever does it, insist on the inventory and the per-section table before anyone touches the file.

Four steps before the token goes live — original conceptual diagram; no performance data.
The shared token will make a hard decision easy to execute. The work for a site owner is to make sure it is still a decision.
Want to publish a guest post on aamax.co?
Place an order for a guest post or link insertion today.
Place an Order