Google plans to adopt a proposed shared signal that would let website owners communicate their preferences on AI training to multiple providers through a single entry in the robots.txt file, rather than setting separate rules for each company.
The signal is being developed by the Internet Engineering Task Force (IETF), an internet standards body. Google said it expects to support the mechanism in early 2027, joining companies including Microsoft, OpenAI and Anthropic in backing the proposal. The draft is still under discussion and has not been finalized.
Now loading...
Under the plan, publishers would use a common policy token to indicate whether participating AI providers may use their content for training. Google said it would treat a site’s use of the token to block training as equivalent to disallowing its Google-Extended crawler, unless the site separately specifies otherwise. Existing provider-specific controls would remain available alongside the shared signal.
Google has also updated its crawler documentation to say it intends to support the proposed signal. The company said the effort responds to calls from publishers, civil society groups and regulators for a simpler, industry-wide way to set AI training preferences. It also pointed to the European Union’s AI rules, which recognize robots.txt as a way for publishers to express machine-readable preferences.
The proposal does not replace Google’s existing controls, including options related to its search generative AI features. Websites that specifically permit Google-Extended can be eligible to have content used to ground responses in Gemini Apps, according to Google.
Some participants in the IETF discussions have welcomed the process but said significant disagreements remain. One publisher said a provision concerning user-initiated activity could limit the signal’s usefulness in protecting artists and creators from AI-generated copies. The IETF process is open to public participation, and work on the draft is continuing.
Google’s planned support is outlined in its announcement, while the proposal remains available as an IETF draft.
