Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping
By Google News AI ·
Google and Microsoft have joined OpenAI and Anthropic in promising to honor a single line in robots.txt that tells AI developers whether a website's content can be used to train their models.
Until now, a site that wanted to say no had to write a rule for each company's crawler, and sometimes for several crawlers run by the same company. The shared signal is meant to replace that sprawl with a single entry that every participating provider reads the same way.
That is genuine progress for overworked webmasters, but the promise is narrower than much of the coverage suggests: The new line states a preference about how content may be used rather than stopping bots from fetching it; it is not an approved standard, does nothing about content already collected, and none of the four major players fully honor it yet.
OpenAI's announcement drew much of the attention, but the company casts itself as one participant among several in an Internet Engineering Task Force (IETF) effort, and the mechanism isn't its own. It comes from an Internet-Draft published the same day by Kevin Kelley of Anthropic, which names a policy token, "__usage_ai-training", in the User-Agent line of an otherwise ordinary robots.txt group.
Training is defined by the IETF's AI Preferences (AIPREF) working group as the process of changing a generative model's learned parameters, excluding models used purely for tasks such as classification or ranking. OpenAI describes what it will honor as "foundational generative AI training," which sounds narrower than the draft's definition.
Calling it a standard may be premature: The IETF's own tracker labels the document an individual draft that the IETF has not endorsed, and that has no formal standing, and it lapses on May 3, 2027 unless revised.
Anthropic is the only one of the four major AI players to say it has started. OpenAI aims to implement the signal over the next few months but has given no date. Microsoft will begin honoring the shared preference as it crawls from February 1, 2027, while Google is aiming for 2027.
Each company is also wiring the line into its own controls differently.
Google says that when a site uses the shared token to opt out without separately specifying Google-Extended, it plans to treat that "as if the publisher had disallowed Google-Extended." Because Google-Extended also governs grounding in Gemini Apps, a site that only wants to stop training could lose its citations and clicks in Gemini answers too, and Search Engine Journal notes that Google's documentation describes no
way to separate the two.
Interestingly, Microsoft goes the other way. It adopts the shared line for training but adds a separate, page-level AI Grounding preference, including a DisallowExtended option that keeps content available to its own products such as Copilot while excluding the grounding services it sells to other businesses.
Meta, not one of the four, has been warmer than it has been about its commitment. Its VP of communications, Andy Stone, called the common signal "an important step," but SiliconANGLE reports the company has not committed to honoring it.
The timing is awkward: Meta has pushed back to November 20 revised commercial terms that would let it crawl websites linked to its business accounts, after a backlash forced it to strip training from the listed uses.
With training removed but not expressly banned, Sam Quigley of the News/Media Alliance told the Mercury News last month, "It's still potentially on…
Originally published by Google News AI
