OpenAI, Google, Microsoft, and Anthropic back a shared AI scraping opt out

In short: OpenAI, Google, Microsoft, and Anthropic say they will support a new, shared way for websites to opt out of AI scraping.

What happened

OpenAI, Google, Microsoft, and Anthropic say they will adopt a proposed standard that helps publishers tell AI companies not to scrape their websites. Scraping is when automated programs copy content from a site, like a fast, nonstop reader that saves what it sees.

The proposal is called a shared “policy token” signal. It is being discussed through the Internet Engineering Task Force, or IETF, which is a group that helps set common rules for how the internet works.

The signal would live inside a site’s robots.txt file. This is a small public text file that websites have used for years to give instructions to crawlers, which are the bots that visit pages for things like search results, archiving, or AI training. Today, publishers often have to set separate instructions for each AI company, and the goal here is to let them set one preference that multiple companies follow.

The companies say their AI crawlers will restrict access to a site if they encounter the shared token. Microsoft also says it will offer more detailed settings, including a choice to opt out of AI training and AI grounding. Grounding is when an AI system is pointed to specific trusted sources, like adding footnotes so it can check facts and reduce made up answers.

Why it matters

For publishers, this could make it simpler to control how their articles, images, and other content are used by AI systems. For regular readers, it could affect what information AI tools can learn from and reference, and how much traffic and revenue websites keep when people look for information online.

Source: The Verge AI