The biggest names in AI are backing a new way to opt out of scraping

OpenAI, Google, Microsoft, and Anthropic say they will honor a shared policy token in robots.txt that tells their crawlers to stay away for AI training. Today, sites have to set preferences separately for each company. This collapses that into one signal.

The proposal comes through the IETF, authored by an Anthropic standards lead. Anthropic has already adopted it; Google and Microsoft plan to support it next year; OpenAI says the coming months. Meta voiced support but did not say it would honor the signal.

We'll say it plainly: this is progress. A single, standards-track signal that the biggest labs commit to honoring is better than a growing patchwork of crawler names that every small publisher has to track. It lowers the cost of saying no, and that matters most for outlets without a technical team.

Microsoft's piece is the most interesting detail. It plans granular controls separating AI training from AI grounding, the practice of pointing a model at a source to support its answers, and says sites will stay in Bing search either way. Google already has similar separation. That distinction is where the real economics live. Training is a one-time take. Grounding is ongoing use of a publisher's work to answer questions that might otherwise have sent a reader to the site.

Now the limits. This is still opt-out. Content that was scraped before the signal existed is not coming back. Compliance is voluntary, and the timelines stretch into next year. And the most telling line in the story is Meta's: supportive words, no commitment. A standard only works if the companies that matter actually obey it, and the companies that don't sign on are exactly the ones a publisher should worry about.

It also doesn't touch payment. Efforts like the Really Simple Licensing standard and Cloudflare's default blocking exist because many publishers don't just want a door they can close; they want a price for walking through it. A clean opt-out may make it easier for AI companies to argue that anyone who didn't opt out consented.

Our read: take the win, and keep pushing. Useful AI depends on good information, and good information depends on publishers who can afford to keep producing it. A shared no is a floor, not a business model.

Robots.txt has long told crawlers what they may index or archive, but it was not designed with AI training in mind. Publishers have responded to rising AI crawler traffic with blocking, licensing schemes, and lawsuits, and this proposal is the first time the major labs have rallied around one shared signal.