{"version":1,"type":"story","url":"https://digestai.news/story/openai-will-honor-a-shared-robots-txt-signal-for-ai-training-preferenc","json":"https://digestai.news/story/openai-will-honor-a-shared-robots-txt-signal-for-ai-training-preferenc.json","markdown":"https://digestai.news/story/openai-will-honor-a-shared-robots-txt-signal-for-ai-training-preferenc.md","slug":"openai-will-honor-a-shared-robots-txt-signal-for-ai-training-preferenc","headline":"OpenAI, Google, Microsoft and Anthropic adopt new AI scraping opt-out standard","summary":"OpenAI, Google, Microsoft and Anthropic will adopt a shared standard that lets publishers block AI crawlers from scraping their sites. The **policy token** signal, proposed by Anthropic’s Kevin Kelley, will live in a website’s **robots.txt** file. Currently, publishers must set preferences separately for each AI company, but this new method applies across all four firms.\n\nThe move follows growing concerns over unauthorized scraping, which fuels AI training. While Anthropic already supports the token, Google and Microsoft plan to implement it **next year**, and OpenAI says it will arrive **in the coming months**. Microsoft will also offer extra controls, letting publishers opt out of **AI training** or **AI grounding**—the process of verifying sources to reduce AI hallucinations. Websites will still appear in Bing search results regardless of these choices.","keyPoints":["Four major AI labs—OpenAI, Google, Microsoft and Anthropic—will adopt a shared scraping opt-out token in robots.txt files","Anthropic’s Kevin Kelley proposed the standard via the Internet Engineering Task Force (IETF) to streamline publisher controls","Microsoft adds granular options: block AI training or grounding, but search results remain unaffected"],"whyItMatters":"Publishers gain a unified way to block AI scrapers without negotiating with each company individually, reducing content theft risks.","category":{"slug":"policy","name":"Policy & Regulation","url":"https://digestai.news/category/policy"},"entities":{"companies":["OpenAI","Google","Microsoft","Anthropic"],"models":[],"people":["Kevin Kelley"]},"firstPublishedAt":"2026-10-30T09:30:00Z","updatedAt":"2026-10-30T18:27:04Z","sourceCount":3,"hasPrimarySource":true,"sources":[{"outlet":"theverge.com","title":"The biggest names in AI are backing a new way to opt out of scraping","url":"https://theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out","publishedAt":"2026-10-30T13:26:00Z","type":"press","primary":false,"lead":true},{"outlet":"Unite.AI","title":"OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences","url":"https://unite.ai/openai-to-honor-shared-robots-txt-signal-for-ai-training-preferences","publishedAt":"2026-10-30T18:27:04Z","type":"press","primary":false,"lead":false},{"outlet":"OpenAI","title":"Making it easier for websites to express AI training preferences","url":"https://openai.com/index/helping-websites-express-ai-training-preferences","publishedAt":"2026-10-30T09:30:00Z","type":"primary","primary":true,"lead":false}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"OpenAI, Google, Microsoft and Anthropic adopt new AI scraping opt-out standard\", 30 October 2026, https://digestai.news/story/openai-will-honor-a-shared-robots-txt-signal-for-ai-training-preferenc","publisher":"Digest AI","title":"OpenAI, Google, Microsoft and Anthropic adopt new AI scraping opt-out standard","datePublished":"2026-10-30T09:30:00Z","url":"https://digestai.news/story/openai-will-honor-a-shared-robots-txt-signal-for-ai-training-preferenc"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}