OpenAI will honor a shared robots.txt signal for AI training preferences
OpenAI announced it will honor a common signal in robots.txt, the file that lets websites express preferences about using their content for foundational generative AI training. The voluntary signal is meant to simplify how publishers communicate the same preference to multiple participating providers, replacing the need to maintain separate rules for each crawler.
Key points
- OpenAI will honor a shared robots.txt signal for AI training preferences
- The signal aims to reduce separate rules for each provider’s crawler
- OpenAI plans to implement the signal in the coming months and urges other AI providers to adopt it
OpenAI says it will implement the signal in the coming months and will share another update with implementation details. The new signal will not replace existing crawler controls in robots.txt, which will continue to be respected. The effort draws on work by the IETF AI Preferences working group, and OpenAI encourages other AI providers to honor the signal as well.
Making it easier for websites to express AI training preferences
OpenAI · 30 October 2026
Loading the full article…
This text was published by OpenAI. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
2sources- OpenAI to Honor Shared Robots.txt Signal for AI Training PreferencesPress · Unite.AI ·
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Policy & Regulation
All →- US DOJ inquires about Nvidia's $12.93 billion Hugging Face deal over AI competition concerns · 17 src
- SEC adds superintelligence and tokenized securities to 2027 exam priorities · 1 src
- U.S. intelligence investigated Chinese spy plot targeting Nvidia CEO Jensen Huang via Megaspeed executive Alice Huang, report says · 9 src
- Wikimedia Foundation warns AI copyright ruling could threaten open web · 2 src
- Pope Leo calls for AI disarmament and ban on lethal autonomous weapons · 2 src
Comments
via GitHub Discussions