Google's Illyes says robots.txt governs crawling, not AI training
Google Search relations staff Gary Illyes explained that the robots.txt standard (RFC 9309) only tells crawlers which URLs they may request; it does not prescribe what a crawler does with the data after fetching it. To keep a particular AI crawler out, a site must list that crawler’s user‑agent token explicitly in the file. Google‑Extended, a token used by Google’s own crawlers, can limit…
Key points
- robots.txt only controls which URLs may be fetched; it does not dictate post‑crawl data usage.
- Blocking a specific AI crawler requires listing its user‑agent token; Google‑Extended controls Gemini training but not ranking.
- An IETF draft proposes a “Content‑Usage” rule (e.g., train‑ai=n), but it is still pending and lacks consensus.
Illyes co‑authored an IETF draft dated Aug. 19 that would add a “Content‑Usage” rule (e.g., train‑ai=n) to robots.txt and a matching HTTP header, allowing sites to express preferences for how crawled content may be used. The draft is still a Working Group Document, has not been submitted to the IETF Steering Group, and the related “train‑ai” label appears in a separate Vocabulary Draft dated Sept. 14 that “reflects no consensus”. Cherry Prommawin advised using percent‑encoded paths for non‑English characters or emojis, as raw characters may confuse older crawlers.
The story so far
4 episodes →- Google's Illyes says robots.txt governs crawling, not AI trainingthis story
Google’s Illyes: Robots.txt Covers Crawling, Not AI Training via @sejournal, @MattGSouthern
Search Engine Journal · 22 October 2026
Loading the full article…
This text was published by Search Engine Journal and written by Matt G. Southern. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Policy & Regulation
All →- Nvidia CEO Jensen Huang calls AI doomsday warnings cheap · 2 src
- Nvidia CEO Jensen Huang pledges $2 billion for Europe’s sovereign AI push · 6 src
- Google faces staff resignations over AI military contracts · 4 src
- Meta removes 'training' from crawl clause in commercial terms effective Oct 30 · 2 src
- Palantir says Maven platform used by NATO has no kill switch · 1 src
Comments
via GitHub Discussions