{"version":1,"type":"story","url":"https://digestai.news/story/google-s-illyes-says-robots-txt-governs-crawling-not-ai-training","json":"https://digestai.news/story/google-s-illyes-says-robots-txt-governs-crawling-not-ai-training.json","markdown":"https://digestai.news/story/google-s-illyes-says-robots-txt-governs-crawling-not-ai-training.md","slug":"google-s-illyes-says-robots-txt-governs-crawling-not-ai-training","headline":"Google's Illyes says robots.txt governs crawling, not AI training","summary":"Google Search relations staff Gary Illyes explained that the robots.txt standard (RFC 9309) only tells crawlers which URLs they may request; it does not prescribe what a crawler does with the data after fetching it. To keep a particular AI crawler out, a site must list that crawler’s user‑agent token explicitly in the file. Google‑Extended, a token used by Google’s own crawlers, can limit whether fetched content is used to train future Gemini models or for grounding in Gemini Apps, but it does not affect search ranking.\n\nIllyes co‑authored an IETF draft dated Aug. 19 that would add a “Content‑Usage” rule (e.g., train‑ai=n) to robots.txt and a matching HTTP header, allowing sites to express preferences for how crawled content may be used. The draft is still a Working Group Document, has not been submitted to the IETF Steering Group, and the related “train‑ai” label appears in a separate Vocabulary Draft dated Sept. 14 that “reflects no consensus”. Cherry Prommawin advised using percent‑encoded paths for non‑English characters or emojis, as raw characters may confuse older crawlers.","keyPoints":["robots.txt only controls which URLs may be fetched; it does not dictate post‑crawl data usage.","Blocking a specific AI crawler requires listing its user‑agent token; Google‑Extended controls Gemini training but not ranking.","An IETF draft proposes a “Content‑Usage” rule (e.g., train‑ai=n), but it is still pending and lacks consensus."],"whyItMatters":"Webmasters cannot rely on robots.txt to prevent their content from being used in generative‑AI training, limiting control over data and influencing SEO and content‑ownership strategies.","category":{"slug":"policy","name":"Policy & Regulation","url":"https://digestai.news/category/policy"},"entities":{"companies":["Google","Mozilla"],"models":["Gemini"],"people":["Gary Illyes","Cherry Prommawin","Martin Thomson"]},"firstPublishedAt":"2026-10-22T10:58:04Z","updatedAt":"2026-10-22T10:58:04Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"Search Engine Journal","title":"Google’s Illyes: Robots.txt Covers Crawling, Not AI Training via @sejournal, @MattGSouthern","url":"https://searchenginejournal.com/google-illyes-robots-txt-crawling-ai-training/593957","publishedAt":"2026-10-22T10:58:04Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":{"title":"AI Optimization Race Hits Global Stage","url":"https://digestai.news/thread/semrush-adds-ai-citation-tracking-to-prompt-tracking-tool","storyCount":5},"cite":{"text":"Digest AI, \"Google's Illyes says robots.txt governs crawling, not AI training\", 22 October 2026, https://digestai.news/story/google-s-illyes-says-robots-txt-governs-crawling-not-ai-training","publisher":"Digest AI","title":"Google's Illyes says robots.txt governs crawling, not AI training","datePublished":"2026-10-22T10:58:04Z","url":"https://digestai.news/story/google-s-illyes-says-robots-txt-governs-crawling-not-ai-training"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}