{"version":1,"type":"story","url":"https://digestai.news/story/cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages","json":"https://digestai.news/story/cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages.json","markdown":"https://digestai.news/story/cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages.md","slug":"cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages","headline":"Cloudflare and OpenAI pilot indexes hundreds of millions of new pages for ChatGPT","summary":"Cloudflare and OpenAI say a research pilot that used about 1% of Cloudflare’s network helped OpenAI discover and index hundreds of millions of new web pages. The companies shared early results on Oct. 15 in a joint blog post, noting examples such as a washing‑machine user guide, up‑to‑date passport fee information and a restaurant’s menu. The pilot leverages Cloudflare’s network signals—page recency, traffic quality and recent updates—to guide OpenAI’s search crawler toward valuable pages that few other sites link to, while also flagging pages that should be excluded, such as a fake login form planted on a legitimate site.\n\nThe post stresses that a signal does not grant permission to access a page and that OpenAI’s crawlers respect robots.txt directives. Site owners can use Cloudflare’s AI Crawl Control, available on every plan, to allow the OAI‑SearchBot (used for ChatGPT search) while blocking GPTBot (used for training). The companies did not disclose how often the newly indexed pages appear in ChatGPT results or the traffic they generate, and they gave no specific timeline for broader rollout beyond the remainder of 2026.","keyPoints":["Pilot used roughly 1% of Cloudflare’s network and indexed hundreds of millions of new pages","Signals include page recency, traffic quality and recent updates to guide OpenAI’s search crawler","Webmasters can allow OAI-SearchBot and block GPTBot via Cloudflare’s AI Crawl Control"],"whyItMatters":"The pilot expands the pool of content ChatGPT can draw from, giving site owners a way to appear in search results without training their data.","category":{"slug":"agents","name":"Agents & Tools","url":"https://digestai.news/category/agents"},"entities":{"companies":["Cloudflare","OpenAI"],"models":["ChatGPT"],"people":[]},"firstPublishedAt":"2026-10-20T15:00:36Z","updatedAt":"2026-10-20T15:00:36Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"Search Engine Journal","title":"ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot via @sejournal, @MattGSouthern","url":"https://searchenginejournal.com/chatgpt-search-can-draw-on-more-pages-after-a-cloudflare-pilot/593720","publishedAt":"2026-10-20T15:00:36Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":{"title":"AI Giants Adopt Invisible Text Watermarks","url":"https://digestai.news/thread/anthropic-s-invisible-text-watermarking-in-claude-ai","storyCount":8},"cite":{"text":"Digest AI, \"Cloudflare and OpenAI pilot indexes hundreds of millions of new pages for ChatGPT\", 20 October 2026, https://digestai.news/story/cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages","publisher":"Digest AI","title":"Cloudflare and OpenAI pilot indexes hundreds of millions of new pages for ChatGPT","datePublished":"2026-10-20T15:00:36Z","url":"https://digestai.news/story/cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}