# Cloudflare and OpenAI pilot indexes hundreds of millions of new pages for ChatGPT

Digest AI · Agents & Tools · published 2026-10-20T15:00:36Z

Canonical: https://digestai.news/story/cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages

## Summary

Cloudflare and OpenAI say a research pilot that used about 1% of Cloudflare’s network helped OpenAI discover and index hundreds of millions of new web pages. The companies shared early results on Oct. 15 in a joint blog post, noting examples such as a washing‑machine user guide, up‑to‑date passport fee information and a restaurant’s menu. The pilot leverages Cloudflare’s network signals—page recency, traffic quality and recent updates—to guide OpenAI’s search crawler toward valuable pages that few other sites link to, while also flagging pages that should be excluded, such as a fake login form planted on a legitimate site.

The post stresses that a signal does not grant permission to access a page and that OpenAI’s crawlers respect robots.txt directives. Site owners can use Cloudflare’s AI Crawl Control, available on every plan, to allow the OAI‑SearchBot (used for ChatGPT search) while blocking GPTBot (used for training). The companies did not disclose how often the newly indexed pages appear in ChatGPT results or the traffic they generate, and they gave no specific timeline for broader rollout beyond the remainder of 2026.

## Key points

- Pilot used roughly 1% of Cloudflare’s network and indexed hundreds of millions of new pages
- Signals include page recency, traffic quality and recent updates to guide OpenAI’s search crawler
- Webmasters can allow OAI-SearchBot and block GPTBot via Cloudflare’s AI Crawl Control

## Why it matters

The pilot expands the pool of content ChatGPT can draw from, giving site owners a way to appear in search results without training their data.

## Sources

1. [ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot via @sejournal, @MattGSouthern](https://searchenginejournal.com/chatgpt-search-can-draw-on-more-pages-after-a-cloudflare-pilot/593720) (Search Engine Journal, 2026-10-20)

Part of the developing story: [AI Giants Adopt Invisible Text Watermarks](https://digestai.news/thread/anthropic-s-invisible-text-watermarking-in-claude-ai) (8 stories)

## Cite

Digest AI, "Cloudflare and OpenAI pilot indexes hundreds of millions of new pages for ChatGPT", 20 October 2026, https://digestai.news/story/cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/cloudflare-and-openai-pilot-indexes-hundreds-of-millions-of-new-pages.json