Cloudflare and OpenAI pilot indexes hundreds of millions of new pages for ChatGPT
Cloudflare and OpenAI say a research pilot that used about 1% of Cloudflare’s network helped OpenAI discover and index hundreds of millions of new web pages. The companies shared early results on Oct. 15 in a joint blog post, noting examples such as a washing‑machine user guide, up‑to‑date passport fee information and a restaurant’s menu. The pilot leverages Cloudflare’s network signals—page…
What you can do with it
AI at Work →Prevent training use with AI Crawl Control
What you get You keep your pages searchable in ChatGPT but stop OpenAI from using the content for training.
Choose which AI bots can crawl your site and see results in ChatGPT.
- Who for
- marketers
- Cost
- free
- Effort
- minutes
Use it for
- Allow search visibility
- Prevent training use
- Monitor AI crawler activity
Watch out Only works if site respects robots.txt
Key points from the news
- Pilot used roughly 1% of Cloudflare’s network and indexed hundreds of millions of new pages
- Signals include page recency, traffic quality and recent updates to guide OpenAI’s search crawler
- Webmasters can allow OAI-SearchBot and block GPTBot via Cloudflare’s AI Crawl Control
The post stresses that a signal does not grant permission to access a page and that OpenAI’s crawlers respect robots.txt directives. Site owners can use Cloudflare’s AI Crawl Control, available on every plan, to allow the OAI‑SearchBot (used for ChatGPT search) while blocking GPTBot (used for training). The companies did not disclose how often the newly indexed pages appear in ChatGPT results or the traffic they generate, and they gave no specific timeline for broader rollout beyond the remainder of 2026.
The story so far
8 episodes →- Cloudflare and OpenAI pilot indexes hundreds of millions of new pages for ChatGPTthis story
ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot via @sejournal, @MattGSouthern
Search Engine Journal · 20 October 2026
Loading the full article…
This text was published by Search Engine Journal and written by Matt G. Southern. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- AWS launches generally available Agent Registry for runtime tool discovery · 4 src
- AWS and Keyfactor detail X.509 PKI architecture for securing AI agents · 1 src
- Eon uses Amazon Bedrock AgentCore gateway to unify multi-cloud investigation tools · 1 src
- Amazon releases TorchNeuron to run PyTorch on Trainium chips, challenging Nvidia's CUDA advantage · 5 src
- Arkose Labs releases Q3 2026 agentic AI trust report · 1 src
Comments
via GitHub Discussions