# Cloudflare adds Bot Preference Sync to generate robots.txt from dashboard

Digest AI · Marketing & Small Business · published 2026-09-18T14:30:00Z

Canonical: https://digestai.news/story/cloudflare-adds-bot-preference-sync-to-generate-robots-txt-from-dashbo

## Summary

Cloudflare announced Bot Preference Sync, a feature that automatically writes robots.txt entries based on the AI bot policy settings a site configures in Cloudflare’s dashboard. The tool creates a block delimited by “# BEGIN Cloudflare Bot Preference Sync” and “# END Cloudflare Bot Preference Sync”, preserving any existing content. Cloudflare says the service runs from the free tier upward and will be enabled by default for new customers on September 15, with Training and Agent categories blocked on pages that display ads unless the owner opts out.

The author, who had been manually fixing his own robots.txt after publishing a critique on August 20, notes that the sync cannot exclude individual crawlers and that the three policy categories (Search, Agent, Training) may not capture more granular business decisions. Cloudflare’s documentation lists four disclosure conditions that determine whether a crawler is treated as opaque and blocked when Training is set to disallow, but no specific companies are named. The piece argues that default‑on behavior hands a vendor the authority to state a site’s AI‑training stance, which may not reflect the owner’s intent.

## Key points

- Bot Preference Sync generates robots.txt from three Cloudflare AI bot policy settings and adds identifiable markers.
- Feature is on by default for new customers on September 15, blocking Training and Agent on ad pages unless opted out.
- Individual crawlers cannot be excluded; finer control requires turning the sync off and editing robots.txt manually.

## Why it matters

Automatic robots.txt generation can simplify compliance for many sites, but default settings may impose unwanted AI‑training restrictions without owners’ explicit consent.

## Sources

1. [Cloudflare Will Write Your Robots.txt, And It Has A Point via @sejournal, @slobodanmanic](https://searchenginejournal.com/cloudflare-will-write-your-robots-txt-and-it-has-a-point/589262) (Search Engine Journal, 2026-09-18)

Part of the developing story: [Cloudflare Automates AI Bot Control](https://digestai.news/thread/cloudflare-adds-disallow-ai-training-to-let-sites-block-ai-training-keep-search) (2 stories)

## Cite

Digest AI, "Cloudflare adds Bot Preference Sync to generate robots.txt from dashboard", 18 September 2026, https://digestai.news/story/cloudflare-adds-bot-preference-sync-to-generate-robots-txt-from-dashbo

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/cloudflare-adds-bot-preference-sync-to-generate-robots-txt-from-dashbo.json