What Changed: Syncing Policy with Enforcement
Cloudflare announced Bot Preference Sync on August 21, 2026, a feature that automatically updates your robots.txt file to reflect the AI bot preferences you set in the Cloudflare dashboard. Available to all customers from Free to Enterprise, it addresses a common pain point: when your robots.txt says one thing but your enforcement rules do another, some crawlers may ignore your stated preferences or try to bypass your rules. By keeping the two in sync, you eliminate that inconsistency.
Previously, you had to manage a static robots.txt file separately from your edge enforcement rules. Now, Cloudflare generates or updates your robots.txt based on your configuration, so what you publish is what you enforce. The feature can be turned on or off at any time, and for new customers it’s on by default.
How It Works: Category-Wide Policies
Bot Preference Sync builds on the AI traffic controls Cloudflare launched on July 1, 2026. Those controls let you set preferences for three categories of AI traffic: Search, Agent, and Training. For Search and Agent, you have three options: Allow, Block on pages that serve ads, or Block everywhere. For Training, you can also choose Disallow, which writes a “no training” preference to your robots.txt.
When you enable sync, Cloudflare prepends the generated directives to your existing robots.txt, so any manual Disallow rules you already have are preserved. The list of bots included is based on Cloudflare’s BotBase directory and is updated periodically. You can view the verified bots classified as Search, Agent, and Training in the public bots directory.
Here’s an example of what gets prepended (shortened and anonymized):
# BEGIN Cloudflare Bot Preference Sync
User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /
...
# END Cloudflare Bot Preference SyncNote that Bot Preference Sync works at the category level, not for individual custom rules. If you have a special arrangement with a specific company and want to grant an exception, you can turn off the sync and manage your robots.txt manually.
Publisher-Friendly Defaults
Cloudflare is also introducing a new default for publishers and ad-supported sites. During onboarding, such customers can select the option “I monetize from pages with ads on this domain,” which sets Training to Disallow by default. This means your content stays out of model training while remaining accessible to search engines that send readers to your pages. You can change this setting at any time.
For non-publisher new customers, no blocks or disallows are added by default. The choice is yours: you can decide later whether to block Search, Agent, or Training, but the starting point is no blocks on your behalf.
Transparency as the Price of Admission
Cloudflare is also tightening requirements for mixed-use crawlers—bots that combine search, agent use, and training behind a single user agent. To avoid being blocked when you set “Disallow Training,” these bot operators must provide additional information:
- Respect a “no training” preference in robots.txt via any mechanism.
- Give site owners a way to opt out of AI summaries.
- Provide URL-level visibility into which pages were made available for training, plus metrics on search results.
- Show publicly that disallowing training does not hurt traditional search results.
Bots that meet these criteria are tracked in the AI bot transparency section of Cloudflare Radar, which shows both best practices honored and violations. Crawlers that don’t provide transparency won’t get the benefit of the doubt—they’re still blocked when you disallow training. This makes transparency a requirement for access.
Limitations and Trade-offs
While Bot Preference Sync simplifies policy management, it’s not for everyone. It’s designed for category-wide decisions, not case-by-case exceptions. If you have complex custom rules or special agreements with specific crawlers, you’ll want to turn off the sync and maintain your robots.txt manually. Also, the sync only affects the robots.txt file; it doesn’t change your edge enforcement rules, so you still need to configure those separately if you want to block traffic at the edge.
Another consideration: the list of bots in robots.txt is based on Cloudflare’s tracking, which may not include every crawler. You should verify that the bots you care about are covered. Finally, the feature is rolling out over the coming week, so existing customers using the legacy managed robots.txt will be prompted to review and confirm their preferences.
Takeaway: Align Your Stated Preferences with Reality
Bot Preference Sync is a practical step toward reducing the gap between what you declare and what you enforce. For most site owners, it’s a low-effort way to ensure your AI bot policies are consistent. But if you need fine-grained control or have special arrangements, you have the option to opt out and manage your own robots.txt. The key is to make a conscious choice: either let Cloudflare sync your preferences, or take manual control—but don’t leave your settings inconsistent, because that’s exactly what some crawlers will exploit.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.