Morning Edition · Sunday, July 12, 2026Published at 1:29 AM EDT · New York
Cloudflare Splits AI Crawler Controls Into Search, Agent, and Training
New domains will block training and agent crawlers by default on ad-supported pages starting September 15, while search indexing stays allowed, and the change reaches free-tier customers.
Cloudflare replaced its blanket AI bot block with three separately managed categories, giving site owners distinct switches for Search, Agent, and Training crawlers. Search covers indexing that returns referral traffic, Agent covers real-ti…
Continue the AI Intelligence Brief
Track frontier labs, chips, export controls, model releases, regulation, and AI infrastructure.
- 5 AI intelligence signals a day
- Frontier labs, compute, and chips
- Model releases and AI infrastructure
- Source-grounded analysis with confidence labels
The Global Intelligence Brief stays free.
Part of a tracked trend
Metering AI Access to Web Content
Infrastructure providers and publishers increasingly gate and price crawler access by purpose, forcing AI training and agent traffic into a licensed market and raising the cost of the open-web data that models depend on.
More from this edition
- Apple Sues OpenAI, Alleging Former Engineers Stole Hardware Secrets for Its ChatGPT Devices
- Meta Starts Charging for Its Own Model With Muse Spark 1.1 and a New API
- Anthropic Returns Claude Fable 5 to Service and Proposes a Shared Jailbreak-Severity Scale
- Meta's Brain2Qwerty v2 Decodes Typed Sentences From Brain Scans at 61% Word Accuracy
- Meta Ships Muse Image and Previews Muse Video, Its First In-House Visual Models
- Musk and Altman Trade Barbs on X as Both Ship New Flagship Models
- SAM-MT Targets Real-Time Segmentation of Many Objects at Once in Video
- Anthropic Publishes the Origin Story of Claude Code, Its Coding Agent
- Anthropic Invites the Public's Hardest Questions About AI
- Anthropic Expands Its Economic Research on How AI Reshapes Work
- Google Consolidates Its Geospatial Models Under Earth AI