Cloudflare's September Crawler Block and the Citation Fingerprint Gap
Two forces are reshaping AI visibility at once: Cloudflare's three-way crawler split will block Training and Agent bots by default on ad-monetized pages from September 15, and new citation research across 25,337 data points shows that comparison pages and listicles, not homepages, are what AI engines actually pull from.
This Week's Signals
Cloudflare Splits AI Crawlers Into Three Classes, Blocks Training and Agent Bots by Default from September 15
Why it matters for your score
Cloudflare protects more than 20% of the global web, and its new three-way classification, Search, Agent, and Training, means a single block setting can now accidentally silence multi-purpose crawlers like Googlebot and BingBot, which Cloudflare evaluates under the most restrictive applicable rule. From September 15, Training and Agent crawlers are blocked by default on ad-monetized pages for all new domains and existing free-tier customers who do not opt out. Any business that wants AI agents to discover and act on its pages needs to review its Cloudflare settings before that date or those crawlers will be silently turned away.
What to do this week
- 1
Audit your Cloudflare crawler settings this week and confirm whether Agent and Training crawlers are permitted on your key landing pages; Cloudflare's September 15 default block applies to ad-monetized pages on free-tier accounts, and multi-purpose crawlers like Googlebot fall under the most restrictive rule you set.
- 2
Check whether your site has a comparison page or listicle for each core service category, industry data shows comparison pages earned 1.87 citations per retrieval, the highest of any page type, and listicles led citations in B2B technology services at 61%.
- 3
Generate or validate your llms.txt file using Faro's /tools/llms-txt tool and confirm it includes a correct H1, a summary section, and organized content sections, industry auditing found only 23.1% of published SaaS files actually met spec, meaning most existing files are adding noise rather than signal.