Blocking the Wrong Crawlers Costs You AI Visibility; IAB Formalizes Citation Tiers
New IAB measurement standards split AI citation tracking into 'directional' and 'decision-grade' data while Anthropic's three-crawler architecture means most robots.txt files are configured for the wrong threat. Both stories demand an immediate audit of your site's crawler permissions.
This Week's Signals
Anthropic's Three-Crawler Split Means Most Sites Are Blocking the Wrong Bots
Why it matters for your score
Anthropic now documents three separate user agents: ClaudeBot for training, Claude-SearchBot for search-result quality, and Claude-User for on-demand retrieval. Blocking retrieval-layer bots like Claude-SearchBot and OAI-SearchBot makes your site invisible to AI answers, while blocking training bots like ClaudeBot carries far less visibility cost. Most businesses set blanket blocks without distinguishing between these functions, quietly erasing themselves from AI responses.
IAB August 2026 Framework: Only 11% of Domains Get Cited by Both ChatGPT and Perplexity
Why it matters for your score
The IAB's new measurement framework distinguishes 'directional' from 'decision-grade' citation tracking, giving businesses a clearer standard for whether their data is reliable enough to act on. The Attrifast cohort found a median blended AI citation rate of 4.1%, but enterprise brands are cited 8x more often than solo or small-site operators — company size outranks domain authority, content volume, and backlinks as a predictor. With only 11% of domains cited by both ChatGPT and Perplexity, treating those platforms as one channel is a measurement error that distorts strategy.
MCP July 2026 Spec Goes Stateless With Close to Half a Billion Monthly SDK Downloads
Why it matters for your score
The largest MCP revision since launch removes session pinning at the transport layer and deprecates legacy HTTP+SSE transport with a 12-month offramp, meaning any tool or server built on session-based connections needs a migration plan now. TypeScript and Python SDKs each crossed 1 billion total downloads, and Tier 1 SDKs collectively receive close to half a billion downloads per month — the protocol is no longer experimental infrastructure. Businesses building or buying AI-connected tools should confirm their vendors are shipping against the 2026-07-28 spec, not the deprecated session model.
What to do this week
- 1
Audit your robots.txt against the full list of eight active AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended) and confirm you are not blocking retrieval-layer bots like Claude-SearchBot or OAI-SearchBot — use Faro's robots.txt tool at /tools/robots-txt to check your current configuration.
- 2
Run your site through Faro's AI readiness scan at /tools/ai-readiness-scan to establish a baseline citation rate you can classify against the IAB's new directional vs. decision-grade standard — the median blended rate in the Attrifast cohort was 4.1%, so compare your result against that benchmark before drawing conclusions.
- 3
If your business connects to any MCP-based tools or agent workflows, ask your vendor or developer to confirm their implementation targets the 2026-07-28 stateless spec and has a migration plan away from the deprecated HTTP+SSE transport before the 12-month offramp closes.