# 2026-08-05: /registry, /ai-signals, /badge, /score-index blocked for
# traditional search crawlers (Googlebot, bingbot, catch-all) only. GSC showed
# 128 URLs stuck in "Discovered — currently not indexed" (never crawled);
# these four sections accounted for 91 of them (68 score-index alone) and were
# eating crawl budget that blog/tools/compare/help need. Ric's call: index and
# rank blogs, tools, and main pages — everything else waits. AI crawlers
# (GPTBot, ClaudeBot, PerplexityBot, anthropic-ai, cohere-ai,
# meta-externalagent, GoogleOther) are left fully open — registry and AI
# Signals exist specifically for them to read.

# 2026-08-20: /dashboard, /admin, /api added for search crawlers — app and
# admin surfaces, never meant to rank (dashboard/sign-in already carry noindex).
# 2026-08-28: the catch-all block was hiding Faro's machine-readable
# surfaces from every agent that is not one of the named AI crawlers below.
# Generic agent fetchers, MCP clients and registry crawlers identify as
# something else entirely, and Disallow: /api/ was blocking the public
# Agent Registry API from all of them. These four Allow lines are longer
# path matches and therefore win over the Disallow lines under them.
User-agent: *
Allow: /
Allow: /mcp
Allow: /api/v1/registry
Allow: /.well-known/
Allow: /registry/
# 2026-09-15: the Faro Registry (Score Index merged into /registry) is built
# for agents as much as people, so /registry, /registry.md and every filtered
# page are open here. Agents fetching on a user's behalf (ChatGPT-User,
# Claude-User, Perplexity-User, MCP clients) fall into this catch-all group.
# Googlebot and bingbot have their own groups below and stay blocked from
# /registry, so Ric's 2026-08-05 crawl-budget decision is unchanged.
# /score-index now 308s to /registry; its Disallow line is left as-is.
Disallow: /ai-signals
Disallow: /badge
Disallow: /score-index
Disallow: /dashboard
Disallow: /admin
Disallow: /api/

# Allow all major AI crawlers explicitly — full access, including the pages
# blocked above for search engines.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GoogleOther
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: meta-externalagent
Allow: /

# Search engines — blocked from agent-facing / thin-programmatic sections.
User-agent: Googlebot
Allow: /
Disallow: /registry
Disallow: /ai-signals
Disallow: /badge
Disallow: /score-index
Disallow: /dashboard
Disallow: /admin
Disallow: /api/

User-agent: bingbot
Allow: /
Disallow: /registry
Disallow: /ai-signals
Disallow: /badge
Disallow: /score-index
Disallow: /dashboard
Disallow: /admin
Disallow: /api/

# Sitemap
Sitemap: https://byfaro.ai/sitemap.xml