Is your robots.txt blocking AI?

robots.txt is a plain text file that sits at the root of your website. For example, yourdomain.com/robots.txt, and it tells web crawlers which parts of your site they can and cannot access. Search engines have respected this file for decades. Now AI companies do too.

When ChatGPT, Claude, or Perplexity want to understand your business or recommend your products, their crawlers visit your site first. If your robots.txt blocks them, they either use old cached data or skip you entirely. AI is often now the first touchpoint between a customer and a business. Being blocked is invisible to you, and catastrophic for your discovery.

The most common problem is a wildcard rule: User-agent: * / Disallow: /. This single line blocks every crawler, including every AI agent. It was often added years ago to block scrapers, without anyone realising it would later block the AI crawlers businesses now depend on for organic discovery.

Which AI crawlers should you allow?

The major AI companies all publish the user-agent names their crawlers use. The most important ones to allow are GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google Gemini and AI Overviews), and anthropic-ai. Together these cover the vast majority of AI-driven search and recommendation traffic.

You can allow specific crawlers without opening your site to all bots. Adding an explicit User-agent: GPTBot / Allow: / block before your wildcard disallow rule gives GPTBot full access while leaving your other restrictions intact.

robots.txt vs llms.txt: what's the difference?

robots.txt controls access: it tells crawlers whether they're allowed in. llms.txt controls understanding: it tells AI what your business does, what you sell, and how to recommend you. Both are needed. robots.txt gets you indexed; llms.txt gets you recommended accurately. The Faro free scan checks both.

How to fix a blocked robots.txt

If this tool found blocked AI crawlers on your site, the fix is straightforward. Add explicit allow rules for each AI crawler before any restrictive wildcard rules in your robots.txt. The copy-paste snippet above includes all the major AI crawlers. Deploy the change, then re-run this tool to confirm access is restored.

After fixing your robots.txt, run the Faro AI Readiness Scan to see how your overall AI discoverability score improves. robots.txt is one of six categories Faro audits. Fixing it alone can meaningfully move your score.

Does blocking AI crawlers protect your content?

This is the most common concern when businesses discover they're blocking AI crawlers. The answer depends on your goals. If you are a news publisher or content creator who doesn't want your work used to train AI models, blocking specific crawlers makes sense, and each AI company publishes the relevant user-agent name to block. However, for the vast majority of businesses, blocking AI crawlers has no protective benefit and significant discovery cost.

Blocking GPTBot doesn't prevent ChatGPT from knowing about you. It already has cached knowledge from previous crawls and third-party mentions. What it does prevent is ChatGPT from getting your current, accurate information when users ask about you. Blocking ClaudeBot doesn't stop Anthropic from training on your data if they've already crawled it. It just stops Claude from having an up-to-date understanding of what you offer. In both cases, you lose the recommendation without gaining the protection you hoped for.

Robots.txt as part of a complete AI readiness strategy

Allowing AI crawlers in your robots.txt is the first step, but it's not sufficient on its own. Once crawlers have access, what they find determines the quality of AI recommendations about your business. Pair your robots.txt fix with an llms.txt file that gives AI agents a structured summary of your business, JSON-LD structured data that tells search engines and AI models exactly what type of organization you are, and an OKF knowledge bundle that satisfies the emerging agent knowledge standard. Together, these four changes form the foundation of an AI-ready website and typically move the Faro AI Readiness Score by 20–35 points.