How to fix Agent Permissions (robots.txt)
The most common reason AI agents can't find your business is a misconfigured robots.txt. Here's how to fix it.
Agent Permissions is frequently the lowest-scoring category, and the easiest to fix. A single line in your robots.txt can block every AI crawler from reading your site.
The most common issue
Many robots.txt files contain a wildcard disallow rule added years ago:
User-agent: * Disallow: /
This blocks everything, including AI crawlers. Even if added intentionally to block search engines (rare), it prevents AI systems from evaluating your business.
The fix
Add explicit allow rules for AI crawlers. Here's the recommended configuration:
User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot-Extended Allow: /
Add these to your existing robots.txt. If you need to block certain pages, use Disallow directives under each agent, not a blanket wildcard.
Training vs evaluation crawlers
OpenAI's GPTBot is used for model training; ChatGPT-User is used by live ChatGPT browsing. You can block one without blocking the other. If you're concerned about content being used for training but still want to be recommended by ChatGPT, block GPTBot and allow ChatGPT-User.
After the fix
Deploy the updated robots.txt and wait for crawlers to revisit. Re-run the Faro scan to confirm the permissions check passes. Score improvements in this category are immediate once the file is live.