How to use the robots.txt Analyzer
Check which AI crawlers your site is blocking or allowing, and get the exact fix for your robots.txt file.
The robots.txt Analyzer fetches your site's robots.txt file and checks which AI crawlers are explicitly allowed, explicitly blocked, or not addressed. It covers GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Googlebot-Extended, and others.
Running the analyzer
Enter your domain and click Analyze. The tool fetches yourdomain.com/robots.txt and parses the User-agent directives. Results show each crawler and its status: Allowed, Blocked, or No rule (defaults to allowed under the robots exclusion standard, but explicit rules are better practice).
Reading the results
Each crawler gets a status badge. A blocked crawler means AI agents from that provider cannot index your site, they will not be able to recommend you. A "No rule" crawler is technically allowed but would benefit from an explicit Allow directive to signal intent.
Applying the fix
The tool generates the correct robots.txt additions for your situation. Copy them and add to your existing robots.txt file. If you don't have a robots.txt, the tool generates a complete starting file. The fix is typically a few lines and takes minutes to deploy.
Note: robots.txt changes take effect on the next crawl, which may be within hours or up to a few days depending on the crawler's schedule.