Thespruceeats

ClaudeBot (Anthropic) is blocked

ClaudeBot appears blocked. Claude won't have fresh data about your products.

Fix: Add both Anthropic crawlers to your robots.txt: User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: /

PerplexityBot is blocked

PerplexityBot appears blocked. You won't show up when people use Perplexity to research your category.

Fix: Add to your robots.txt: User-agent: PerplexityBot Allow: /

Google-Extended (Gemini) is blocked

Google-Extended appears blocked. Your content won't appear in Google's Gemini responses or AI Overviews , a large and growing traffic source.

Fix: Add to your robots.txt: User-agent: Google-Extended Allow: / Google-Extended is the crawl identity used for Gemini, AI Overviews, and other Google AI products. Blocking it removes you from Google's AI answers.

No Content-Signal header

No Content-Signal header. AI systems have to guess whether they're allowed to use your content for training, search, or as agent input. Declaring your policy proactively builds trust with AI platforms.

Fix: Add a Content-Signal HTTP response header to tell AI systems your content permissions: Content-Signal: ai-train=yes, search=yes, ai-input=yes In Next.js (next.config.js): headers: async () => [{ source: '/(.*)', headers: [{ key: 'Content-Signal', value: 'ai-train=yes, search=yes, ai-input=yes' }] }] This header is part of the emerging Content Signals standard (contentsignals.org) adopted by Cloudflare and others. It declares explicitly what AI can do with your content.

No Markdown content negotiation

Server doesn't return Markdown when requested with Accept: text/markdown. AI agents consume HTML which wastes context and introduces parsing errors. Markdown responses are the emerging standard for agent-optimised content delivery.

Fix: Enable Markdown content negotiation so AI agents can request clean, token-efficient versions of your pages. Quickest path: enable Cloudflare's 'Markdown for Agents' feature (Pro plan+) which auto-converts HTML to Markdown at the CDN edge when an AI agent requests it. No origin changes required. Alternatively, in Next.js you can detect the Accept header and serve .md files: if (req.headers.accept?.includes('text/markdown')) { res.setHeader('Content-Type', 'text/markdown'); } This is one of the highest-leverage changes you can make for AI agent accessibility.