Zeplin

No robots.txt found

No robots.txt detected (server returned 404). AI agents don't know what they're allowed to access , many default to not crawling.

Fix: Create a file at /robots.txt with this minimum content: User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://yourdomain.com/sitemap.xml

No Content-Signal header

No Content-Signal header. AI systems have to guess whether they're allowed to use your content for training, search, or as agent input. Declaring your policy proactively builds trust with AI platforms.

Fix: Add a Content-Signal HTTP response header to tell AI systems your content permissions: Content-Signal: ai-train=yes, search=yes, ai-input=yes In Next.js (next.config.js): headers: async () => [{ source: '/(.*)', headers: [{ key: 'Content-Signal', value: 'ai-train=yes, search=yes, ai-input=yes' }] }] This header is part of the emerging Content Signals standard (contentsignals.org) adopted by Cloudflare and others. It declares explicitly what AI can do with your content.

No Markdown content negotiation

Server doesn't return Markdown when requested with Accept: text/markdown. AI agents consume HTML which wastes context and introduces parsing errors. Markdown responses are the emerging standard for agent-optimised content delivery.

Fix: Enable Markdown content negotiation so AI agents can request clean, token-efficient versions of your pages. Quickest path: enable Cloudflare's 'Markdown for Agents' feature (Pro plan+) which auto-converts HTML to Markdown at the CDN edge when an AI agent requests it. No origin changes required. Alternatively, in Next.js you can detect the Accept header and serve .md files: if (req.headers.accept?.includes('text/markdown')) { res.setHeader('Content-Type', 'text/markdown'); } This is one of the highest-leverage changes you can make for AI agent accessibility.

No llms.txt found

No llms.txt found. This standard (like robots.txt but for LLMs) tells AI exactly what to use and how to describe you. Less than 5% of sites have it.

Fix: Create a plain text file at /llms.txt. Minimum: # [Your Company] > [One sentence: what you do and who you serve] ## What we do [2–3 sentences] ## Key pages - /pricing , Plans and pricing - /docs , API documentation Use Faro's llms.txt Generator to build a complete version automatically.

Sitemap declared in robots.txt

Sitemap not referenced in robots.txt. Agents have to guess its location.

Fix: Add one line to the bottom of your robots.txt: Sitemap: https://yourdomain.com/sitemap.xml