When a human visits your website, they navigate using visual hierarchy, menus, and intuition. When an AI agent visits your site to evaluate or summarize it, it doesn't use any of those. It crawls raw content, makes decisions about what to prioritize, and forms a picture of your business based on what it finds first. The problem is that without guidance, agents often spend their evaluation time on login pages, legal disclaimers, and navigation boilerplate rather than your actual product or value proposition.
The llms.txt specification was created to solve this. It gives website owners a simple, standardized way to tell AI agents exactly what to read and in what order.
What is llms.txt?
llms.txt is a plain text file, written in markdown, placed at the root of your domain at https://yourdomain.com/llms.txt. It functions similarly to robots.txt, but instead of restricting access, it provides guidance. It tells AI agents which pages contain your core content, which are supplementary, and which they can skip.
The specification was proposed by Answer.AI and Jeremy Howard as a response to a practical problem: AI models and agents frequently waste context windows on irrelevant pages when evaluating sites. A well-structured llms.txt file helps agents spend their limited context budget on the pages that actually explain what you do.
Anthropic, the company behind Claude, added support for llms.txt in its own documentation. Cloudflare, Perplexity, and a growing number of AI infrastructure providers have referenced the specification as the emerging standard for AI agent orientation on the web. It's simple enough to implement in minutes and structured enough that agents can parse it reliably.
What does an llms.txt file look like?
The format is intentionally minimal. Here's what a complete llms.txt file looks like for a SaaS business:
# Acme Analytics > Acme Analytics is a B2B SaaS platform for product analytics. > We help growth teams understand user behavior and reduce churn. ## Core - [Product Overview](/product): What Acme Analytics does and who it's for - [Pricing](/pricing): All plans, limits, and pricing — updated monthly - [Documentation](/docs): Full product documentation - [API Reference](/api): REST API endpoints and authentication ## Supplementary - [Blog](/blog): Guides and analysis on product analytics - [Case Studies](/customers): How customers use Acme Analytics - [About](/about): Company background and team ## Skip - [Privacy Policy](/privacy) - [Terms of Service](/terms) - [Login](/login) - [Sign Up](/signup)
What should go in the Core section?
The Core section should contain the pages that give an AI agent the most accurate and complete picture of your business. For most SaaS companies, that means:
- Pricing page with a clear URL and descriptive anchor text like "Pricing: all plans, limits, and monthly costs"
- Product or features page that describes what your software actually does
- API reference if you have one, because agents evaluating software for integration purposes need this immediately
- Documentation root so agents can navigate your full knowledge base
The anchor text you use in square brackets matters. "Pricing" tells an agent very little. "Pricing: all plans from free to enterprise, monthly and annual billing" tells it a lot and improves the quality of the summary it can generate about your business.
What's the difference between llms.txt and llms-full.txt?
The specification also defines an optional companion file, llms-full.txt, which contains the full text content of your core pages concatenated into a single file. This lets agents that support it get your full context in one request rather than crawling multiple pages.
For most businesses, llms.txt is the higher-impact file to start with. It's simpler to create, works with all major AI crawlers, and gives agents the navigation structure they need to prioritize effectively. The full-content file is worth adding later once your core content is stable.
Generate your llms.txt file in 60 seconds
Free tool. No account needed. Download and publish.
Common mistakes when creating an llms.txt file
The most common mistake is listing too many pages in the Core section. The purpose of Core is prioritization. If you list 30 pages as core content, you've told the agent that everything is equally important, which is functionally the same as having no guidance at all. Keep Core to your four to six most critical pages.
The second common mistake is using vague anchor text. "Home," "Product," and "Services" don't tell an agent what's on those pages. Write anchor text as if you're writing a short label for a confused visitor who has never heard of your company. "Product: how the AI readiness scanner works and what it checks" is better than "Product."
The third mistake is including the llms.txt file but forgetting to update it when your site structure changes. An llms.txt pointing to URLs that return 404 errors actively harms the agent's ability to evaluate your site. Treat it as a living document that needs occasional maintenance.
Does Google or traditional SEO use llms.txt?
No. Google's crawler doesn't currently use llms.txt as a ranking signal or navigation guide. The file is specifically for AI agents and language model systems. That said, publishing an llms.txt doesn't harm your SEO in any way; it's a supplementary file that search crawlers will simply ignore.
The distinction matters for planning: don't think of llms.txt as replacing your sitemap or your SEO strategy. Think of it as a parallel layer of optimization for a different category of visitor; one that's growing rapidly in importance as AI agents become more capable and more frequently used for research, evaluation, and purchasing decisions.
In short
llms.txt is a simple markdown file at your domain root that tells AI agents which pages to read first, which are supplementary, and which to skip. Creating one takes under 10 minutes, requires no developer support, and immediately improves how AI agents understand and summarize your business. Keep the Core section to four to six pages, write descriptive anchor text, and update the file when your site structure changes.
Frequently asked questions
Does publishing an llms.txt file affect my search engine rankings?
No. llms.txt is not used by Google or other traditional search engines as a ranking signal. It's designed for AI agents and language model systems. Publishing one has no positive or negative effect on your SEO.
Is llms.txt an official web standard?
llms.txt is a community specification proposed by Answer.AI, not an official IETF or W3C standard. However, it has been adopted by Anthropic for its own documentation and is referenced by a growing number of AI infrastructure providers. Adoption is rapid and it has become a de facto standard for AI agent orientation.
Can I have more than one llms.txt file?
The specification defines one llms.txt at the domain root and one optional llms-full.txt companion file containing concatenated full content. Subdirectory-level llms.txt files are not part of the current specification, though this may evolve.
How do I check if my llms.txt file is working correctly?
Navigate directly to https://yourdomain.com/llms.txt in a browser. If you see your markdown content, the file is published correctly. You can also run the Faro AI Readiness Scan on your site; it checks for a valid llms.txt file as one of its 15 scored checks and will flag any issues.
The fastest way to create a correct, well-structured llms.txt file for your business is the Faro llms.txt Generator. Fill in a few fields, preview the output, and download the file to publish at your domain root. It takes under 60 seconds.