Ahrefs published research showing that 97% of llms.txt files get zero requests from AI crawlers. The finding has spread fast — it's become the go-to citation for anyone arguing that llms.txt is a wasted effort. The problem is that it measures the wrong thing.
The Ahrefs study measured crawler activity: how often do AI systems fetch llms.txt before or during indexing, in the same way Googlebot fetches robots.txt before crawling? The answer is almost never. But that's not the primary mechanism through which llms.txt works — and the difference matters a lot if you're deciding whether to create one.
What llms.txt Was Designed to Do
The llms.txt specification, proposed by Jeremy Howard and documented at llmstxt.org, was designed to give language models a concise, structured summary of a website — what it does, who it serves, where to find key information, and how to navigate the site's most important content. The file is meant to be read at inference time, when an AI model is actively trying to answer a question about your business, not during a background crawl.
Here's the mechanism: when a user asks ChatGPT or Claude a question that triggers a web search or browsing action, the model fetches pages it determines are relevant. If your domain comes up as a candidate, some models will fetch /llms.txt as a quick-read orientation before deciding which pages to request in full. This happens during a conversation, not during a background indexing sweep. Ahrefs' crawler traffic data cannot capture this pattern — it would require access to real-time inference logs across every major LLM provider, which no third party has.
This is not a hypothetical distinction. Anthropic's Claude documentation explicitly notes that Claude can browse and read web pages during conversations. When Claude browses your site in response to a user query, it reads what's there — including llms.txt if present. The Ahrefs crawler study sees pre-indexing traffic, not real-time inference traffic.
What the 97% Stat Actually Measures
The Ahrefs study is well-executed within its scope. It analyzed 3.7 million llms.txt files and looked at how often they received requests from known AI crawler user agents — GPTBot, ClaudeBot, Gemini, and similar. The 97% figure means that 97% of those files saw zero requests from those specific agents within the measurement period.
But the measurement has two important gaps. First, it only captures traffic from identifiable AI crawler user agents. Inference-time web browsing by models like GPT-4o, Claude, or Perplexity uses different request patterns than background crawler traffic — and these are much harder to distinguish from normal browser traffic in server logs. Second, the study measures whether a file was fetched, not whether it influenced a response. A model that fetched your llms.txt file last week and cached its content into its context window will use that information for weeks without triggering a new fetch.
The honest read of the 97% figure is this: almost no AI system runs a background crawl specifically to index llms.txt the way Googlebot indexes pages. That was never the primary purpose of the file.
The Two Real Use Cases for llms.txt
There are two scenarios where llms.txt reliably influences AI responses, and neither one is captured in the Ahrefs crawler study.
The first is inference-time context injection. When an AI agent or user explicitly provides your domain or URL to a model during a conversation, many models will fetch and read llms.txt as a quick-read summary. This is common in agentic workflows where a user says something like "evaluate [company].com for our procurement process" — the model reads your llms.txt to understand what you do before going deeper. The quality of that file directly affects what the model concludes about your business.
The second is system context loading. Several AI tools and platforms load llms.txt files from listed domains at startup or session initialization to give the model background context before a user asks a question. Developer tools, enterprise AI assistants, and some browser extensions do this. Again, this traffic is not reliably captured in crawler analytics.
What Does Affect AI Citation Frequency
The Ahrefs study, despite its methodological gap, points at something real: most of the signals that determine whether an AI cites or recommends you are not about llms.txt at all. They're about the overall quality of your site's structured information.
Faro's scan data puts this in clearer terms. The checks most correlated with low AI readiness scores are: missing JSON-LD structured data (present on only 42% of sites we scan), no machine-readable pricing (absent on 71% of sites), and blocked AI crawlers in robots.txt (present on 59% of sites). These failures prevent AI systems from reading and understanding your site regardless of whether you have a llms.txt file.
The right frame for llms.txt is that it's a quality-of-signal file, not a discovery file. It doesn't make AI systems find you — they find you through search, through links, and through user queries. What it does is help AI systems understand you more accurately and completely once they have found you. That's a narrower claim than "llms.txt gets you more AI citations," but it's accurate.
A Practical Framework for What to Prioritize
If you're deciding where to invest your time on AI agent readiness, here is a priority-ordered framework based on what Faro's audit data shows actually moves the needle:
- Fix crawler permissions first. If GPTBot or ClaudeBot is blocked in your
robots.txt, nothing else matters. An AI cannot read a site it is blocked from. Use Faro's robots.txt analyzer to check your current configuration. - Add machine-readable pricing. AI agents evaluating products for buyers need to find your pricing. Create a
/pricing.mdfile or add pricing to yourllms.txtwith structured format. This is the highest-impact change for businesses that sell to other businesses. - Implement structured data with sameAs links. JSON-LD with a complete
sameAsarray (LinkedIn, GitHub, Crunchbase, Twitter/X) gives AI systems corroborating sources to confirm your identity and details. The Faro AI Schema Creator builds the correct markup for you. - Create or improve your llms.txt. Yes, even after the Ahrefs study. A well-written
llms.txthelps AI systems understand your product accurately during inference-time web browsing. The file should include what you do, who your target customer is, your pricing summary, and links to your key documentation pages. - Build an OKF bundle. The Open Knowledge Format (
/okf/) is a newer standard that goes further thanllms.txt— it provides machine-readable versions of your full product documentation, team, pricing tiers, and API specifications. Use the Faro OKF Generator to create one.
Should You Still Create a llms.txt File?
Yes. The Ahrefs study does not show that llms.txt has no effect. It shows that AI systems do not crawl it in background sweeps. That's a meaningful distinction, and it changes how you should think about the file's value — but it doesn't eliminate that value.
A llms.txt file takes 30 minutes to create if you use the right tools. The cost of not having one is that every AI agent that does read it during an inference-time session gets a worse understanding of your business than it would have otherwise. In a competitive market where other businesses in your category are building these files, not having one is a marginal disadvantage that compounds over time.
The more important lesson from the Ahrefs finding is that llms.txt alone will not make you visible to AI. It is one signal among many. The businesses that do best in AI search are the ones that systematically fix the full stack: crawler permissions, structured data, pricing transparency, documentation quality, and context files like llms.txt and OKF.
In Short
Ahrefs measured crawler traffic to llms.txt files and found almost none. That is factually correct. But it measures a use case that was never the primary one. The file's value is at inference time, not crawl time. Fix your robots.txt, your structured data, and your pricing transparency first — then create a thorough llms.txt to help AI agents understand what they find.
Check your full AI readiness score at byfaro.ai — 30+ checks in 30 seconds, no account required.
FAQ
Does llms.txt actually help with AI search visibility?
It helps at inference time, when an AI model is actively browsing your site during a conversation. It does not trigger background indexing crawls. Think of it as a quality-of-signal file: it helps AI understand you more accurately, not find you in the first place. Fix crawler permissions and structured data first.
What did the Ahrefs llms.txt study actually find?
Ahrefs analyzed 3.7 million llms.txt files and found that 97% received zero requests from known AI crawler user agents during the study period. The study correctly concludes that AI crawlers do not systematically index these files. It does not measure inference-time usage, which is not captured in server log data.
What is the most important thing to fix for AI search visibility?
Based on Faro scan data, the highest-impact fixes are: unblocking AI crawlers in robots.txt (affects 59% of sites), adding machine-readable pricing (affects 71% of sites), and implementing complete JSON-LD structured data with sameAs links (affects 58% of sites). These outweigh llms.txt in terms of measurable impact on AI agent readiness scores.
How is llms.txt different from robots.txt?
robots.txt controls crawler access: which bots are allowed and which are blocked. llms.txt provides content context: a human-readable and machine-parseable summary of what your site contains and who it is for. Robots.txt is a permission file. llms.txt is a signal quality file. Both are part of a complete AI readiness setup, but they solve different problems.