Do AI Agents Actually Use Your Sitemap.xml?

Most sites maintain a sitemap.xml file and quietly assume every crawler that matters reads it, because that is how search has worked since the mid-2000s. Googlebot and Bingbot do read it. The agents behind ChatGPT, Claude, and Gemini mostly do not, and treating your sitemap as your AI-discovery plan leaves the levers these agents actually use untouched.

A traced study of 1,033 agent runs found sitemaps fetched directly in just 4% of them. Agents found what they needed a different way: through links, through docs pages, and occasionally through an llms.txt file, but only when something on the site actually pointed to it. That changes what "make my site discoverable to AI agents" should mean in practice.

In this article

• Do AI crawlers actually read your sitemap.xml?
• What is sitemap.xml actually built for?
• What do AI agents fetch instead of your sitemap?
• Does llms.txt replace your sitemap.xml?
• How do OAI-SearchBot and ClaudeBot actually find your pages?
• Sitemap.xml vs llms.txt vs internal links: what each one does
• Should you still keep a sitemap.xml?
• What actually helps AI agents find and index your pages?
• FAQ
• In short

Do AI crawlers actually read your sitemap.xml?

Mostly no. A traced study of 1,033 agent runs found sitemaps fetched directly in just 4% of them, and neither OpenAI's nor Anthropic's own crawler documentation describes sitemap parsing as part of how their bots discover pages. Googlebot crawls broadly on its own schedule and uses your sitemap as a hint about which URLs exist and how to prioritize them. AI agents work differently: most fire per query or per task, following links they encounter along the way rather than ingesting a bulk URL list up front.

That 4% is not zero. Some agent fetches do touch a sitemap, likely when a task specifically calls for a full inventory of a site's URLs. But building your AI-discovery strategy around sitemap submission, the way you might for Google Search Console, puts your effort into the file agents use least.

What is sitemap.xml actually built for?

A sitemap.xml file exists to tell a search engine which URLs on your site exist and how to prioritize crawling them, with extensions for images, video, and news content. According to Google's own sitemap documentation, an XML sitemap is the most versatile format: it is extensible, can supply metadata about images and video, and can list localized versions of a page. Google explicitly notes it has no preference between sitemap formats, only that one exists.

That design assumes a crawler that runs broad, scheduled sweeps and needs a map to crawl efficiently across a large site. It is a batch-discovery tool built for a batch-crawling model. AI agents answering a specific question do not run that kind of sweep; they fetch the pages a task actually requires, discovered through links, not through a pre-submitted inventory.

What do AI agents fetch instead of your sitemap?

Links. The same traced study found docs pages reached in 83% of agent runs, about 3.4 docs pages per run, almost entirely by following links rather than reading a sitemap. An agent lands on one page, usually through a search result or a direct link, then follows the links on that page to reach the next one. That is closer to how a person browses a site than how a search engine crawler sweeps one.

This puts the weight back on ordinary on-page navigation: clear internal links, a visible docs section, and a homepage that actually links to your deeper content. A page an agent cannot reach by clicking through from somewhere else is a page most agents will not find, sitemap or not.

Does llms.txt replace your sitemap.xml?

No, they do different jobs. An llms.txt file is a curated markdown pointer: a short, human-and-agent-readable summary with links to the pages an agent most needs, not a comprehensive inventory of every URL on your site. Per the llms.txt specification, it proposes a single markdown file at the site root offering background and guidance, plus optional clean markdown versions of individual pages. The same specification notes that OpenAI, Anthropic, and Gemini each publish their own llms.txt file for their developer docs, and that Chrome's Lighthouse now audits for one as part of its agentic browsing checks.

But an llms.txt file only works if an agent finds it, and the same study found 86% of its fetches came from a page that linked to it first. Left unlinked, it sits at your root doing nothing. About one in three agents that did read it went on to fetch a page it pointed to, which makes it a real discovery multiplier once found, just not a sitemap replacement and not something that helps on its own without a link pointing to it.

How do OAI-SearchBot and ClaudeBot actually find your pages?

Through crawled links and robots.txt permissions, not sitemap submission. OpenAI's own crawler documentation describes OAI-SearchBot as the crawler that surfaces your pages in ChatGPT's search results, GPTBot as the crawler that collects content for model training, and ChatGPT-User as a live, single-request fetch triggered when a person asks ChatGPT to look at a specific page. None of the three is described as reading a sitemap to decide what to crawl.

Anthropic's own documentation describes the same split for ClaudeBot, Claude-SearchBot, and Claude-User, and states all three honor robots.txt disallow rules and the non-standard Crawl-delay directive. Discovery, in every case the labs document themselves, runs through links and permission rules, not a submitted URL list.

You can check exactly which of these crawlers your site currently allows, without reading four separate crawler policy pages, with Faro's robots.txt tool. It is worth pairing with the full breakdown in OAI-SearchBot, Claude-SearchBot, Claude-User: a robots.txt guide, since training crawlers and search crawlers carry separate permissions and a lot of sites only configure one.

Sitemap.xml vs llms.txt vs internal links: what each one does

These three mechanisms solve different discovery problems and none of them substitutes for the others. The table below lines up their purpose, who actually reads them, and how often AI agents used each one in the traced study.

Mechanism	Built for	How it gets used	AI agent usage (traced study)
sitemap.xml	Search engine crawl scheduling (Googlebot, Bingbot)	Submitted via Search Console or referenced with a Sitemap: line in robots.txt	Fetched directly in 4% of runs
llms.txt	A curated pointer for AI agents, if they can find it	Must be linked from the homepage, footer, or an HTTP Link header	86% of its fetches came from a page linking to it; about 1 in 3 readers then fetched a page it lists
Internal links / docs pages	Any crawler or agent following ordinary HTML links	Normal navigation, footer links, in-content links	Docs pages reached in 83% of runs, about 3.4 per run

Source: Google's sitemap documentation, the llms.txt specification, and a September 2026 study of 1,033 traced agent runs.

Should you still keep a sitemap.xml?

Yes. A 4% direct-fetch rate from AI agents is not a reason to drop a file that still does real work for Google and Bing, and most content management systems generate one automatically at no ongoing cost. The mistake is not having a sitemap; it is treating "I submitted a sitemap" as equivalent to "AI agents can find my content," when the study above shows those are two different problems with two different solutions.

Keep the sitemap for the search engines it was built for. Spend your AI-discovery effort on the things the traced data says agents actually use: crawlable links, correctly configured search-bot permissions, and an llms.txt file that is actually linked from somewhere.

What actually helps AI agents find and index your pages?

Link your llms.txt from somewhere an agent will actually see it, not just at the root and nowhere else. A footer link, a <link rel="alternate" type="text/markdown" href="/llms.txt"> tag in your page head, or an HTTP Link header all work; the llms.txt guide walks through each option, since an unlinked file at your root sits in the 14% of fetches that never happened.

Beyond that, four things the traced data and the crawler documentation both point to:

• Allow the right search crawlers in robots.txt, not just the training ones. GPTBot and ClaudeBot cover training; OAI-SearchBot, Claude-SearchBot, and Claude-User cover live AI search and assistant visibility, and blocking the wrong one by accident is common. See OAI-SearchBot, Claude-SearchBot, Claude-User: a robots.txt guide.
• Make sure your main content is readable without JavaScript. An agent that cannot render your scripts sees an empty shell, regardless of what your sitemap lists. See can AI crawlers run JavaScript.
• Return a real 404 for missing pages instead of a soft 404. An agent that gets a 200 response for a page that does not exist draws the wrong conclusion about what your site actually contains. See why soft 404s confuse AI agents.
• Give agents structured, machine-readable business information beyond prose, through tools like Faro's OKF generator, so they are not inferring facts about your business from paragraph text.

Checking all of this by hand, across robots.txt, llms.txt linking, JavaScript dependence, and 404 behavior, is the kind of audit that falls off a team's radar between quarters. Faro's AI visibility scan checks sitemap presence, whether llms.txt is actually linked anywhere, and whether missing pages return a real 404, as part of its content discoverability checks, so you get the full picture in one pass instead of four separate ones.

FAQ

Does ChatGPT read my sitemap.xml?

There is no official statement from OpenAI that OAI-SearchBot or GPTBot parses sitemap.xml for discovery, and a traced study of 1,033 agent runs found sitemaps fetched directly in only 4% of runs. Submitting one will not hurt anything, but it will not drive ChatGPT visibility the way it drives Google indexing.

Is llms.txt a replacement for my sitemap.xml?

No. A sitemap lists your URLs for search engine crawl scheduling. An llms.txt file is a short, curated pointer for AI agents, and it only works if something links to it: 86% of its fetches in the traced study came from a page that pointed to it first.

Do I still need a sitemap.xml in 2026?

Yes, for Google, Bing, and any traditional search engine that still uses one to schedule crawling. Keep it. Just do not treat it as your AI-agent discovery strategy, since the same data shows AI agents barely touch it directly.

How do AI agents actually find new pages on my site?

Mostly through links. The traced study found docs pages reached in 83% of agent runs, about 3.4 per run, almost entirely by following links from page to page rather than reading a sitemap or a bulk URL list.

Which AI crawlers should I allow in robots.txt to show up in AI search results?

OAI-SearchBot and Claude-SearchBot specifically control whether you appear in ChatGPT and Claude search results, separate from the GPTBot and ClaudeBot training crawlers. The full breakdown is in OAI-SearchBot, Claude-SearchBot, Claude-User: a robots.txt guide.

In short

Sitemap.xml is a Google and Bing tool that AI agents mostly skip: a traced study of 1,033 agent runs found it fetched directly in only 4% of them. Agents instead rely on links, reaching docs pages in 83% of runs at about 3.4 per run, and on a properly linked llms.txt file, which accounted for 86% of its own fetches coming from a page that pointed to it. Keep your sitemap for traditional search engines. Put your AI-discovery effort into crawlable internal links, correct search-crawler permissions, and an llms.txt file that is actually linked from somewhere an agent will find it.

Run Faro's AI visibility scan to see whether your sitemap, your llms.txt linking, your search-crawler permissions, and your 404 behavior are actually set up the way the data above says they need to be, rather than guessing from four different specification pages.