Technical SEO

97% of llms.txt Files Never Get Read, Ahrefs Finds in 137,000-Site Study

Almost nobody reads llms.txt files, not even the AI bots they were written for. Ahrefs analysed server logs and live traffic for 137,210 domains in Ahrefs Web Analytics, using May 2026 data, and found that 97% of llms.txt files received zero traffic that month. The file has become one of SEO's most debated conventions, and the data suggests the debate is louder than the usage.

The headline numbers

The study's headline figures are stark. Of the 137,210 domains analysed, 28% publish an llms.txt file, which is a substantial adoption rate for a convention that is barely a year old in mainstream SEO discussion. But 97% of those files received zero traffic in May 2026. In other words, more than a quarter of the sites in the study bothered to create the file, and almost none of them ever saw a single request for it. The file exists. It is simply not being read.

Of the requests that did reach llms.txt files, 96% came from bots rather than humans. That is expected for a machine-readable file, but it underlines the point: llms.txt is not a document people consult. It is a file that machines are supposed to fetch, and the data shows that most machines do not bother either. The gap between adoption and usage is the story here. SEOs adopted llms.txt on the promise that AI systems would read it. The logs say they mostly do not.

Who actually fetches llms.txt

The breakdown of who does fetch the file is revealing. Some 19.5% of fetches came from named AI tools, with GPTBot on top and Claude-Code second, ahead of every AI search and assistant bot. Another 12% of requests came from tools studying llms.txt itself: GEO and AEO tools, checkers and researchers measuring adoption rather than using the file's contents. The Chrome Lighthouse llms.txt audit produced roughly 1 in 1,000 fetches, a small share that shows even the tooling built to check for the file barely moves the needle.

The most striking detail concerns the bots the file was supposedly built for. AI retrieval bots, OAI-SearchBot and PerplexityBot, account for just 1.1% of requests. Slackbot fetched llms.txt files more often than PerplexityBot did. AI training crawlers fetch llms.txt nearly five times more than AI retrieval bots do, which suggests the file is more relevant to training data collection than to live search answers. And zero requests came from AI bots for llms.txt files that do not exist, which means no bot is going looking for the file speculatively. It only gets fetched where it already exists, and even then, rarely.

What Google says about llms.txt

Google's position, as the study reports it, is unambiguous. Google's guide says machine-readable files like llms.txt are not needed to appear in generative AI search, and John Mueller has called llms.txt 'not done for search', describing it as a 'temporary crutch' for AI coding tools parsing developer documentation. That framing matters, because it separates two different uses of the file: helping coding assistants understand documentation, which is a real if narrow job, and influencing search or AI answers, which Google says it does not do.

The data supports that distinction. The bots that fetch llms.txt most are coding and general AI tools, led by GPTBot and Claude-Code, not the retrieval bots that power AI search answers. A file that helps an AI coding assistant parse your API documentation is doing something useful, even if no search bot ever reads it. But a file published in the hope of winning citations in ChatGPT or AI Overviews is, on this evidence, addressing an audience that is not listening.

Why it matters for SEOs and link builders

One caveat deserves emphasis before drawing conclusions. Ahrefs notes that its Web Analytics customers skew more technical and SEO-aware than the web as a whole, so the 28% adoption figure should be treated as an upper bound. The real web adopts llms.txt less, not more. But that skew cuts both ways: if even this technical, SEO-aware sample sees 97% of its files unread, the file's practical value as a visibility lever is thin indeed. The study does not say llms.txt is harmful. It says it is, for most sites, unread.

For SEOs, the practical takeaway is to stop treating llms.txt as a ranking or citation tactic and to keep it, at most, as a low-cost courtesy for coding tools where developer documentation matters. The hours spent debating, drafting and maintaining the file are better spent on the content that retrieval bots actually read: clear, well-structured pages that answer real questions. For link builders, the lesson is the same. Citations in AI answers come from content worth citing, not from a text file asking to be cited. The bots vote with their fetches, and they have voted for your pages, not your llms.txt.

Sources

Sources: Ahrefs blog, updated November 6, 2026 (originally published June 15, 2026), reporting on an analysis of server logs and live traffic for 137,210 domains in Ahrefs Web Analytics.

Related reading on this site: Google Will Support the AIPREF AI-Training Opt-Out Signal in Early 2027, ChatGPT Fans Out: Most Prompts Never Search the Web, Ahrefs Finds, ChatGPT Retrieves Twice as Many URLs as It Cites, Ahrefs Finds, Ahrefs: Half of AI Citations Vanish Within a Day, Most Return, Schema Markup Barely Moves AI Citations, Ahrefs Study Finds, and Google Says It's OK to Block Some JavaScript Resources With Robots.txt.

More from Linkbaits

Turn this playbook into links

Linkbaits helps you create content that earns citations and backlinks. Start with the toolkit and build your first linkable asset this week.

Start free trial