Google's John Mueller has pointed out an unexpected wrinkle in the growing debate around llms.txt: some of the crawler activity hitting these files may exist because third-party websites are actively finding llms.txt files and linking to them. Once those URLs are published on an ordinary crawlable page, a bot no longer needs native support for the proposed llms.txt convention to discover the file. It can simply follow the link.
Mueller raised the observation on Bluesky, with the comment highlighted in Barry Schwartz's September 18 Search Engine Roundtable recap. He noted that there are now multiple “everything llms.txt” sites that check domains for the file and then link to the files they find. His practical point is simple: when a crawler encounters one of those index sites, it sees conventional hyperlinks and may follow them to the corresponding /llms.txt URLs.
A crawler request does not prove native llms.txt support
This creates an attribution problem for anyone studying server logs. If a crawler requests example.com/llms.txt, it is tempting to conclude that the crawler deliberately probed the conventional path because it understands llms.txt. That conclusion is no longer safe. The bot may have learned about the exact URL from an external page that linked to it.
Technically, there is nothing unusual about that behavior. Web crawlers discover URLs through links constantly. A plain-text file can be fetched after its URL is encountered on another page even if the crawler assigns no special semantic meaning to the filename. Discovery and protocol adoption are two separate questions.
That distinction is especially important for llms.txt, because adoption claims have often relied on observations that particular bots requested the file. A request proves that the URL was fetched. Without additional evidence about how it was discovered and what the crawler did with the contents, it does not prove that the bot implements the proposed convention.
Third-party indexes can manufacture discoverability
The emerging directories Mueller describes effectively create an external-link graph around a file that would otherwise sit quietly at a predictable location on the publisher's domain. An indexer can probe thousands of domains for /llms.txt, record successful responses and publish hyperlinks to those files. General web crawlers subsequently crawling the directory inherit a ready-made list of URLs.
This means llms.txt can become more crawlable because the ecosystem is studying llms.txt. The file does not necessarily need to be discovered through a special AI-specific mechanism. SEO tools, research projects, directories and validators can expose it to the ordinary web graph, after which conventional link discovery takes over.
There are already public services and directories designed to locate or catalog these files, so Mueller's scenario is not merely theoretical. As those indexes expand, the chance that an otherwise obscure llms.txt URL acquires external links increases as well.
This helps explain seemingly contradictory llms.txt data
Recent studies have produced an interesting picture. An Ahrefs analysis of 137,210 domains found that 28% published an llms.txt file, but 97% of those files received no requests during the May 2026 measurement period. Of the small minority that were fetched, most requests came from bots, with only part of that activity attributable to named AI systems.
Other server-log analyses have similarly found that ordinary crawling and tooling can account for substantial llms.txt traffic. That evidence already suggested caution when interpreting a bot hit as proof that an AI platform actively relies on the file. Mueller's observation adds a specific mechanism that can produce those hits: third-party indexes can introduce the URL to crawlers that might never have checked the conventional path independently.
This does not mean every observed llms.txt request comes from a directory, nor does it prove that no AI tool deliberately fetches the files. Different crawlers can behave differently, and agents or developer tools may intentionally retrieve llms.txt in some contexts. The point is narrower and more useful: raw request counts alone cannot identify the discovery mechanism.
Mueller has remained skeptical about llms.txt as an AI discovery signal
The comment fits Mueller's broader position on the proposed standard. Earlier in 2026, he argued on Google's Search Off the Record podcast that llms.txt cannot solve the problem of helping an AI system decide which websites deserve to be surfaced for a query. The file is self-authored by the site it describes, so an external system still needs independent signals to differentiate competing sources.
Mueller and Google's Martin Splitt also emphasized that normal HTML pages and links remain fundamental to crawling and discovery. That matters here because the third-party-index phenomenon is another example of the same underlying web mechanism: links make URLs discoverable. A special filename is not required once another crawlable page points directly to it.
None of this establishes that publishing an llms.txt file is harmful. It simply limits what can responsibly be inferred from its crawl activity. The file may have uses for agents, developer tools or systems that deliberately request it, but a crawler appearing in the access log is not by itself evidence of AI-search adoption.
SEOs testing llms.txt need better experimental controls
For technical SEO and GEO experiments, the practical lesson is to separate discovery from consumption. If the goal is to determine whether a crawler independently probes /llms.txt, researchers should account for external links to the test file. Once a public directory has indexed it, the experiment is contaminated for that specific question because the crawler has another path to discovery.
Referrer information can sometimes help, but it is not sufficient on its own because automated clients do not always send a useful referrer. Researchers can instead compare newly created files with no known external links, monitor when directories discover them, correlate crawl timing with those discoveries and inspect crawler behavior before and after the file becomes externally linked.
A stronger test should also look beyond whether the file was fetched. Does the crawler subsequently request URLs listed only inside llms.txt? Does that behavior repeat consistently across domains? Does the relevant AI provider document support for the convention? Those questions provide much stronger evidence of actual use than a single HTTP request for the file.
The llms.txt ecosystem may now be affecting the thing it is measuring
There is an almost circular quality to Mueller's observation. The SEO and AI-search community wants to know whether crawlers discover llms.txt, so tools are built to find and catalog llms.txt. Those tools publish links to the files, crawlers follow the links, and researchers then observe more crawler requests for llms.txt. Without careful attribution, the ecosystem can partially generate the evidence it later interprets.
That makes this small observation unusually relevant for anyone collecting AI-crawler data. It is a reminder that web measurements are shaped by the link graph surrounding the object being measured. Once a proposed standard becomes popular enough to attract directories, validators and monitoring tools, its traffic patterns no longer reflect only the behavior of the platforms the standard was originally intended to serve.
A crawler fetching llms.txt tells you that the crawler found the URL. It does not tell you how it found it, why it fetched it or whether the file influenced anything downstream. Mueller's third-party-index explanation adds one more reason to keep those claims separate — and one more reason for GEO experiments to treat server-log evidence with considerably more care.