AI crawler discovery may already be using familiar infrastructure: sitemap files and RSS feeds. John Mueller described seeing AI crawlers request both in his own server logs during Google’s October 1, 2026 episode of Search Off the Record, “Do sitemaps still matter?”. The observation is useful for technical audits, but it does not establish how every AI provider discovers or uses a website.
The official episode listing confirms the discussion of sitemaps, RSS and related discovery topics. The specific remarks are reported with excerpts by Search Engine Journal: Mueller said AI training crawlers usually lack a sitemap submission console and suggested conventional naming or RSS feeds for sites seeking discovery. He did not identify the crawlers or establish what they did with the files. The original video and linked transcript could not be independently retrieved during our verification, so the detailed observation is attributed to that report rather than presented as a transcript we inspected.
Discovery and permission are separate questions
A sitemap location can be advertised using the Sitemap line in robots.txt. The Sitemaps protocol explicitly documents this mechanism and notes that the directive is independent of user-agent groups. This identifies the file’s location; it does not grant permission for every subsequent use of the listed content.
Sitemap: https://example.com/sitemap.xmlThe address above is an illustrative placeholder. Before publishing such a line, confirm the actual file exists, contains the intended URLs and is reachable under the access policy chosen for the site.
A feed link can likewise be part of the discovery surface an auditor reviews. Treat conventional paths and published links as implementation choices to test, not as evidence that a particular provider will ingest the file or cite its contents.
What a server log can prove
A request record can show that an endpoint received a request at a particular time, from a particular network address, with a stated user agent and a recorded response. That is valuable evidence for investigating access. It is narrower than evidence of inclusion in a model, an answer or a training dataset.
Our recommendation is to label the measurement precisely. “Observed request to sitemap” is more defensible than “AI indexed the website”. Even a successful response does not tell the publisher how the recipient processed or retained the content.
The user-agent string also deserves scrutiny. Preserve its raw value, then document how identity was verified against the relevant provider’s information where available. An unverified name should remain an unverified request, rather than being counted automatically as activity by a known AI company.
A practical discovery audit
Start with the files the site already exposes. Inventory sitemap indexes, child sitemaps and feeds, then check the public references that point to them. Compare the intended publishing setup with observed requests over a defined period.
| Audit question | Evidence to retain |
|---|---|
| Was the discovery file requested? | Timestamp, endpoint and raw request record |
| Was a useful response served? | Status, response size and relevant delivery-layer records |
| Who appears to have requested it? | Claimed identity and verification method |
| Were listed pages requested later? | Separate page-request records, with association limits stated |
| Did content appear in an AI answer? | A separately captured answer and citation observation |
Choose the observation window before comparing results. A short quiet period should not be treated as proof that a provider never uses a sitemap, and a burst of requests should not become proof of a permanent discovery pattern.
Separate crawl monitoring from citation monitoring
For GEO reporting, keep three streams distinct: access to discovery files, access to content pages and observed appearances in AI answers. They can be investigated together, but they answer different questions.
If a sitemap was fetched and a URL later appeared in an answer, the sequence alone does not prove the sitemap caused that citation. The system may have obtained the page through another route. Record the relationship as an observation unless additional evidence supports a causal explanation.
This is also why logs should not be described as the only visibility report. They are one evidence source for requests seen by the infrastructure being monitored. Captured answers, referral observations and provider-specific reporting can address other parts of the discovery and audience journey.
Build an operational baseline
Before changing discovery files, record their current locations and the coverage of the logs available to the team. Agree which system owns the measurements and how missing or unverifiable records will be handled. Then document any change and compare equivalent periods.
Mueller’s reported observation is a reason to inspect familiar files rather than assume that AI discovery requires a new submission interface. The useful result is a clearer evidence trail: what was advertised, what was requested and what remains unknown about its downstream use.