Blocking an AI crawler does not necessarily prevent a website from being cited in Google AI Mode. In a small observational experiment conducted in July 2026, HasData tested 10 queries in Google's AI search experience and recorded 82 distinct cited domains. Of those, 52 appeared in HasData's crawler-policy dataset, and 27 blocked at least one AI crawler.
The finding is noteworthy for publishers managing AI crawler access, but it needs careful interpretation: the 27 domains did not necessarily block Googlebot or the specific Google systems used to retrieve sources for AI Mode. The experiment therefore does not demonstrate that Google circumvented robots.txt rules or other access restrictions.
Primary source: HasData — AI Crawler Block Index, research updated September 29, 2026.
What HasData actually tested
HasData examined citations generated by Google AI Mode for 10 test queries in July. Across the responses, researchers identified 82 unique cited domains. They then checked which of those domains were present in their separate dataset of website crawler preferences.
Fifty-two cited domains matched entries in the HasData dataset. Of those 52, 27 had restrictions targeting at least one AI crawler. This is a cross-reference between citation observations and recorded crawler rules—not a controlled test showing that a prohibited crawler fetched a prohibited URL.
The experiment's numbers
| Metric | Observed result |
|---|---|
| Google AI Mode test queries | 10 |
| Distinct cited domains | 82 |
| Cited domains also in HasData dataset | 52 |
| Matched domains blocking at least one AI crawler | 27 |
| AI Mode test period | July 2026 |
| Research update | September 29, 2026 |
The 27-domain figure represents approximately 51.9% of the 52 matched domains, but that percentage should not be generalized to all AI Mode citations. The underlying experiment included only 10 queries, and not every cited domain was represented in the crawler-policy dataset.
Why 'blocking an AI crawler' is not a universal block
Publishers can configure rules for individual user agents. A robots.txt directive aimed at one AI training crawler may have no bearing on another crawler, including Googlebot. Different crawlers may serve different purposes and follow different documented policies.
For example, a site might include:
User-agent: GPTBot
Disallow: /This communicates a restriction to a crawler matching GPTBot. It does not automatically block Googlebot, all AI-related systems or every search engine. Nor does it establish that the site has opted out of all possible search-result citations.
Does an AI Mode citation prove that Google bypassed a restriction?
No. A citation indicates that a source was referenced in the observed answer. It does not identify which crawler fetched the underlying content, when that content was obtained or whether any applicable restriction was violated.
The experiment also does not establish that all 27 domains blocked the same crawler. Without crawler-specific and URL-specific evidence, it is not possible to infer a uniform mechanism behind their appearance in AI Mode.
Search crawling versus AI training crawling
One of the most important distinctions for publishers is between systems used to discover and index public web pages for search and crawlers associated with AI model training or other purposes. A publisher may wish to allow conventional search indexing while restricting certain AI training activities.
Those choices are not necessarily contradictory. The relevant effect depends on the crawler, the rule, the URL and the search or AI product's documented behavior.
That is why a broad statement such as 'the website blocks AI bots' is often too imprecise for a technical audit. A useful audit records the exact user-agent groups and directives.
What this means for GEO and AI visibility
For generative engine optimization research, the finding suggests that an observed AI-crawler restriction is not enough to predict whether a domain will appear as a citation in Google AI Mode. Citation presence should be measured independently from robots.txt configuration.
However, the small experiment does not show how often blocked domains are cited across the wider web, whether blocking changes citation probability or which sources Google would have used under a different robots.txt policy.
How publishers can audit their own configuration
- Inspect the live robots.txt file. Record exact user-agent groups, disallow directives and affected paths.
- Separate crawler purposes. Distinguish search indexing crawlers from AI training and other bots.
- Check Google-specific guidance. Review the controls documented for Google Search and relevant AI experiences rather than assuming all AI bots behave alike.
- Review HTTP enforcement separately. Firewall blocks and robots.txt instructions are different technical mechanisms.
- Monitor actual citations. Record whether specific URLs or domains appear in AI Mode for representative queries.
- Preserve evidence. Save query wording, date, region, screenshots and cited URLs for repeatable comparisons.
- Avoid universal conclusions. Do not extrapolate from a handful of tests to all markets or query types.
Why the 10-query sample matters
Ten queries are sufficient to document a particular observed behavior but not to estimate the prevalence of that behavior across Google's search ecosystem. Citation selection can vary by topic, query wording, location, language and time.
The dataset match also introduces a second limitation. The 30 cited domains not represented in HasData's crawler-policy dataset cannot be classified from this comparison. They should not be treated as either blocking or allowing AI crawlers without further evidence.
Does the result mean robots.txt controls are ineffective?
No. The study does not test whether a particular compliant crawler obeyed a specific robots.txt directive. It examines citation appearances alongside known restrictions for at least one AI crawler.
Robots.txt can remain meaningful for the crawler it addresses. A publisher seeking to control access should assess the exact behavior of each relevant system rather than equate the presence of an AI Mode citation with a failed robots.txt rule.
What the study does not establish
- That Googlebot was blocked by the 27 cited domains.
- That Google bypassed a robots.txt directive.
- That blocked pages were fetched after a restriction was published.
- That an AI training crawler supplied the cited material.
- That the findings represent all Google AI Mode searches.
- That blocking or allowing any single AI crawler guarantees citation inclusion or exclusion.
FAQ
How many domains did Google AI Mode cite in HasData's experiment?
It cited 82 distinct domains across 10 queries.
How many cited domains appeared in the HasData dataset?
Fifty-two of the 82 cited domains matched dataset entries.
How many of those blocked an AI crawler?
Twenty-seven blocked at least one AI crawler, not necessarily Googlebot.
Did Google ignore robots.txt?
The experiment does not establish that. The observed restrictions may apply to other crawlers.
Does blocking GPTBot prevent Google AI Mode citations?
Not necessarily. GPTBot and Googlebot are distinct crawlers with different operators and purposes.
Can these results be generalized to all AI Mode searches?
No. The sample contained only 10 queries.
NetContentSEO analysis
HasData's experiment highlights why AI citation monitoring and crawler governance must be treated as separate measurements. A website can restrict one AI crawler while remaining visible through another search system. For publishers, the practical response is to audit exact crawler rules, verify access controls and measure AI Mode citations directly—without claiming that the observed citations prove Google circumvented any block.
Primary source: HasData — AI Crawler Block Index.