Only 4.6% of 457,225 German homepages explicitly mention GPTBot in their robots.txt files, according to a large-scale study updated on October 28, 2026. That is the headline result of an analysis by SEO researcher Hasan Kalkan using HTTP Archive data. But the number needs careful interpretation: mentioning a crawler is not the same as blocking it, and not mentioning one does not guarantee unrestricted access.
The original research, also discussed in r/TechSEO, compares German websites with a worldwide sample, looks at differences by website popularity and ecommerce platform, and tracks changes from February to October 2026. For technical SEO teams navigating AI crawlers, the findings offer a useful reality check on the claim that the web is rapidly closing itself off to AI.
What the 457,225-site sample actually represents
Kalkan used HTTP Archive's monthly mobile datasets. In October 2026, the dataset contained 552,750 homepages hosted on domains ending in .de, of which 457,225 had a robots.txt file containing at least one rule group. The study's percentages use those 457,225 homepages as the denominator.
That scope matters. These are .de homepages represented in HTTP Archive and meeting the study's robots.txt criteria, not a census of every German business, every German-owned website or every domain used in Germany. HTTP Archive's underlying collection also reflects sites with sufficient Chrome User Experience Report activity.
For comparison, the worldwide dataset included 15,366,242 homepages, of which 12,023,939 had robots.txt rule groups. The research examined explicit user-agent names and counts of Allow and Disallow directives for 13 AI-related identifiers.
GPTBot: 4.6% mentioned, 2.5% with a block indicator
In the German sample, 4.6% of homepages explicitly name GPTBot in robots.txt. A smaller 2.5% have at least one Disallow rule and no Allow rule associated with that identifier. Worldwide, the corresponding figures are 8.3% mentioning GPTBot and 3.5% meeting the study's block indicator.
The study deliberately calls the second measure an indicator. Its underlying aggregate data do not preserve the specific paths in those rules. As a result, the indicator cannot distinguish a complete prohibition such as Disallow: / from a rule that excludes only a subset of URLs.
Equally important, a robots.txt file can mention GPTBot without restricting it. Platforms may include AI user agents in default configurations or group them with general crawlers. And a site that does not explicitly name GPTBot may still restrict it through rules for User-agent: *. Therefore, neither 4.6% nor 2.5% should be presented as a definitive percentage of German sites that allow or completely block AI training crawlers.
Other AI crawler identifiers in German robots.txt files
The study found explicit mentions for ClaudeBot on 3.7% of sampled German homepages, Bytespider on 3.0%, CCBot on 2.9% and Google-Extended on 2.8%. The corresponding Disallow-without-Allow indicators were 1.4%, 2.1%, 1.6% and 1.1%.
Search and user-triggered agents were mentioned less often: OAI-SearchBot appeared on 1.7% of German homepages, with a 0.5% block indicator, while ChatGPT-User appeared on 2.1%, with a 0.8% indicator.
These are descriptive measurements of explicit robots.txt entries. They do not establish how often each bot actually visits a website or whether the operator deliberately chose every listed rule.
The surprising reversal: large German websites restrict GPTBot more often
Germany's overall GPTBot mention rate is below the worldwide figure, but the pattern changes dramatically among popular sites. For German homepages with a CrUX popularity rank of up to 100,000, 28.8% explicitly name GPTBot and 23.0% meet the Disallow-without-Allow indicator. The comparison group of similarly popular homepages outside Germany shows 13.9% and 10.2%, respectively.
The study's German top-100,000 group contains 1,088 homepages. Its size measure comes from CrUX navigation-based popularity buckets, not business revenue or brand awareness. Smaller samples at the very top of the ranking require particular caution.
Most of the German sample lies in the long tail: 443,770 of the 457,225 homepages have popularity ranks beyond one million. The aggregate 4.6% figure is therefore strongly influenced by smaller websites. The research does not prove why larger sites differ; enterprise governance, platform defaults and legal policies are possible explanations, not established causal findings.
Ecommerce platforms reveal why 'named' is a misleading metric
Platform comparisons underline the importance of default robots.txt templates. Among the German ecommerce sites identified in the dataset, 99.8% of Squarespace Commerce homepages name GPTBot, but just 0.2% meet the study's block indicator. That enormous difference shows why a mention count cannot be used as a proxy for opposition to AI crawling.
Other platforms show very different patterns. Magento stores in the sample have an 8.1% GPTBot mention rate and 5.4% block indicator; WooCommerce shows 2.3% and 1.1%; Shopify 1.8% and 0.3%; and Wix ecommerce just 0.1% and 0.0%.
These are unadjusted platform comparisons. Store size, operator behavior, default configuration and sample composition may all contribute. They should not be interpreted as evidence that one platform automatically delivers better AI visibility than another.
Is AI crawler blocking becoming more common?
Using a matched sample of German homepages present in both February and October 2026, Kalkan found GPTBot explicit mentions rising from 3.5% to 4.4%. The Disallow-without-Allow indicator rose from 2.4% to 2.6% on those same pages. Worldwide, mentions in the matched sample increased from 7.6% to 7.9%, while the block indicator declined from 4.3% to 3.5%.
These figures are not the same denominator as the October snapshot and should not be mixed casually. The study reports 5,717 German homepages adding an explicit GPTBot entry and 1,956 removing one over the comparison period, but the aggregated counts do not explain each operator's reason for the change.
The trend suggests growing awareness of AI crawler controls among some German site operators. It does not justify the claim that all websites are increasingly blocking all AI bots.
GPTBot and OAI-SearchBot serve different purposes
The most important practical distinction for AI Search is between crawler identities. According to OpenAI's official crawler documentation, GPTBot is used to crawl content that may contribute to training generative AI foundation models. By contrast, OAI-SearchBot supports website discovery for ChatGPT's search features.
OpenAI says the controls are independent. A publisher can disallow GPTBot to indicate that content should not be used for model training while allowing OAI-SearchBot to remain eligible for ChatGPT Search results. OpenAI notes that websites opting out of OAI-SearchBot will not be shown in ChatGPT search answers, although they may still appear as navigational links.
A third identifier, ChatGPT-User, is associated with certain user-triggered requests and is not used for automatic web crawling. OpenAI warns that robots.txt rules may not apply to every such user-initiated action. These agents should not be lumped together under a generic 'OpenAI bot' rule without considering the intended outcome.
A practical robots.txt policy for publishers
Before changing robots.txt, decide what you want to control. Are you limiting use of content for model training? Are you restricting automated discovery for AI search results? Or are you managing access to resource-intensive parts of your website?
For a publisher that wants to restrict GPTBot while retaining ChatGPT Search eligibility, the intended policy can be expressed with separate user-agent groups:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This is a simplified example, not a universal recommendation. Existing wildcard rules, specific path exclusions, security controls and other user-agent groups must be evaluated together. A real implementation should be tested on the site's actual robots.txt file rather than pasted without review.
Remember that robots.txt is a crawler instruction, not authentication or access control. It does not prevent a human from loading a public page, and a noncompliant bot may ignore it.
Can robots.txt statistics prove that bots obey the rules?
No. The HTTP Archive measurements capture robots.txt declarations, not actual bot compliance. Kalkan includes a limited server-log example from a large shop that blocks Bytespider, but the observed requests were too sparse and their origins insufficiently verified to establish compliance conclusively.
The study also notes that some requests claiming to be Bytespider came from networks other than ByteDance. This is a reminder that user-agent strings can be spoofed. For operational monitoring, examine server logs alongside published crawler IP ranges and reverse-DNS or other documented verification methods where available.
Even a verified crawler's presence in logs does not necessarily show which content it used or how that content affected an AI product. The study does not measure model training, citation frequency or referral traffic.
What the findings mean for AI Search and GEO
AI Search optimization is often described as a content challenge: publish clear answers, structure information and establish authority. Those steps can matter, but technical access remains foundational. A site that inadvertently blocks a search-oriented crawler may limit its eligibility for that system's answers, regardless of content quality.
At the same time, permitting a crawler does not guarantee that an AI assistant will cite the website. Visibility depends on many factors beyond robots.txt, and the study does not provide a ranking or citation model.
SEO teams should therefore audit crawler policies as a separate technical workstream: inventory explicit user-agent groups, review wildcard behavior, compare training and search crawlers, verify server responses and document intentional restrictions. Changes should be tied to a business decision rather than a generic instruction to allow or block every AI agent.
What this research does not prove
The dataset does not cover every website in Germany, does not show the precise paths behind the aggregated block indicator, does not establish crawler compliance, and does not measure whether a website appears in ChatGPT or another AI assistant.
Nor does it prove that German website owners are uniquely hostile or welcoming toward AI. Differences in platform defaults, site size and sample composition can substantially affect the observed rates.
These limitations make the study more useful, not less: they show which questions the available evidence can answer and which require server logs, full robots.txt inspection or separate visibility data.
NetContentSEO assessment
The most important finding is not simply that 4.6% of sampled German homepages name GPTBot. It is that crawler policy is fragmented, often inherited from platform defaults and frequently misunderstood. Large sites behave differently from the long tail, and training-related robots controls are not interchangeable with AI Search discovery controls.
For publishers, the immediate action is an audit, not a panic-driven change. Identify which bots matter to your strategy, verify the effective robots.txt rules and preserve the distinction between controlling model training and maintaining visibility in AI-powered search.
Sources: Hasan Kalkan, HTTP Archive study, updated October 28, 2026; OpenAI, Overview of OpenAI Crawlers; Reddit r/TechSEO discussion.