A HasData observational study found that 234 of 592 websites declaring a GPTBot block in robots.txt still returned HTTP 200 to test requests using the GPTBot user-agent string. That represents approximately 39.5% of the sites in this particular subset. The dataset was updated September 29, 2026.
The result reveals a distinction that matters for publishers: a rule declared in robots.txt is not the same thing as an HTTP-level access denial. But it is equally important not to overstate the finding. HasData sent requests that identified themselves with GPTBot's user-agent string; these were not verified requests from OpenAI's official crawler. The test therefore does not demonstrate that OpenAI ignored robots.txt or used any of the returned content for AI training.
Sources: HasData — AI Crawler Block Index; HasData public dataset on GitHub.
What HasData measured
HasData examined websites with declared GPTBot restrictions and checked how their servers responded to HTTP requests carrying the GPTBot user-agent string. Among 592 websites in the reported subset, 234 returned the standard HTTP 200 OK response.
HTTP 200 indicates that the server accepted and fulfilled the request at the HTTP layer. It does not, by itself, indicate whether the requesting software was authorized under robots.txt, whether it behaved as a compliant crawler or whether any content was later stored, indexed or used to train a model.
The numbers at a glance
| Metric | Finding |
|---|---|
| Sites declaring a GPTBot restriction in the tested subset | 592 |
| Sites returning HTTP 200 to test requests | 234 |
| Share returning HTTP 200 | 39.5% |
| Dataset update | September 29, 2026 |
| Request identity | GPTBot user-agent string, not a verified official OpenAI crawler |
The 39.5% figure applies to this tested subset. It is not a measured percentage of all websites, all GPTBot-blocking domains or all OpenAI crawler traffic.
Robots.txt is a set of crawler instructions, not an HTTP firewall
The robots.txt protocol communicates which paths a site asks compliant crawlers not to fetch. For example, a website might publish the following directive:
User-agent: GPTBot
Disallow: /That directive instructs a crawler matching the named user-agent not to fetch pages on the site. It does not require the web server to return HTTP 403 Forbidden whenever a client sends a request claiming to be GPTBot.
A server can therefore return HTTP 200 to a manually issued request while the site still has a valid robots.txt restriction. The protocol relies on compliant crawlers consulting and honoring the instructions.
Why a GPTBot user-agent string is not proof of crawler identity
User-agent strings are ordinary request headers. A testing client can set its user-agent to GPTBot without being operated by OpenAI. This is known as user-agent spoofing or impersonation.
Because HasData's requests were not verified as originating from OpenAI's infrastructure, the responses reveal what the tested servers did when they received that header. They do not establish how the authentic GPTBot crawler would behave when encountering the same robots.txt instructions.
To investigate genuine crawler activity, site operators should use relevant official crawler-verification guidance and corroborating network information rather than trusting a self-declared user-agent string alone.
What HTTP 200 actually proves
At most, a successful HTTP 200 response shows that the test request received an HTTP success status under the tested conditions. It does not necessarily prove that every page was accessible, that the body contained the intended content or that authenticated and verified crawler traffic would receive the same response.
Responses can differ by IP address, path, geographic region, request headers, firewall configuration, bot-management rules and rate limits. An HTTP status code is one observable part of server behavior, not a complete measure of crawler access.
Does this mean OpenAI ignores robots.txt?
No. The study did not establish that OpenAI's verified GPTBot made the requests or violated the published directives. It would be inaccurate to interpret the 234 HTTP 200 responses as 234 cases of OpenAI disregarding robots.txt.
Likewise, a site returning HTTP 200 does not prove that the content entered a training dataset. Crawler behavior, content acquisition and model training are separate questions that require different evidence.
Three separate layers of AI crawler control
| Layer | What it does | What it does not prove |
|---|---|---|
| Robots.txt directives | Communicate crawl preferences to compliant bots | That the server will deny every HTTP request |
| HTTP or firewall enforcement | Can block or challenge requests according to configured rules | That a bot's declared identity is genuine |
| Publisher permissions and AI training policies | Express or govern allowed uses in applicable contexts | That a successful HTTP response authorizes training |
How publishers can audit their GPTBot configuration
- Read the live robots.txt file. Confirm that the intended user-agent group and path rules are present at the correct host.
- Check the syntax. Validate that directives are correctly formed and do not conflict with other groups.
- Decide what control you need. If the objective is to communicate crawler preferences, robots.txt may be the relevant mechanism. If the objective is to prevent HTTP access, additional enforcement is needed.
- Review CDN and WAF settings. Check whether bot-management rules apply to the intended traffic without disrupting legitimate visitors or other crawlers.
- Verify bot identity. Do not assume every request carrying the GPTBot user-agent is authentic.
- Test multiple paths. Check representative public URLs and any sensitive endpoints rather than inferring site-wide behavior from one response.
- Inspect logs. Look for request patterns and network evidence over time.
- Document changes. Save configuration versions and test results so future discrepancies can be investigated.
Why the distinction matters for Cloudflare and other CDNs
Many publishers configure crawler preferences through a content delivery network or security provider. Such systems may offer robots.txt management, bot detection and HTTP-layer blocking as separate controls. Changing one setting does not necessarily change the others.
That separation can create confusion. A publisher may believe a robots.txt disallow directive means all matching requests will be denied by the server. HasData's results illustrate why operators should verify which mechanism is actually active.
Limitations of the HasData observation
The study is useful as a test of server responses under a defined request identity, but it has important limitations. Its GPTBot user-agent requests were not the verified OpenAI crawler; HTTP 200 is not a complete content-access test; and the observed sample is not representative of every website.
Server rules can also change over time. A response recorded in September may differ from what the same site returns today. Any site-specific conclusion requires a current test under documented conditions.
FAQ
How many sites returned HTTP 200 despite declaring a GPTBot block?
HasData reported 234 out of 592, approximately 39.5%, in the tested subset.
Does HTTP 200 mean robots.txt failed?
Not necessarily. Robots.txt provides crawler instructions; it does not require an HTTP-level denial.
Did HasData test OpenAI's real GPTBot?
No. The requests used the GPTBot user-agent string, without verification that they came from OpenAI's crawler.
Does this prove OpenAI used the pages for AI training?
No. The test does not measure training-data use.
How can a publisher enforce a hard block?
Where appropriate, operators can use server or firewall controls in addition to robots.txt, while verifying identity and assessing possible effects on legitimate traffic.
NetContentSEO analysis
HasData's 39.5% finding is best understood as a measurement of the gap between declared crawler preferences and HTTP-level enforcement—not as evidence of GPTBot misconduct. For publishers, the practical lesson is to define the desired control clearly, verify the live robots.txt rules and independently test any CDN or firewall restrictions. Accurate AI crawler governance requires distinguishing instructions, technical access and downstream content use.
Sources: HasData AI Crawler Block Index and public GitHub dataset.