OpenAI has detailed how its internal online storage platform, Habitat, evolved to support products serving more than one billion ChatGPT users across 40 geographic regions. In the second part of a technical series published on October 23, 2026, OpenAI describes changes to Azure Cosmos DB operations, data replication, distributed caching and service implementation that reduced global median storage-read latency from roughly 120 milliseconds to 10 milliseconds—a 12-fold improvement. The engineering report offers a detailed view of the infrastructure behind globally distributed AI products.
The 12× figure measures Habitat read latency at the 50th percentile, not end-to-end ChatGPT answer-generation speed. That distinction is important: faster application storage can improve responsiveness and development flexibility, but it does not mean model inference or every ChatGPT interaction is twelve times faster.
Habitat: a common storage platform at enormous scale
OpenAI says Habitat has grown from a simple Python client and database into a storage platform serving more than 500 petabytes of data. It handles application information needed by OpenAI products, including settings, state and other properties that must be available reliably across regions.
Read operations account for more than 90% of Habitat requests. This read-heavy profile makes caching especially valuable, but the underlying system must also support reliability, isolation and data residency across varied products and workloads.
Multi-tenancy and the limits of managed databases
Habitat's storage design has shifted between shared and isolated Azure Cosmos DB containers as OpenAI weighed operating cost, workload isolation and the risk of one tenant degrading others. Its current default uses multi-tenant containers while relying on Azure's infrastructure primitives and close engineering collaboration with the database provider.
That collaboration became essential after incidents during regional failover and failback exposed limits that had not been tested at OpenAI's scale. The company says its largest Azure Cosmos DB account stores tens of petabytes and serves more than 20 million queries per second. OpenAI and Azure now coordinate production-scale testing, monitoring, connection reuse, gateway protection and other operational safeguards.
The broader lesson is that managed cloud services do not eliminate infrastructure engineering. At sufficiently large scale, unusual workloads and failure modes require direct cooperation between the customer and cloud provider.
Why global latency became the next challenge
OpenAI's storage and compute resources are not always in the same geographic region. Requests that cross regions incur network delays, and replicating every large dataset everywhere is constrained by datacenter capacity, power, cooling and cost.
Habitat therefore uses a layered approach. Critical, relatively small datasets can be replicated close to users, sometimes across all 40 serving regions. Other workloads benefit from placement decisions that balance latency against available capacity. For most remaining reads, the platform relies on integrated caching near the compute serving users.
Valkey caching with near-real-time invalidation
OpenAI added a distributed lookaside cache based on Valkey, placing cached objects close to where requests are processed. Ordinary time-to-live expiration can trade freshness against cache-hit rates: short lifetimes cause extra database reads, while long lifetimes risk serving stale data.
To improve both performance and freshness, Habitat uses Change Data Capture through Azure Cosmos DB's change feed. When underlying records change, OpenAI's infrastructure propagates invalidations to regional caches within seconds. Subsequent requests can then fetch current values and refill the cache. The design reduces repeated cross-region reads without depending exclusively on arbitrary cache-expiration windows.
Rust and a 12× reduction in median read latency
OpenAI also rewrote the Habitat service from Python to Rust to reduce service-layer overhead. Together with selective geographic replication and regional caching, the work brought global median read latency down from approximately 120 ms to 10 ms during 2026.
The engineering post emphasizes that tail latency remains a challenge when a request misses the cache and must retrieve data from distant storage. The company continues to work on locating data closer to where it is most likely to be accessed.
Change Data Capture becomes shared infrastructure
The same CDC platform supports more than cache invalidation. OpenAI describes millions of captured changes per second flowing through thousands of containers and partitions into more than 100 Kafka topics, supporting secondary indexes, data movement, analytics and other near-real-time functions.
Operating this system requires controls for worker crashes, regional outages, checkpoint efficiency and downstream delivery failures. OpenAI describes an internal control plane that abstracts these details from application developers so product teams can react to changes without managing the underlying pipelines themselves.
What's next: exabyte-scale storage
OpenAI says it is developing a custom online storage system designed for exabyte-scale growth and larger Habitat data types. The proposed architecture combines the RocksDB Cloud core from Rockset with Raft-based replication and disaggregated storage. The trade-off is lower flash-storage demand in exchange for a lower peak query rate.
This is a system under development, not an announcement that the new architecture has already replaced Azure Cosmos DB.
What it means for SEO and AI Search
This is an infrastructure story, not a Google or Bing ranking update, a change in AI citation selection or evidence that any website will receive more AI Search visibility. Its relevance to the broader AI Search ecosystem is indirect: globally available, low-latency storage supports responsive AI-powered applications and the product features built on top of them.
For SEO professionals tracking AI Search, the distinction between infrastructure performance, information retrieval, answer generation and source citation remains essential. Improvements in one layer cannot be interpreted as a new ranking factor in another.
NetContentSEO assessment
OpenAI's report is valuable because it quantifies the scale and engineering complexity required to serve AI products globally. The headline improvement—12× faster median storage reads—is grounded in specific design decisions: selective replication, Valkey caching, CDC-based invalidation and a Rust service rewrite. It illustrates how reliability, latency and operational efficiency become product constraints at billion-user scale, without implying an undocumented change to AI search rankings or citations.
Primary source: OpenAI Engineering, “Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II),” October 23, 2026.