Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

OpenAI published the second part of its account of Habitat, the storage platform behind its products. The company says Habitat serves over a billion users across 40 regions and holds more than 500 petabytes. This installment covers the Azure Cosmos DB layer underneath it: how OpenAI went from one shared container to many isolated ones and back again, and how it cut global median read latency from roughly 120 ms to roughly 10 ms.

This matters beyond OpenAI. Every login, setting and piece of state in ChatGPT runs through this layer. And the outages it describes show how much of the AI boom now depends on a small number of cloud relationships working at a scale few have tested.

The most useful part of this post is the round trip on multi-tenancy. OpenAI started with one shared container, ran into noisy neighbors where niche features disrupted core login and signup, and moved critical work into dedicated containers. That cut blast radius, but it turned nearly every product team into capacity planners. Teams forgot to pre-scale before launches, and engineers ended up in incident rooms on launch day. So OpenAI went back to a shared default, this time with hot-key throttling, request priority tiers, and a new serverless mode the Azure team built for it. OpenAI says it hasn't had a provisioning or noisy-neighbor incident in the months since. We find that credible as a direction, and it's a case study worth stealing: isolation has an organizational cost that doesn't show up on an architecture diagram.

The outage section is the part we keep coming back to. In late 2024, a power outage hit the primary region for one of its largest accounts, and failover took hours, which neither side had tested at that size. About a month later, failing back set off a connection stampede that overwhelmed the gateway. OpenAI's own conclusion is blunt: at its scale, managed services become less managed. The answer was joint dashboards, a production-sized test account, and direct paging between engineering teams.

That's the right fix for OpenAI. It's also a signal of concentration. When one customer runs what it believes is the largest Cosmos DB deployment anywhere, with tens of petabytes and more than 20 million queries per second on a single account, the provider ends up building features for that customer and co-owning its failures. That works until incentives diverge. It also means smaller customers on the same service inherit tooling shaped around someone else's edge cases.

The latency work is solid engineering: replicate only the smallest, most latency-sensitive data such as authentication metadata widely, use a Valkey cache close to users with invalidations driven by the database's change feed, and rewrite the proxy from Python into Rust. One detail we want to flag: in regions short on power and cooling, OpenAI says it sometimes has to choose between storage near users and more serving compute. Physical capacity is now shaping product latency, and that pressure won't ease soon.

The forward note is telling. OpenAI is building its own exabyte-scale storage system on RocksDB Cloud from Rockset with Raft replication, using the same document model. The likelier read is a hedge: keep the Azure partnership while building the option to rely on it less.

Part I covered Habitat's evolution from a Python client library into a service. Part II focuses on the storage layer: Cosmos DB's partition model, where throughput per physical partition shrinks as data grows and partitions split, and the change-data-capture platform that streams millions of row changes per second through more than a hundred Kafka topics for caching, indexing and analytics.