Rapidly scaling online storage to serve over 1 billion ChatGPT users

OpenAI published part one of a deep dive on Habitat, the online storage platform behind ChatGPT, Codex, the API, and internal services. The numbers are the headline: more than 70 million requests per second, support for over one billion people each week across nearly 40 regions, and more than 500 petabytes of data.

Every login, settings check, or new conversation can trigger many lookups. If Habitat is slow or down, the product feels broken. This is the invisible machinery behind consumer AI at planetary scale—and a rare look at how OpenAI kept a Python-first stack alive while growing more than 10x year over year for three years.

For the industry, it is also a reminder that “AI product” reliability is increasingly a distributed-systems problem: placement, residency, connection pools, and tail latency—not just model quality.

We’re glad this post exists. Frontier labs usually sell magic; here staff engineers Jon Lee, Chaomin Yu, and Ben Ries walk through the unglamorous path from a DevDay 2023 Python library for GPTs to a centralized service that owns routing, auth, encryption, caching, and residency on top of Azure Cosmos DB and related stores.

The strategic read is deliberate technical debt. By mid-2025 the client library couldn’t take coordinated changes across dozens of services without multi-day rollouts and rollback landmines. OpenAI pulled Habitat into its own service, accepted Python’s CPU and latency tax, and bet that Codex and GPT would later make a migration—toward Rust, in their framing—achievable. They say that bet proved correct. That is a quietly important AI-for-infra story: coding models as a hedge that lets you ship the wrong language longer without dying.

The operational war stories are where the post earns trust. Asyncio scheduling delay dominating p99s. Statsig feature-flag JSON parsing stalling every worker in a pod on the minute. Client connection pools using LIFO reuse and creating metastable overload—fixed by switching toward FIFO and leaning on Istio/Envoy for smarter balancing. Constraining the API to a NoSQL object-and-edge model inspired by TAO so product teams can’t invent unbounded SQL that takes out the database. Habitat as a chokepoint for ACL, audit logs, and limiting access from external, internal, and agent actors.

Useful AI needs this kind of plumbing. A model that reasons brilliantly is worthless if session state and settings can’t be fetched fast in forty regions. We’re advocating for the engineering honesty here: scale first for developer unblock and safety controls, then rewrite when the models make the rewrite cheap.

The downside if everyone copies only the headline numbers is cargo-culting “70M QPS” without the constrained API and central security choke. Habitat does less on purpose. I’m watching part two on multi-tenancy and Cosmos partnership details, whether the Python-to-Rust migration timeline becomes public, and how openly other labs describe agent-aware access controls on storage—because agents as first-class threat actors inside the data plane is the quiet sentence that matters most.

Desk take: this is how you keep a billion-user AI product standing—boring, ruthless platform work, with coding models as a force multiplier on the migration, not a substitute for systems judgment.

Habitat began as a simple Python client against a single database for custom GPTs, then became a distributed platform. OpenAI says engineers often plan for 10x scale over years; Habitat had to absorb that growth repeatedly while product demand kept climbing. A follow-up post is promised on multi-tenancy reliability, read-performance layering, and scaling the Azure Cosmos DB partnership.