A user request in ChatGPT can fan out into hundreds of database calls. The slowest one is the one the user feels. That single sentence from OpenAI’s September 11, 2026 post on scaling online storage explains why Habitat, its online storage platform, is a useful case study for anyone running a service layer in front of a database.
What Habitat actually is
Habitat began in mid-2024 as a small Python client-side library talking to Azure Cosmos DB. Product engineers did not have to think about schema lookup, routing, authorization, encryption, serialization, request shaping, or connection pooling. Adoption spread without a central mandate.
By mid-2025 the library had hit its limits. A change like migrating critical datasets to regionally distributed Cosmos DB accounts required shipping new routing logic behind a feature flag, rolling it out across dozens of services, adding shadowing, fixing bugs, and re-rolling. OpenAI describes one such rollout that took days and still ended in an outage when a team rolled back to a buggy client. The fix was to pull storage logic into its own service, creating a single point of control for deployments, observability, and security policy enforcement.
Today Habitat handles more than 70 million requests per second, supports products used by over 1 billion people each week, spans almost 40 geographic regions, and serves more than 500 petabytes of data.
The Python bet, and the bill it comes with
OpenAI kept the service in Python even knowing the overhead. A Python service adds network latency and CPU and memory cost compared to local library execution, and the team expected a rewrite eventually. They treated it as deliberate technical debt to unblock product developers, and wagered that Codex and GPT would make the later migration tractable. That bet, they write, proved correct.
The interesting part for builders is what they had to manage in the meantime. Asyncio gives concurrency for I/O-bound work but not CPU parallelism. Habitat also does routing, compression, encryption, checksumming, health checking, shadowing, and hedging. With that much CPU work, asyncio scheduling delay can dominate tail latency: traces showed downstream storage responding quickly while requests stalled waiting for a coroutine to be rescheduled to parse the response.
Their mitigation is counterintuitive if you optimize for throughput per process. They keep each process serving only a small number of concurrent requests and scale out worker processes instead. They also monitor the event loop directly, scheduling background tasks and recording the gap between expected and actual execution time. At high utilization that jitter can reach hundreds of milliseconds, and in edge cases several seconds.
Two concrete bugs worth copying the fix for
The first was feature flag configuration. Statsig polled for refreshed configs every minute with no jitter, and the config carried every production rule across every service. With up to eight Python processes per pod, each pod periodically stalled all workers to parse a giant JSON file. The fix: a smaller targeted config, a longer refresh interval, and jitter on background tasks.
The second was connection pooling fighting load balancing. Client-side pooling meant a busy client process might hold only a handful of server connections and send all its load to a handful of processes. Utilization varied widely, with some tail processes carrying 5-10x the concurrent requests of the average. If you run a proxy or gateway layer, that asymmetry is worth measuring before you tune anything else. The same instinct shows up in prefix-aware routing for LLM latency, where where a request lands matters as much as how fast the backend is.
What to take from this
Habitat’s story is not a case for Python at this scale. OpenAI is explicit that the inefficiencies would not hold at 100x and that a rewrite was almost certain. It is a case for knowing which resource you are actually short on. They chose speed of product iteration over per-request efficiency, then bought back tail latency with process-level tuning, event loop instrumentation, and small operational fixes.
If you run a service layer in front of a datastore, the practical next step is to instrument scheduling delay, not just CPU and memory. The supplied material does not specify how Habitat’s multi-tenancy reliability or read-path optimization work; OpenAI says those are coming in a second post.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.