OpenAI scales Azure Cosmos DB to serve 1 billion ChatGPT users

Initially, OpenAI relied on a shared default container, which led to performance issues like noisy neighbors and throughput bottlenecks. The team then moved to separate containers for critical workloads, but this created significant capacity planning challenges for product teams. To resolve this, OpenAI collaborated closely with the Azure team to develop a new multi-tenant approach using co-managed containers. This included implementing request tiering and a new serverless mode that allows physical partitions to scale up to 10,000 request units per second.

The partnership also addressed reliability gaps exposed by a regional power outage in late 2024, which revealed that failover times for such massive accounts were untested and problematic. OpenAI and Azure now jointly monitor and test the infrastructure, sharing dashboards and engineering channels to manage failure modes at a scale where standard managed services are no longer sufficient.