TL;DR
OpenAI replaced its monorepo Git host with Pando, a Git-compatible Rust backend, after Codex multiplied parallel pushes and fetches. Median push latency fell nearly 80% (about 5x) and it now serves nearly 1 PB of Git data per day. The core ideas: store each push as an immutable pack in blob storage, coordinate refs per branch (causal consistency) instead of per repository, and answer fetches by reusing stored packs, delivered as packfile URIs from regional caches.
Source: Scaling Git at OpenAI by Aaron Friel, Taylor Blau and Ted Nyman, published 2026-11-02. Checked against the original article on 2026-11-03. All performance figures are OpenAI's own, not independently reproduced.
The problem
Codex let each engineer run far more work in parallel, which changed the shape of the load as well as its size. From around the end of 2025 the old system struggled.
Many writers: more branches, pushes and merges, with several changes per engineer waiting on CI at once. Pushes were slow and merges failed.
Many readers of the same history: each push can start several CI jobs, dev environments and research jobs that fetch the same commit, and the server built a fresh pack for each one.
Reliability: operations sometimes succeeded less than 90% of the time.
They decided the limit was how they served the monorepo, not Git itself. They kept Git for its ecosystem, so the stock client and protocol stay unchanged and client fixes can go upstream.
Designs they rejected:
Three-phase commit across full replicas: reads are local and refs look consistent, but each push needs several round trips, replica disk and network latency sit on the push path, and refs update one at a time. In OpenAI's simulation, write latency grew as disks saturated.
Shared write-ahead log in blob storage: pack uploads run in parallel, but every push, even to unrelated branches, contends on one manifest updated with compare-and-swap. The CAS round trip caps the publish rate, and adding blob capacity doesn't help. Replicas must also read and index each pack before serving it, so under load fetches of new commits fall behind.
Both fail because they keep two assumptions: one consistent snapshot of all refs, and a unique pack built for each client. Pando drops both.
Architecture
Write path. A push streams into blob storage as one immutable pack and is validated in a single pass, without being unpacked into a local repository. One multi-key transaction then writes a "ref version" (new tip, the pack that supplied it, the earlier ref versions it depends on) and moves the ref pointer, which gives read-your-own-write. The ref pointer is the only contended key, guarded by a CAS-like atomic operation. Metadata work is O(1) per push, even for very large pushes, because only some commits are recorded synchronously.
Read path. Pando maps the client's haves and wants to ref versions using commit mappings recorded at push time, then picks the stored packs that cover the gap. This takes O(log Δ) metadata reads, where Δ is the ref-version distance. It replies with an empty inline pack plus packfile URIs, and the client downloads the packs from Express, a regional pack cache routed by consistent hashing (over 99% hit rate). A separate cache sits in front of the metadata store.
Compaction. Each push makes an L0 pack. Background workers group packs into "covers" over runs of ref versions that grow geometrically (the monorepo reaches L4), and repack a full cover into one better-compressed pack. Covers double as a skip list, which is what makes pack selection logarithmic. Packs at every level are kept and served; none are garbage-collected.
Pando servers never apply packs or repack, so they mostly negotiate and select packs on commodity hardware. Scaling out means adding Pando servers and growing the metadata and blob stores alongside them.
Bottlenecks and how each was handled
| Bottleneck | What Pando does | Reported result |
|---|---|---|
| One coordination point (a repo-wide snapshot) serializes unrelated pushes | Causal consistency: per-ref versions, CAS on the ref pointer only, O(1) metadata per push | Median push latency down nearly 80%; "many thousands" of concurrent pushers without slowdown |
| Building a unique pack per client on fetch | Reuse stored packs and return them as packfile URIs | 57 TB served in one peak hour, over 99% of bytes from local disk cache |
| Ref advertisement over 100 MiB with millions of branches | Advertise only a chosen subset; new feature branches see only trunk | Advertisement size stays flat as branches grow |
| Pushed packs that can't be reused by other readers | Pack depends only on what the server advertised, so anyone holding trunk fetches the branch in one pack | Median pushed pack under 20 KiB, p99 under 3 MiB |
| Tens of thousands of small packs a day to catch up on trunk | LSM-style compaction into geometric covers (L0 to L4); covers act as a skip list | Pack selection is O(log Δ) in the version gap |
| Replicas applying and repacking every push | Servers never unpack or repack; they only negotiate and select packs | About 1 PB/day on "a handful of CPU cores" |
| Delivering bytes to many regions | Express regional caches with consistent hashing; separate metadata cache | Clones and fetches saturate 10 Gbps links on standard cloud VMs |
| Client edge cases with reused packs (concurrent downloads, interrupted or resumed transfers) | Fixes sent upstream; parallel download and indexing of independent packs; tracing and byte counts for packfile-uri | Several changes already in upstream Git |
--force-with-lease needs the branch tip, but the pack should be built against trunk only | "Explicit haves": the server advertises the branch tip separately from the history the pusher may omit | Lease check and reusable trunk-based packs both work |
Tradeoffs
The central bet is giving up a repository-wide snapshot to get independent, parallel branch updates.
No repo-wide snapshot: updates to one ref are serializable, but there is no point-in-time view of all refs during a clone or fetch. If
fooandbarboth move mid-clone, one client can see foo v1 with bar v2 while another sees foo v2 with bar v1. OpenAI's bet is that users and agents only rely on the order of updates to a single ref.Reused packs carry redundant bytes: a client may download objects it already has. In exchange, the server sends the same bytes to many clients with almost no computation.
Trunk-only advertisement inflates some pushes: pushing one branch many times, or stacking branches, resends everything outside trunk each time, costing client bandwidth.
Storage for every compaction level: packs at all levels are kept and served, never garbage-collected. Uncompacted L0 packs are cheaper for reading single objects.
Cache dependence: the read path assumes heavy overlap between requests. That holds for one busy monorepo; it may not for many cold repositories.
Custom client: the best results need OpenAI's own Git build (shipped to developer machines and in the ChatGPT desktop app) until all patches land upstream.
How it was built, verified and rolled out
One engineer owned the core design, Codex agents wrote most of the implementation, and a handful of engineers deployed it. It went from kickoff to production in about 10 weeks and has carried production traffic since April 2026. It is a greenfield Rust codebase with no legacy dependencies.
Because agents wrote the code, correctness rested on layered checks rather than review alone:
A Git conformance suite so agents could test against standard Git behavior
Deterministic simulation testing of invariants, which caught race conditions and regressions in negotiation and pack planning
A formal "theory of Pando" in Lean and TLA+, written by agents using GPT-6 Astra, used to find and fix bugs and prove properties
Long-running agent sessions (hundreds of hours) fixing edge cases, plus round-the-clock agent red-teaming of Git and Pando
Rollout without downtime. Pando first ran as a read- and write-through proxy in front of the old backend, keeping both in sync. It deduplicated and combined requests before proxying, which made the old system healthier during the move. Clients moved in phases through an internal wrapper around the Git CLI, and OpenAI migrated hundreds of integrations that used legacy APIs, webhooks and repository downloads. Codex skills let developers ask Codex to migrate their own checkouts and services.
OpenAI is sending client fixes upstream and exploring ways to share Pando, including for many agents collaborating on one repository. No product or open-source release is announced.
Reusable lessons
Plan for agent fan-out. One push becomes several CI and agent fetches of the same commit. Size the read path for the write rate times the fan-out, not for human traffic.
Question the guarantees, not just the implementation. 3PC and a shared WAL both failed because they kept a repo-wide snapshot. Most workflows only need ordering per branch, and narrowing the guarantee removed the bottleneck.
Shrink the contended state to one key. Make writes immutable appends, flip a single pointer with CAS, and push everything else (compaction scheduling, extra commit mappings) to async work.
Separate negotiation from byte delivery. Keep the protocol exchange small and move bulk bytes over plain HTTP so regional caches can carry them.
Serve reusable artifacts, not bespoke responses. Accepting some redundant bytes in exchange for cache hits and near-zero server CPU was the winning trade.
Control what clients may omit. Advertising only trunk made every pushed pack reusable by any reader. What a server advertises shapes what it can cache.
Stay protocol-compatible and fix the client upstream. Zero workflow change made adoption possible; the client patches unlocked the last gains.
Migrate behind a proxy. A read- and write-through proxy in front of the old system allowed a no-downtime cutover and relieved the old system while it lasted.
When agents write the code, invest in verification. Conformance suites, deterministic simulation and formal specs made a 10-week rewrite of critical infrastructure safe to ship.
Instrument the phases. OpenAI added tracing to tell pack generation from transfer time. Negotiation, transfer, indexing and checkout are separate costs, and a median hides the tail.
Open questions
What are the real throughput limits: pushes per second, and behavior with many writers on one hot ref such as trunk? The article's 1,000-writer simulation is labeled a synthetic teaching model, and the "many thousands of pushers" claim comes without numbers. Commenters on Hacker News and Lobsters raised the same gap.
Does Pando support atomic multi-ref pushes?
What are p99 push and fetch latencies and failure rates? Only medians are given.
What does keeping every compaction level cost in storage?
Sources
Scaling Git at OpenAI, Aaron Friel, Taylor Blau and Ted Nyman, OpenAI, 2026-11-02 (primary)
Scaling Git at OpenAI: Pando Explained, Kingy AI
Scaling Git at OpenAI, daily.dev (includes community reaction)