2.9 Billion Commits a Month, and Nobody Asked Who Checks Them
GitHub's CTO has posted a postmortem that admits something a hyperscaler rarely says out loud: the growth explains it. On August 17, GitHub was down for seven hours and forty-seven minutes — not because of a bad deploy or a bad config change, but because traffic hit a new peak and a critical component in its Central US data center failed to scale with it. Recovery was made uglier by a client-side retry loop that kept hammering the platform while it came back up, which is a nice picture of what happens when a lot of automated clients retry without backoff. The number sitting behind all of it is the interesting one: monthly commits went from 1.4 billion to 2.9 billion in four months. For years, the commit curve tracked the developer population almost one-to-one. It just stopped. GitHub's own postmortem frames it as a capacity story — 3 million added CPU cores, 120 petabytes of storage, Azure now serving 58% of platform load up from 12% in May — and for GitHub, that is fixable with money. For everyone whose software rides on those commits, it is a different conversation.
The part the postmortem doesn't touch is the check on the claim. Every one of those 2.9 billion commits is an implicit statement that the change works and breaks nothing downstream, and the machinery that verifies that claim still lives in your pipeline, still sized for the day code arrived at human typing speed. Staging is one queue per organization. Full-stack test environments are expensive enough that teams ration them. Review is one pair of eyes at a time. A sponsored piece on The New Stack by Signadot's CEO makes the argument cleanly — generation is machine-paced and exponential, verification is human-paced and nearly flat, and the distance between those curves is the infrastructure problem of the next few years — and it's worth remembering it's sponsored when you read it, but the numbers underneath don't come from him. They come from GitHub. The three ways teams currently handle the widening gap are: buy faster review tooling, throttle the agents, or merge unverified and pay for it later as broken staging and production incidents. The August 17 outage, in miniature, is what the third option looks like at planetary scale.

Here is the asymmetry I keep coming back to. GitHub can buy capacity; cores and storage scale linearly with the budget. You cannot buy a reviewer the same way, and you cannot buy time for the queue — verification is the one part of the stack where the fix is architectural, not financial. The teams that pull ahead over the next year are probably not the ones producing the most commits; they'll be the ones whose verification capacity rises with their generation capacity, so a doubled commit curve means

Comments
Post a Comment