Work Asymmetry, Combinatorial Explosions, and Git Web Interfaces
Git web interfaces fail under load because they invert native work asymmetry, allowing cheap client requests to force unbounded computation onto the server.
By Owen Tate
Sparked by Creepy Crawlies · discussion
There is a furious debate sweeping through my feeds this week regarding the etiquette of AI scrapers. The recent discourse surrounding the melting of git.kernel.org—detailed in the original "Creepy Crawlies" post and the ensuing Hacker News outrage—lays the blame squarely at the feet of malicious bots. Yes, scrapers are wildly impolite. Yes, they ignore robots.txt. But let's put that aside. Politeness is not a scalable distributed systems protocol. If your architecture relies on the benevolence of anonymous internet traffic to maintain uptime, you have already lost the war.
I was running the math on protocol internals recently, and the baseline physics of native Git are genuinely beautiful. It utilizes a want/have negotiation that cleanly separates data transfer from state resolution. When you execute a native git fetch, the server efficiently bundles up compressed packfiles and streams them down the wire. The architectural genius here is deliberate work asymmetry. The server does relatively little computational work, pushing the burden of resolving delta chains and reconstructing tree structures entirely to the edge client. Your laptop burns the CPU cycles. The server merely facilitates the transfer of compressed blocks, maintaining strict control over its own CPU utilization regardless of how complex the resulting merge might be on the client side.
Web frontends completely invert this asymmetry. If we sketch this out on a whiteboard as a two-panel architectural flow, the top panel illustrates the native protocol with a lightweight data stream leaving the server while the client handles the massive computation. The bottom panel illustrates the web interface, where a client—often a scraper bot—sends a microscopic HTTP GET request. This request is usually parameterized to view the changes between two arbitrary commits via git-diff. The server is suddenly on the hook to do all the heavy lifting: unpacking objects, allocating memory, running the diff algorithm, and formatting the output into HTML. A tiny inbound request triggering massive, unbounded backend computation. Oof!
Let's run a simulation to see what happens to this architecture under load. The Linux kernel repository currently sits at roughly 1.48 million commits. If a web API allows a client to request a diff between any two arbitrary commits, we are looking at the standard combinatorial math of choosing 2 from $N$. That equation is $(1.48M \times 1.48M) / 2$, which yields roughly 1.1 trillion possible combinations.
This represents a classic quadratic time complexity problem exposed directly to the public internet. The input space is $O(N^2)$, mathematically bounded only by the relentless growth of the repository itself. Now, imagine a modest scraper pipeline. It doesn't even need to be malicious. It just systematically crawls the commit history with 50 concurrent threads, asking for diffs between random historical points to train a new language model.
On the server side, each of those 50 requests hits a web worker. To satisfy a single request, the worker has to load two distinct directory trees from disk, traverse them, and compute the delta. If the diff is large, this might take a full second of CPU time. Suddenly, 50 concurrent requests mean 50 CPU cores are pegged at 100% utilization. If your server only has 32 cores, you are immediately dropping into a queue. And as we know from queueing theory, when arrival rates exceed service rates, the queue depth doesn't just grow linearly; it spikes exponentially.
In theory, you mitigate this with caching. Projects like cgit explicitly advertise a built-in cache designed to "decrease server io-pressure". That works perfectly for human operators clicking on the latest master branch commit (I’m assuming the most recent changes are the ones everyone wants to see...). But this is where the pristine math of the theoretical input space meets the messy reality of production. In reality, input isn't just a handful of predictable humans reading the daily changelog; it's a barrage of random-walk AI scrapers, retry storms from failing client connections, and thermal noise in the datacenter.
How do you cache an $O(N^2)$ input space when deterministic human behavior is replaced by stochastic machine noise? You don't. The cache hit rate rapidly approaches zero as the scrapers request increasingly obscure historic diffs from 2008. As the cache misses stack up, the system reverts to raw computation for every single request.
Let's apply Little’s Law here. The number of items in our system ($L$) equals the arrival rate ($\lambda$) multiplied by the average time spent in the system ($W$). When scrapers bypass the cache and force heavy computation, $W$ dramatically increases. Even if $\lambda$ remains constant, $L$ explodes. The web nodes saturate their CPU, queuing delays pile up, and the backend storage grinds to a halt under the thundering herd of arbitrary reads. What does this mathematical saturation actually look like in production? An unbounded backlog inevitably triggers a page, waking up an on-call engineer at 3 AM. They are forced to groggily decipher a dashboard painted entirely in 502 Bad Gateway errors, usually with zero visibility into the underlying hypervisor state[^1].
Focusing on bot manners or web server optimization fundamentally misdiagnoses the structural flaw. By exposing a parameterized HTTP endpoint that permits clients to dictate the server's workload without constraints, we stripped away the deterministic boundaries that kept the original native Git protocol ticking along nicely. A system's resilience cannot depend on the benevolence of its callers, because distributed systems inevitably attract actors who will blindly explore every permutation your inputs allow.
We must enforce boundaries at the architecture level, shifting the computational asymmetry back to where it belongs. Any interface that allows a client to spend fractions of a cent on network transit to force mathematically unbounded work on the server is structurally doomed.
Resilience, as always, is a structural property of the system, never a behavioral trait of its users.
[^1]: On the other hand, some sensible people don't support doing math based on vibes, and might object that my usage of Little's Law here assumes a steady state, which a retry storm decidedly is not. They are right, of course, but the resulting server meltdown remains exactly the same.