Restarting a nuclear plant for a math error
The tech industry justifies staggering physical and legal costs to build artificial general intelligence, but its own math proves the technology is stalling.
By Ida Vann
Sparked by The Economics of Recursive Self-Improvement [pdf] · discussion
I spent my weekend doing what I usually do: sitting at my desk, drinking too much coffee, and staring incredulously at a dense, 40-page technical paper on recursive self-improvement so that you don't have to. Outside my window, the broader tech industry continues to go completely feral over the impending arrival of artificial general intelligence, aggressively demanding that we reconfigure our physical reality to accommodate their server racks. To justify the staggering legal and physical costs currently being extracted from the public, they point to this concept of an incoming intelligence explosion. We are being asked, as a society, to maintain a rather grim accounting ledger where the ends supposedly justify the means.
On the debit side of this ledger, the entries are concrete, documented, and devastating. We have Microsoft arranging to restart the Three Mile Island nuclear plant to feed its data centers (a historically fraught location for a battery pack, to put it mildly). We have tech giants draining a 500-milliliter bottle of water every time you ask ChatGPT a series of prompts, actively stressing municipal water supplies in drought-prone areas. And we have OpenAI executives standing before the UK Parliament to argue that it is fundamentally impossible to build their products without committing mass copyright infringement, openly demanding absolute legal immunity to ingest the entirety of recorded human knowledge.
Now, if you ask the more grounded machine learning researchers about the progress of large language models, they will point out that scaling laws have historically held true. They have a valid point. Expanding neural networks and throwing exponentially more computing power at them over the last decade has yielded remarkable, undeniable leaps in capability. The technology is genuinely useful for parsing massive datasets or drafting boilerplate text. But the entire trillion-dollar regulatory defense of the AI industry relies heavily on AGI being an unstoppable, imminent force of nature that will eventually cure cancer and solve global warming. If the machine is destined to save humanity, breaking a few eggs—and the national electrical grid—is framed as a justified sacrifice. If, however, it turns out we are just boiling rivers for a slightly better autocomplete, these externalities transform from an unfortunate side effect into a monumental scandal.
So, let us look at the credit side of the ledger. Let us look at their own math.
The 40-page PDF I spent my Saturday squinting at attempts to mathematically quantify the core mechanics of "recursive self-improvement," or RSI. If you aren't familiar with the term, recursive self-improvement is the foundational theology of the AGI boom. It describes a theoretical scenario where an artificial intelligence system becomes capable enough to write the underlying code for a slightly better artificial intelligence system. That successor model then builds an even better one, triggering an exponential runaway reaction known as an intelligence explosion.
The paper sets out to prove that this phenomenon is mathematically viable by mapping the model's ability to compound its own capability (yes, they are genuinely using economic scaling metrics to measure hypothetical sentience), essentially tracking how much smarter a network gets when you feed it the synthetic training outputs of its immediate predecessor. To make their case, the authors provide pages of dense Greek letters, algorithmic proofs, and back-of-the-envelope calculations mapping the exact mathematical threshold required for this intelligence explosion to achieve escape velocity. For the authors, autonomous machine cognition is simply a rigid engineering equation waiting to be balanced.
Translating their core mathematical equation into plain English reveals a remarkably simple, unforgiving threshold: for a model to successfully enter a runaway recursive self-improvement loop, each new generation must improve upon the previous one by a minimum of 15 percent. Anything less than that 15 percent baseline, and the system eventually flatlines, suffocating on the degrading quality of its own synthetic data. The paper then takes this required 15 percent baseline and plots it against the actual, real-world data of current state-of-the-art models attempting to optimize themselves, revealing that these models are currently clocking in at an improvement rate of roughly 9 percent per generation. These systems are charting a painfully standard curve of diminishing returns, and the machine simply runs out of statistical steam.
That is a brick wall.
It is hardly shocking to learn that the theoretical foundation of the current tech bubble relies on mathematically bankrupt vaporware. You do not even have to look outside the software engineering echo chamber to find deep, structural skepticism about the narrative being peddled to regulators and investors. Over in a massive Hacker News thread calling out flimsy survey data and diminishing returns, the actual developers being asked to build this future are thoroughly unconvinced.
They correctly point out that the paper's optimistic projections rely heavily on incredibly noisy benchmarks, and that historically, "computers improving computers" never resulted in sudden, uncontainable magic. We have had code compilers optimizing other code compilers for decades, and at no point did the recursive process inadvertently spawn a digital deity. The thread is a fascinating masterclass in watching seasoned software engineers look at the soaring promise of an unconstrained intelligence explosion and diagnose it as an absolute shitshow of napkin math. The people closest to the metal understand perfectly well that physical computing constraints and algorithmic degradation represent hard limits, entirely immune to being waved away by a charismatic CEO.
Which brings us back to our ledger, and the breathtaking hubris of the corporate executives demanding we foot the bill for their failing equations. Microsoft and Constellation Energy are planning to restart a dormant nuclear reactor precisely because they have successfully convinced themselves, their venture capital backers, and pliant regulators that the intelligence explosion is an inevitable scientific fact. The industry is actively attempting to muscle through Microsoft's $100 billion "Stargate" project—a dizzying data center temple designed to house millions of specialized GPUs for a machine god that their own internal equations prove is not coming.
They are demanding total copyright immunity, the wholesale enclosure of the open web, and the unchecked extraction of natural resources to fuel an engine that is demonstrably stalling out. The immense disconnect between the staggering physical costs being extracted in the real world and the mundane mathematical reality buried in a 40-page technical paper amounts to a multi-billion-dollar grift, one predicated on keeping the public too intimidated by the jargon to check the underlying arithmetic.
The tech elite would very much like us to believe that resistance to their resource extraction is futile because artificial general intelligence is simply a law of physics playing out in real time. We are supposed to feel small, like inevitable victims of progress who must politely endure the disruption of our electrical grids and the theft of our creative output. But the math does not support the mythology. We do not have to accept the strip-mining of the open web and the boiling of our rivers as the unavoidable price of a utopian future that only exists in a VC pitch deck. We can simply look at the math, say no, and shut off the tap.