Reproducing the SonarSource AI evaluation and the market for lemons

We’re going to look at the methodology of a recent paper on AI coding assistants, SonarSource arXiv:2605.20049, which evaluates model performance on isolated code snippets and concludes that feeding language models code that is clean improves their token efficiency by 8%. The standard setup for verifying this kind of evaluation is to download the researchers' dataset, run the same local unit tests they ran to verify the AI's edits against the prompt, and accept the reported gains, but if we look at a recent independent replication dataset that took that exact evaluation suite and added the one necessary step that the paper omitted—writing a basic script to check whether the global codebase still compiles and passes integration tests after the AI's isolated edit is applied—the underlying economic reality of the current AI tooling market becomes glaringly apparent.

Evaluating this properly requires extracting the exact commit hashes referenced in the paper, applying the AI-generated diffs to the local files, and attempting to coax a successful build out of ten entirely different build systems (including Maven, Gradle, Go modules, and whatever custom python scripting Django uses this week). The independent team that published the replication noted that running these evaluations was tedious because every open-source project has slightly different, poorly documented assumptions about the local environment, which is presumably why the original authors skipped it, but it's the only way to measure what actually happens when organizations accept these edits in a real engineering pipeline.

When the replication team actually ran the full test suites on the edits that the original paper claimed were "successful" based on local metrics, it turned out that 42% of them broke the build by violating some invariant in a completely different file.

This isn't surprising. If your incentive as an AI researcher is to publish a paper claiming an 8% improvement in token efficiency, you're locally optimizing for publication, which means you're fundamentally disincentivized from doing the grueling, low-glamour ops work of verifying whether the edit actually works in reality (because if you did, you'd have to report the massive regression rate and your paper wouldn't get accepted). From a mechanism design point of view, the industry is currently building tools that are extremely good at generating code that looks superficially correct to a human reviewer but is structurally fatal to the repository, and then selling them to companies who evaluate developer velocity by lines of code merged.

That's a tire fire.

Appendix A: Pre-emptive responses to context window arguments

• People on HN will inevitably claim that this 42% regression rate is just a temporary artifact of small context windows, and that throwing a 2M token window at the problem with RAG solves global state awareness.
• If you actually do the math on a 2M token context fill at $3/1M, you are paying 2 * 3 = $6.00 per query to give the model the opportunity to not break the build.
• Even ignoring the economic absurdity of paying six dollars a pop for an autocomplete, empirical testing shows that just dumping 500 files into context degrades needle-in-a-haystack performance, meaning the model still breaks the global state, it just takes 30 seconds longer to do it and costs 3000x as much.