806 Repositories, One Causal Design: the Speed Was Temporary, the Complexity Wasn't

A product manager I worked with years ago measured the team by counting bullets in the release notes. Not commits, not revenue, not retention. Bullets. A long release note was a good month and a short one was a bad month, and every engineer on that team understood the scoring system within about a week of it going up on the wall.

They meant well. They were trying to move as much value to customers as possible, and features shipped felt like a fair stand-in for value delivered. So the team shipped features. They crammed the interface until it was genuinely confusing to use, and they burned months building things nobody had asked for while the things customers kept asking for sat untouched. When the product underperformed anyway, the PM leaned harder on the number, started suspecting engineering of slacking, and the pressure between product and engineering eventually broke the team.

What settled it was value stream mapping and then instrumentation. The mapping surfaced the bottleneck, which turned out to be sitting in product rather than in engineering. Then we added logging to see feature use at a granular level, tied that to hours spent building and bug-fixing, and found that the majority of engineering time was going into features that were barely being touched. Month after month the team had hit its number while active usage declined.

I liked that PM, for what it's worth. They weren't gaming anything. They corrupted the metric by taking it seriously, which is a sadder and much more common failure than cynicism.

I've thought about them repeatedly over the past few weeks, reading through the current round of AI-coding-productivity numbers, because the shape is the same: a team measures what shows up in the first month and it turns out not to be the thing that mattered.

Why a difference-in-differences design settles what vendor telemetry couldn't

The reason to read He et al., Speed at the Cost of Quality is the design. Five researchers at Carnegie Mellon identified 806 GitHub repositories with at least ten stars that adopted the modern Cursor client, inferring adoption from the presence of .cursorrules or .cursor files, and matched them against 1,380 control repositories using propensity score matching. Then they pulled monthly GitHub and SonarQube data from January 2024 through August 2025, with most of the adoptions landing between August 2024 and March 2025, and ran a difference-in-differences estimator built for staggered adoption.

That last choice is what separates this from every before-and-after chart you've been shown. A comparison of adopters against themselves would be worthless here, because 2024 and 2025 were not quiet years in software and everything moved at once. Difference-in-differences compares the change in the adopting repositories against the change in similar repositories that didn't adopt over the same months, and because these teams adopted at different times, the not-yet-adopted projects serve as controls for the ones that already had. It's the same logic health economists use on a policy that rolls out state by state.

Every claim in this space before it rested on correlational data, and a lot of that data belonged to the vendor selling the tool. Correlational vendor telemetry is easy to wave off, and people did wave it off, reasonably. This one is harder to dismiss: it's a quasi-experimental causal design, it was accepted at the 23rd International Conference on Mining Software Repositories in Rio on 13 and 14 April 2026, and there's a replication package on Zenodo if you'd rather check the arithmetic than trust the abstract. I'm citing the preprint and the CMU author copy directly rather than the write-ups, because the numbers drift in the retellings.

This is not a fresh result, either. The preprint went up in November 2025, the conference was in April, and the measurement window closed last August. What's new is that people are finally arguing about it.

What the causal evidence says about AI coding tools and technical debt

Both halves of the finding are real, and they have very different half-lives. Velocity jumped: projects saw a 3 to 5 times increase in lines added in the first adoption month. Averaged across the whole window the effect settles at about 28.58% more lines added and 2.63% more commits, and the authors are explicit that the gains dissipate after two months.

Quality moved the other way and stayed there. Static analysis warnings rose 30.3% and code complexity rose 41.6%. A secondary reading of the paper's results table puts the baseline complexity increase at 9% once you control for codebase size and how active the project was, with warnings up across 16 of the 20 categories tracked and the biggest spikes in naming conventions and code hygiene. Those figures are the summary's reading, not lines I pulled out of the paper myself.

One result cuts against the prevailing story. He et al. found no statistically detectable change in duplicated line density, which sits awkwardly next to the widely-cited industry figures showing large jumps in code duplication over roughly the same period. Those two findings disagree. I'm not going to blend them into a single tidy narrative about copy-paste code, because one of them comes from a matched causal design and the other doesn't, and pretending they point the same direction is how a weaker number borrows credibility from a stronger one.

So the useful finding here isn't the direction, it's the asymmetry. The benefit decays inside a quarter and the cost persists past the end of the measurement window, which is the productivity paradox I've written about before, now with a control group attached.

Why the velocity gains disappear after about two months

The paper describes a self-reinforcing cycle, and it's mechanical rather than mysterious. Adoption raises velocity, higher velocity grows the codebase faster, and the bigger and messier codebase accumulates warnings and complexity that somebody has to work around. Then that accumulated debt suppresses the velocity that produced it, which is why month three can look like month zero even though the tool is still installed and everybody is still using it.

The authors put it as a cycle "where initial productivity surges give way to maintenance burdens," and they're careful to say the results "challenge both unbridled optimism and categorical pessimism." That's the honest read. Nobody's code got worse in some moral sense. The team just spent its speed buying something it now has to carry, and the carrying happens quietly, which is part of why the cleanup doesn't happen on its own.

What the authors say they can't tell you

The limits are stated in the paper, and they're worth repeating because they're what separates an honest read from a hype read. This is quasi-experimental, not a randomized rollout, so "causes" here means the best inference a matched design supports rather than the certainty of an experiment. Adoption is inferred from config files, which means a repository can carry a .cursor directory and barely use the thing, or use it heavily without ever committing one. Quality is measured through static-analysis proxies rather than defects that reached a user.

The sample is open-source projects with voluntary participation and limited code review, so a team with mandatory QA gates and enforced review may land somewhere else entirely. The authors also can't separate poor prompting from tool limitations. And 345 repositories showed GitHub Copilot traces across both groups, so this isn't a clean AI-versus-no-AI comparison. It's Cursor layered on top of whatever assistance was already in the room.

None of that undoes the result. It bounds it, which is different, and a bounded finding you can act on beats an unbounded one you can't.

What to measure instead of first-month throughput

I treat every measure as the test of a hypothesis, never as a target to hit. That's my whole answer to the release-notes story. You start from what you believe creates value, form a hypothesis, decide what you're actually trying to achieve, and only then pick the signals that would tell you whether you were right. When a number comes in badly, that's the most useful thing a metric can hand you: evidence that your picture of the world was wrong, and an invitation to go repair the picture instead of flogging the number.

Applied here, the hypothesis behind an AI tooling investment is usually something like "our teams will deliver more of what customers need per unit of engineering time." First-month lines added does not test that hypothesis. It tests whether the tool is installed. A quarterly ROI memo built on the first month isn't lying about the gain, and the gain is real. It's just measuring the half that expires, which is how a tooling decision turns into a Cheap Yes whose bill arrives two quarters later as an unexplained slowdown. That's also where the constraint moved for most of the teams I talk to, and it's why I keep coming back to epistemic debt as the more useful frame than technical debt alone. The messy code is the visible part. The expensive part is code nobody left on the team can explain.

Everybody asks whether to adopt AI tooling; almost nobody asks how much

The dose question is the one with an actual answer in this paper, and it's buried in the least-quoted section. He et al. ran a dynamic panel GMM estimation, which is a way of asking how last month's complexity affects this month's output, and it gives you numbers you can put in a spreadsheet.

A 100% increase in code complexity causes a 64.5% decrease in development velocity as measured by lines added. A 100% increase in static analysis warnings causes a 50.3% decrease. Set those against Cursor's own contribution, which the same estimation puts at a 1.84 times baseline increase in lines added, and you get a break-even point: the productivity gain is fully cancelled out by roughly a 3 times increase in code complexity, or roughly a 5 times increase in static analysis warnings.

That's a threshold a team can locate itself against this week, because both inputs already exist in any SonarQube-class dashboard. Pull your aggregate complexity and your total warning count for the month before your first AI-assisted commit landed. Pull the same two numbers for last month. The ratio between them is your dose reading, and roughly 3 times on complexity or 5 times on warnings is where the arithmetic says the speed you bought has been fully spent. If you're nowhere near that after a year of heavy use, you're fine, and you now know it rather than assuming it. If you're most of the way there and still climbing, you have a quarter or two of runway before the tool stops paying for itself, and you know that too.

I've never been the one who opens the file six months later and finds out what we actually built. I've been the one standing next to that person while they find out, which is its own education, and it's why I want this on a dashboard instead of living in somebody's private sense of dread.

The gain shows up in the first month whether or not you measure it. The cost only shows up if you do.

If you're trying to work out whether your own tooling investment is paying, email matthew@fieldway.org.

Sources

• He, Miller, Agarwal, Kästner & Vasilescu, Speed at the Cost of Quality, arXiv:2511.04427 – arxiv.org/html/2511.04427v3
• CMU author copy (MSR '26) – cs.cmu.edu/~ckaestne/pdf/msr26.pdf
• Zenodo replication package – zenodo.org/records/18368662
• TestCollab, Why AI Coding Tools Create More Bugs Than They Fix – testcollab.com/blog/hidden-cost-of-vibe-coding