Agents Refactored 26% of Commits. Design Smells Fell by Exactly Zero.

For a few weeks this summer I was fairly sure I wasn't producing anything.

The quiet was structural rather than personal. I was working without a defined objective for my pod, and the initiative tracker in front of me had every column you'd want on it – metric, projected impact, confidence, ease, priority rank – with every single cell blank. That's a normal state for work that's still finding its shape, and it wasn't anyone's failing, but it did mean nobody had yet said what done looked like. The session meant to settle direction hadn't happened. So I kept doing good work into the space where a goal layer should have been, with nothing to land it against.

Then I sat down and took stock of one week.

That week I disproved a metrics panic. An entire engineering thread was chasing a regression that wasn't there, and the anomaly turned out to be a marketing tag that only ever fired on cold page loads. While I was down in the data I found a large year-over-year decline in organic traffic that nobody had looked at yet, which was a much bigger problem than the one I'd been sent after. I filed a relevance bug against a search change before it shipped. I worked a feature's impact metrics against all 42 child stories underneath it and found one flow with no canonical events at all, which meant the conversion path the team had been discussing for weeks was structurally unmeasurable. And I wrote a memo recommending we not build a feature, with the experiment design attached in case anybody wanted to argue about it.

Every item on that list produced an absence: a panic that didn't happen, a bug that never reached production, a feature that didn't get built, a measurement gap found before decisions got stacked on top of it. Not one of them left an artifact anyone can open. It was the most useful week I'd had in months and it felt, from the inside, like nothing at all, because the output of the work was a negative.

I don't have a quick fix for that feeling. I kept a running list in Obsidian mostly so there'd be something to read on the days the feeling got loud.

I've been chewing on that week ever since, because there's a study on AI coding agents that arrives at the same place from the opposite direction.

Agents refactor constantly, and the design problems don't move

The paper is Agentic Refactoring: An Empirical Study of AI Coding Agents by Horikawa et al., built on 15,451 refactoring instances across 12,256 pull requests and 14,998 commits drawn from the AIDev dataset. One limit governs how far any of these numbers travel: this is real-world open-source Java, and only that. Sourcegraph, writing up the same study, says it plainly – those percentages refer to the open-source Java dataset, not to AI refactoring in general. Every percentage in this post inherits that limit.

Agents explicitly target refactoring in 26.1% of commits, 3,907 out of 14,998, containing 7,127 detected refactoring instances. So this isn't a story about neglect. More than one commit in four is an agent deciding, unprompted, that some code ought to be cleaner.

The median change in both design smell count and implementation smell count is 0.00. That reading comes from LinearB's analysis of the Horikawa et al. dataset, which puts it flatly: agents show no reduction in design or implementation smell counts, and the median change in both categories is zero. That isn't a modest improvement, and it isn't a wash hiding big wins and big losses on either side of the middle. Zero, at the middle of the distribution, across a quarter of all commits.

If you'd told me a quarter of the commits in a codebase were refactoring commits, I'd have assumed the smell count went down. It doesn't go down.

The refactoring they do a great deal of is the mechanical kind

Agentic refactoring skews low-level. 35.8% of agent refactorings are low-level operations against 24.4% for human developers, and only 43.0% are high-level against 54.9% for humans. The middle band is effectively a tie at 21.2% versus 20.7%. In the paper's own breakdown the three most common agentic types are Change Variable Type at 11.8%, Rename Parameter at 10.4% and Rename Variable at 8.5%, and those three alone account for 30.7% of all agentic refactoring instances.

The stated motivations line up with that. Maintainability sits behind 52.5% of agentic refactorings against 11.7% for humans, and readability 28.1% against 25.7%. Agents are unusually eager to tidy, and tidying is what they produce.

None of this is damage. The structural effects are real and they point the right way: a median change of -15.25 in class lines of code, and -2.07 in weighted methods per class, according to the study's quality-impact analysis – which I'm reading through a secondary aggregation of the paper rather than the paper's own tables, so treat those two figures as close rather than exact. The same source notes that low-level edits marginally increase cyclomatic complexity, a trade between easier reading and messier control flow. Sourcegraph's summary of the gains is that they were small, "with high-level design decisions remaining with human developers."

Files get shorter, classes get lighter, names get better, and the design problems a senior engineer would name if you handed them the repo are all still sitting exactly where they were.

What they skip is the part that requires knowing what the system is for

Here's the comparison that made me put my coffee down. Agents target code duplication in 1.1% of their refactorings, where human developers target it in 13.7%. Modularity and reuse: 4.6% for agents against 12.9% for humans. That's close to an exact inversion of what experienced developers prioritize, and duplication and modularity happen to be the first two things most engineers name when they say a codebase needs work.

The mechanism isn't mysterious. Renaming a variable needs only the file it lives in. Removing a duplicate means deciding which of the two implementations is the true one, who depends on the other, and whether the difference between them is an accident or a requirement somebody wrote down once and never documented. Pulling a shared module out means having an opinion about what the system is for. That's what the agent can't see, and retrieval isn't the weak link here: the answer lives in decisions that were never written down anywhere an agent could read them. It's the same shape as epistemic debt: the missing thing isn't code, it's knowledge about the code.

There's a second-order problem stacked on top. 53.9% of agent refactorings arrive inside tangled commits, bundled with feature work or bug fixes rather than isolated as refactoring, which makes even the good cleanup hard to review or attribute on its own. If you've been trying to work out why your codebase feels heavier than your velocity chart suggests it should, that's part of it, and it's a cousin of the complexity that persists long after the speed-up has been spent.

The half of the job that's still yours is subtraction

We've delegated addition beautifully and delegated subtraction not at all, and our measurement systems can't tell those two apart. That's the pattern I recognize in this study, and it isn't really a Java problem or an agent problem. I've spent twenty years watching teams get credit for the first one.

That's the part I'd push back on hardest. A tracker that counts merged changes will rate an agent's refactoring quarter above a senior engineer's month spent deleting a subsystem, and it will be wrong every time it does it. The same instrument that undervalued my week undervalues theirs, for the same reason: it can only see what got added.

The split I'd draw isn't complicated. Agents are genuinely good at the consistency layer, and 26.1% of commits is a lot of tidying nobody has to do by hand now. What stays human is naming the thing that should stop existing – the duplicate that shouldn't have two homes, the module that shouldn't be a module, the feature that shouldn't get built. You already have the skill this needs; it's the one you use in review when you say "we already have this somewhere else, don't we." Nobody has automated that sentence, and the numbers say nobody is close.

Most of the Strategy work I do ends the same way, in a recommendation on what not to pursue and why. It's uncomfortable to sell and uncomfortable to sit with, since the deliverable is an argument for a thing that now won't exist. Worth deciding deliberately how much to delegate before the tidying volume convinces anyone the design work is handled, because the hard 20% hasn't moved much.

Two things keep sitting next to each other in my head: a week of work I couldn't point at, and a quarter of all commits that removed no design problems. Both get measured by what isn't in the repo afterward, and absence is hard to see, harder to credit, and still where a lot of the value is.

If you'd like a second read on what your team should stop building, email matthew@fieldway.org.

Sources

• Horikawa et al., Agentic Refactoring: An Empirical Study of AI Coding Agents, arXiv:2511.04824
• LinearB, Here's where AI coding agents are delivering reliable code refactoring
• Sourcegraph, AI Code Refactoring: How to Refactor at Scale
• Emergent Mind, Agentic Pull Requests