AI Agents Raised Pull Requests 36%. Shipped Work Didn't Keep Pace.

A Harvard study of 718 companies found AI agents raised code output far faster than shipped work. Why, and what a founder should measure instead.

Matthew Stublefield

6 min read

Man in gray tank top holding gray metal bar

When software teams adopted AI coding agents, the number of pull requests they opened went up 35.7%. The amount of finished, shipped work went up far less, and the researchers who measured it could rule out an increase of more than about 31%.

That's from a Harvard working paper by Fiona Chen and James Stratton, which used engineering data from 718 companies on the Jellyfish analytics platform. It's the most careful look I've seen at the question every founder paying for AI coding tools is asking: if my engineers write more code, do I get more product?

The answer, so far, is some, and much less than the code numbers suggest. If you're judging your team's AI investment by how much code it produces, you're measuring the part of the system that was never the bottleneck.

What the study measured

Chen and Stratton looked at two kinds of tools: AI assistants that suggest code as engineers type, and AI agents that take on larger chunks of work. They compared companies before and after adoption, using the timing differences between companies to separate the effect of the tools from everything else going on.

They measured two things. The first was code output: lines of code added, commits, and pull requests opened. The second was finished work: Jira issues and epics resolved, which in the teams they studied happens once the work has been reviewed, tested, and deployed. An issue is a piece of work; an epic is a larger feature or project made of many issues.

The paper has been revised since it first circulated, and the numbers changed, so it's worth being precise. In the current version, dated November 1, agents raised lines of code by 44%, commits by 36%, and pull requests opened by 35.7%. Resolved issues rose 13.7%, a result the authors describe as borderline significant, and epics 18%, which wasn't statistically significant. For assistants, the effects on finished work were close to zero: 2% on issues and 1.7% on epics, neither significant.

So the agents probably did produce some more finished work. They just produced a lot more code to get there, and the authors' estimates rule out output growing anywhere near as fast as the code did.

Where the extra code goes

The paper's explanation is a familiar one if you've ever run a software team: code review becomes the bottleneck. After AI adoption, code waited longer to reach a reviewer, spent longer in active review, and came back with more requested changes, which the authors read as a sign that the quality of what was being submitted went down.

That matches other data. LinearB's 2026 benchmarks found AI-authored pull requests waited 4.6 times longer before review began. And the bottleneck doesn't stop at review. In a survey of 60 marketers by Appcues and Navattic, 75% said their product is shipping faster than last year, and only 7% said their launch process is keeping up. Work that gets through review still has to reach customers in a way they notice and use.

Writing code has gotten much cheaper. Reviewing it, launching it, and getting customers to adopt it haven't, and neither has deciding what's worth building in the first place.

Why the code numbers feel so convincing

Part of what makes code volume tempting is that it matches how the work feels. James Shore, writing for O'Reilly at the end of October, points to the well-known METR study as a warning: developers in it rated themselves 20% faster with AI, and were measured as 19% slower. Shore is careful to note the study predates the latest models, so it isn't evidence that AI can't help. It's evidence that people's sense of their own speed isn't a measurement.

Shore's advice is blunt: don't measure lines of code, and don't measure pull requests either, because an engineer, or an agent, can make a pull request any size it likes. Once a count like that becomes a target, it stops telling you anything about the work. The Harvard paper is a good illustration of why. The pull request count rose fast. The thing the pull requests were supposed to produce didn't keep up.

I've seen this before, without the AI

None of this is new. AI just made it faster and more visible.

When I came in to help Stride's Career-as-a-Platform team, the 80-person team was five months past its launch date and hadn't shipped a feature to users. Leadership had tried to fix it with more capacity, including expanding the offshore engineering contract to 60 people, and every attempt made things slower and more expensive. Tickets averaged four to six weeks of cycle time while containing only three or four days of actual work. When I looked at the code repository, only a handful of the 60 engineers were contributing meaningful code, and just two people were doing the bulk of the code review.

Adding engineering capacity had the same effect then that adding AI agents has now: more work entered a system whose real constraint was somewhere else. At Stride, the constraint was upstream – engineers were coding before anyone understood the requirements – and in the review queue downstream. Fixing those is what eventually tripled the team's throughput, with the same engineers writing code at the same speed. I wrote about what that took.

What to measure instead

If you're paying for AI coding tools, and especially if you're under pressure to show a return on them, code volume is the most tempting number to report and the least informative. A few better ones:

Finished work, not started work. Count what reaches customers – resolved issues, shipped features, releases – not pull requests or commits. If code output is rising much faster than finished work, the gap is your bottleneck, and it's telling you where to look.

Time waiting for review. How long does a pull request sit before anyone looks at it? If that number is climbing, more code generation will make it worse.

What customers actually use. A feature that ships and goes unused cost you the same engineering time as one that matters. Features shipping while metrics stay flat is the most expensive version of this problem.

How many things you decided not to build. If building is cheaper, the discipline of choosing what to build matters more, not less. A team that ships more of the wrong things faster is moving backwards with better velocity.

The question that didn't get cheaper

A cheap yes leads to an expensive no, and AI has made the yes cheaper than ever. When anything can be built in an afternoon, it's tempting to build everything, and the cost shows up later as review queues, launch debt, and features nobody uses.

The Harvard paper is careful to say its measures are proxies, and that some gains may not show up in Jira counts. That's fair. But the direction is clear, and it matches what I've seen in teams with and without AI: the work that limits a software company is usually deciding what's worth building and getting it through to customers, not typing it. Tools that make the typing faster are useful. They don't touch the part that decides whether any of it was worth doing.

–Matthew

More Field Notes

  1. Consulting Operations

    I Cancelled the Test Before We Built It. The Math Said It Couldn't Win.

    A test that can't produce a readable result shouldn't be built, and the time to decide what would make you stop is before the results come in.

  2. Consulting Operations

    Competitive Intelligence Just Became the No. 1 Research Priority. Cadence Decides Whether It Lands.

    Demand for competitive intelligence is rising, and three datasets agree on what makes it land: how often it reaches the people deciding.

  3. Consulting Operations

    Before We Built a Resume Builder, We Asked Whether a Resume Was Even the Right Thing to Build

    Before building a resume builder for young job seekers, we asked whether a resume was the right thing to build. The answer changed the product.

Want help running a sharper practice?

The reading and synthesis behind your client work, handled – a living deliverable kept current, so more of your time goes where your name is actually on the line.