What the Engineering Evidence Says About AI Coding Value

Key takeaways

• The best-controlled studies of AI coding tools disagree, and the disagreement is informative. They measure different things on different tasks with different developers.
• Faster completion of a defined task, higher delivery throughput and a developer's sense of being more productive are three separate claims. A business case should say which one it rests on.
• The one large industry survey that links AI adoption to delivery outcomes found small estimated falls in throughput and stability in 2024.
• Measurement is getting harder, not easier, because the developers most enthusiastic about AI are the least willing to work without it.
• For a CFO or CIO: ask for measured delivery outcomes, not usage or self-reported time saved.

Three claims that get blended

When a vendor or a team says an AI coding tool "makes developers more productive", the statement can mean three things.

1. Task speed. A developer completes a defined task faster.
2. Delivery performance. The team ships software more often and more reliably.
3. Perceived productivity. Developers feel faster or more effective.

Task speed, delivery performance and business value sit on different rungs. A gain on a lower rung is not evidence for a higher one.

These can all be true, all be false, or point in different directions. Value to an organisation sits mostly in the second. The evidence below addresses each in turn.

Task speed: two controlled studies, two answers

Peng and colleagues, 2023. Researchers ran a controlled experiment with GitHub Copilot. Recruited developers were asked to implement an HTTP server in JavaScript as quickly as possible. The group with the AI pair programmer completed the task 55.8% faster than the control group.

METR, July 2025. METR ran a randomised controlled trial with 16 experienced developers working on large open-source repositories they had contributed to for years. Developers submitted 246 real issues, each randomly assigned to allow or prohibit AI tools. With AI allowed, tasks took 19% longer. Developers had expected AI to speed them up by 24%, and even after the study still believed it had sped them up by about 20%.

The two studies are not in conflict. One tested a short, defined task from scratch. The other tested open-ended work in a large, familiar codebase. A speed-up claim without a task description is close to meaningless.

Perception is not measurement

The METR result contains a second finding that matters more for management: the developers' belief about their own speed was wrong in direction. That does not mean self-reported time savings are worthless. It means they are a hypothesis, and the evidence needed to confirm them is a measured outcome.

Delivery performance: the DORA findings, 2024 and 2025

The 2024 DORA report (Google Cloud, October 2024) surveyed software teams about AI adoption. More than 75% of respondents said they rely on AI for at least one daily professional responsibility, and more than a third reported moderate to extreme productivity increases. A 25% increase in AI adoption was associated with a 7.5% increase in documentation quality, a 3.4% increase in code quality and a 3.1% increase in code review speed.

At the delivery level the report found an estimated decrease in delivery throughput of 1.5% and an estimated decrease in delivery stability of 7.2% as AI adoption increased. 39% of respondents reported little to no trust in AI-generated code. DORA's reading was that improving the development process does not automatically improve software delivery without adherence to the basics.

The 2025 DORA report, State of AI-assisted Software Development (Google Cloud, September 2025, about 5,000 respondents), moved the picture on. AI adoption rose to 90%, more than four in five respondents said AI had raised their productivity, and AI adoption was now associated with higher delivery throughput, a reversal of the 2024 finding. Making sure software works as intended before it reaches users remained the open challenge, and DORA described AI as "a mirror and a multiplier": it amplifies teams that already work well and exposes weaknesses in those that do not.

This is where the idea of a downstream constraint applies. If AI makes code generation faster but changes are larger or harder to review, the bottleneck can move to review, testing and release, and delivery measures can fall even as individual output rises. That is our interpretation, not a finding of the report. See The AI Abundance Paradox.

Measurement is getting harder

METR tried to repeat its study in late 2025 with a larger group and newer tools. In February 2026 it reported that the new data gave an unreliable signal. Its reasons: more developers declined to take part because they did not want to work without AI, and 30% to 50% said they chose not to submit some tasks for the same reason. Both effects remove the cases where AI helps most.

The raw estimates were a speed-up of 18% for returning developers (confidence interval from 38% faster to 9% slower) and 4% for new developers (15% faster to 9% slower). METR said it believes developers are likely more sped up in early 2026 than in early 2025, but that its data is only very weak evidence for the size of the change.

For a buyer, the implication is uncomfortable. The most credible kind of evidence, a randomised comparison, is becoming difficult to collect, so organisations will have to design their own measurement.

What a finance leader should do

1. Ask which claim. Task speed, delivery performance or perceived productivity? Only the middle one is a business outcome.
2. Ask for delivery measures. Lead time, deployment frequency, change failure rate and recovery time, before and after, with the comparison group stated.
3. Treat self-reported time saved as capacity released, not value. It becomes value when throughput rises, work is redeployed or cost leaves the base.
4. Count the human load. Include review and rework time, which tends to sit outside the tool invoice. See The Cost of Verification.
5. Use an evidence level. A before-and-after comparison is at best an early operational case. A matched comparison or holdout is needed to support a material investment.

Sources: Peng, Kalliamvakou, Cihon and Demirer (2023); METR (July 2025 and February 2026); DORA, Accelerate State of DevOps 2024; DORA, State of AI-assisted Software Development 2025.