AI might actually be slowing down software developers, new research suggests, despite their own positive predictions that AI coding tools will help their work.
Model Evaluation & Threat Research (METR) ran a randomized, controlled trial to study the use of early-2025 AI tools used by experienced developers working on their own open source repositories.
The results weren't positive, with the study finding that developers take 19% longer on average when using AI tools.
That said, METR noted this is a single snapshot of AI capabilities in a single setting — results may vary, METR's essentially saying — and the company hopes to use the figures as a baseline to track and estimate AI acceleration and adoption.
Experienced developers aren’t seeing benefits
METR's study used 16 developers with at least five years of experience who provided a list of 246 issues, such as bug fixes or wished-for features, that would make up their real day-to-day work. The developers in the study largely used Cursor Pro and Claude Sonnet.
"When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts," the post noted.
"This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%."
The METR team listed a few factors they believe likely contributed to the slowdown — the first of which was simply being over-optimistic about the usefulness of AI.
Beyond that, the study showed developers accepted fewer than 44% of AI generated code, with most of those taking part in the study saying they had to make major changes to clean up code. All told, roughly 9% of their time was spent reviewing or cleaning AI output.
Developers reported that AI appeared to struggle in large, complex environments, while the human coders themselves had so much experience in their areas of expertise that the AI couldn't keep up, an issue exacerbated by the fact that it "doesn't utilize important tacit knowledge or context."
The researchers also spotted a few potential causes for the AI slowdown, though they stressed the impact was unclear. That included developers using the AI to experiment and waiting on AI to generate code.
The researchers stressed that the results don't mean that AI is never useful. "Furthermore, these results do not imply that future models will not speed up developers in this exact setting — this is a salient possibility given the rapid pace of progress in AI capabilities recently," the paper said.
"Finally, it remains possible that further improvements to current AI systems (e.g. better prompting/agent scaffolding, or domain-specific finetuning) could yield positive speed up in this setting."
Does AI help or hinder?
Studying real-world impact is necessary, METR said, as AI benchmarks are often unrealistic or can't be easily applied to real-world tasks.
"While coding/agentic benchmarks have proven useful for understanding AI capabilities, they typically sacrifice realism for scale and efficiency—the tasks are self-contained, don’t require prior context to understand, and use algorithmic evaluation that doesn’t capture many important capabilities," the post noted.
"These properties may lead benchmarks to overestimate AI capabilities."
The METR study follows research by Atlassian that suggested AI tools are saving 86% of developers a full day of work each week, letting them focus on improving code quality or creating new features.
However, half of those surveyed said they are losing that AI-gained time because of "organisational inefficiencies" — essentially, bad management is eating up any savings.
"This pressure-cooked mix of innovation and strain demands a closer look at how AI is reshaping the developer experience, and what that means for the future of software development across the industry,” said Atlassian CTO Rajeev Rajan.
Make sure to follow ITPro on Google News to keep tabs on all our latest news, analysis, and reviews.
MORE FROM ITPRO
- CEOs think workers are becoming hostile to AI tools, but they’re pushing ahead with adoption anyway
- AI coding tools are finally delivering results for enterprises
- Big tech promised developers productivity gains with AI tools – now they’re being rendered obsolete
-
Let’s talk about digital sovereigntyIn the age of AI and cloud, where data resides is a key consideration
-
Lack of visibility creates "cascade" of security risk, says KiteworksNews Organizations that don't keep track of data breaches, shadow AI, and third-party counts face dramatically worse outcomes across every metric
-
Microsoft touts new Copilot features in Excel, but says you shouldn’t use them if you want accurate resultsNews Microsoft has warned against using new AI features in Excel for “tasks with legal, regulatory, or compliance implications” – so when can you use it?
-
Senior developers are all in on vibe coding, but junior staff lack the experience to spot critical flawsNews Experienced developers are far more confident in using AI-generated code
-
Hexaware partners with Replit to take secure 'vibe coding' to the enterpriseNews The new collaboration enables business teams to create secure, production-grade applications without the need for traditional coding skills
-
Microsoft says AI is finally having a 'meaningful impact' on developer productivity – and 80% 'would be sad if they could no longer use it'News Researchers at Microsoft wanted to demystify how AI is being used by software developers – their findings show the benefits are finally becoming clear.
-
AWS CEO Matt Garman just said what everyone is thinking about AI replacing software developersNews Junior developers aren’t going anywhere, according to AWS CEO Matt Garman
-
Google's new Jules coding agent is free to use for anyone – and it just got a big update to prevent bad code outputNews Jules came out of beta and launched publicly earlier this month, but it's already had a big update aimed at improving code quality and safety.
-
GitHub just launched a new 'mission control center' for developers to delegate tasks to AI coding agentsNews The new pop-up tool from GitHub means developers need not "break their flow" to hand tasks to AI agents
-
Using an older version of Python? You’re leaving ‘money and performance on the table’ if you don’t upgrade – and missing out on big developer efficiency gains
News New research from JetBrains shows a majority of enterprises are using a version of Python that’s a year or more older – and it's having a big impact on efficiency and performance.
