AI might actually be slowing down software developers, new research suggests, despite their own positive predictions that AI coding tools will help their work.
Model Evaluation & Threat Research (METR) ran a randomized, controlled trial to study the use of early-2025 AI tools used by experienced developers working on their own open source repositories.
The results weren't positive, with the study finding that developers take 19% longer on average when using AI tools.
That said, METR noted this is a single snapshot of AI capabilities in a single setting — results may vary, METR's essentially saying — and the company hopes to use the figures as a baseline to track and estimate AI acceleration and adoption.
Experienced developers aren’t seeing benefits
METR's study used 16 developers with at least five years of experience who provided a list of 246 issues, such as bug fixes or wished-for features, that would make up their real day-to-day work. The developers in the study largely used Cursor Pro and Claude Sonnet.
"When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts," the post noted.
"This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%."
The METR team listed a few factors they believe likely contributed to the slowdown — the first of which was simply being over-optimistic about the usefulness of AI.
Beyond that, the study showed developers accepted fewer than 44% of AI generated code, with most of those taking part in the study saying they had to make major changes to clean up code. All told, roughly 9% of their time was spent reviewing or cleaning AI output.
Developers reported that AI appeared to struggle in large, complex environments, while the human coders themselves had so much experience in their areas of expertise that the AI couldn't keep up, an issue exacerbated by the fact that it "doesn't utilize important tacit knowledge or context."
The researchers also spotted a few potential causes for the AI slowdown, though they stressed the impact was unclear. That included developers using the AI to experiment and waiting on AI to generate code.
The researchers stressed that the results don't mean that AI is never useful. "Furthermore, these results do not imply that future models will not speed up developers in this exact setting — this is a salient possibility given the rapid pace of progress in AI capabilities recently," the paper said.
"Finally, it remains possible that further improvements to current AI systems (e.g. better prompting/agent scaffolding, or domain-specific finetuning) could yield positive speed up in this setting."
Does AI help or hinder?
Studying real-world impact is necessary, METR said, as AI benchmarks are often unrealistic or can't be easily applied to real-world tasks.
"While coding/agentic benchmarks have proven useful for understanding AI capabilities, they typically sacrifice realism for scale and efficiency—the tasks are self-contained, don’t require prior context to understand, and use algorithmic evaluation that doesn’t capture many important capabilities," the post noted.
"These properties may lead benchmarks to overestimate AI capabilities."
The METR study follows research by Atlassian that suggested AI tools are saving 86% of developers a full day of work each week, letting them focus on improving code quality or creating new features.
However, half of those surveyed said they are losing that AI-gained time because of "organisational inefficiencies" — essentially, bad management is eating up any savings.
"This pressure-cooked mix of innovation and strain demands a closer look at how AI is reshaping the developer experience, and what that means for the future of software development across the industry,” said Atlassian CTO Rajeev Rajan.
Make sure to follow ITPro on Google News to keep tabs on all our latest news, analysis, and reviews.
MORE FROM ITPRO
- CEOs think workers are becoming hostile to AI tools, but they’re pushing ahead with adoption anyway
- AI coding tools are finally delivering results for enterprises
- Big tech promised developers productivity gains with AI tools – now they’re being rendered obsolete
-
NIS2: Why are firms struggling to comply?In-depth The Network and Information Systems 2 (NIS2) Directive continues to trip up organizations in critical industries, as leaders grapple with complex supply chains
-
Destination Earth: The digital twin helping to predict – and prevent – climate changeHow the European Commission is hoping to fight back against climate change with a perfect digital twin of our planet, powered by Finland's monstrous LUMI supercomputer
-
OpenAI's plan to acquire AI coding startup Windsurf ended in disaster – here’s how the deal fell apartNews The acquisition by Cognition comes after a rumored $3bn offer from OpenAI fell through
-
Atlassian says AI has created an 'unexpected paradox' for software developers – they're saving over 10 hours a week, but they’re still overworked and losing an equal amount of time due to ‘organizational inefficiencies’News While AI is helping save developers over 10 hours a week, these productivity boosts are being offset by growing workloads and poor operational efficiency, Atlassian says.
-
AI coding tools are booming – and developers in this one country are by far the most frequent usersNews AI coding tools are soaring in popularity worldwide, but developers in one particular country are among the most frequent users.
-
GitHub CEO Thomas Dohmke thinks there’s still a place for junior developers in the age of AINews GitHub CEO Thomas Dohmke believes junior developers still play a crucial role in the hierarchy of software development teams, and AI won't change that any time soon.
-
MCP servers used by developers and 'vibe coders' are riddled with vulnerabilities – here’s what you need to knowNews Security researchers have issued a warning over rampant vulnerabilities found in MCP servers used by developers and 'vibe coders'.
-
Mistral targets security-conscious developers with new AI coding assistantThe coding assistant, available now in private preview, will be fully customizable
-
Big tech promised developers productivity gains with AI tools – now they’re being rendered obsoleteOpinion Big tech promised software developers huge benefits with AI tools, but now they face job cuts as companies ramp up automation.
-
Anthropic’s new AI model could be a game changer for developers: Claude Opus 4 ‘pushes the boundaries in coding’, dramatically outperforms OpenAI’s GPT-4.1, and can code independently for seven hoursNews Claude Opus 4 boasts huge performance capabilities and is fine-tuned for software developers.
