GitHub says Copilot improves code quality – but are AI coding tools actually producing results for developers?
Questions over the true impact AI coding tools continue to linger


Software development has frequently been identified as an area ripe for improvement through generative AI adoption, but a recent study has challenged perceptions around how beneficial AI coding tools are for developers.
The data science team at software development specialists Uplevel looked into the impact generative AI coding assistants are having on the efficiency and efficacy of developers.
The investigation found their performance has largely remained unchanged regardless of whether they are using AI tools or not.
Uplevel’s findings counter claims made by GitHub and other industry stakeholders, which have stated generative AI increases code quality and developer efficiency.
Uplevel Data Labs analyzed the performance of 800 developers among its customers, examining the difference in performance between teams with and without access to GitHub Copilot, one of the most popular AI coding assistants.
The research looked at metrics including cycle time, pull request (PR) throughput, bug rate, and extended working hours (also referred to as ‘always on’ time).
Uplevel found that, overall, Copilot provided no significant change in efficiency metrics.
Get the ITPro daily newsletter
Sign up today and you will receive a free copy of our Future Focus 2025 report - the leading guidance on AI, cybersecurity and other IT challenges as per 700+ senior executives
Comparing the throughput, cycle time, and complexity of pull requests, including PRs with associated tests, the researchers found Copilot did not meaningfully help or hinder developers and had no effect on coding speed.
There was some change in certain metrics, but the actual changes recorded were deemed ‘inconsequential’. For example, Uplevel recorded a 1.7 minute decrease in average cycle time for developers with access to GitHub Copilot.
This clashes with findings from GitHub, which indicated that in the two years since GitHub Copilot was released to the public, developers were coding 55% faster.
Copilot-assisted code found to contain more errors
The developer platform recently published the results of another study of 202 developers, which found those with access to Copilot had increased functionality, readability, and overall code quality.
For example, GitHub found readability of code written with the assistance of Copilot improved by 3.62%, with similar increases in reliability (2.94%), maintainability (2.47%), and conciseness (4.16%).
GitHub said these numbers were statistically significant, but the findings from Uplevel’s investigation revealed more concerning effects of AI coding tools on code quality.
The firm found that while throughput stayed at around the same level code quality actually decreased, with significantly higher bug rates.
RELATED WHITEPAPER
The bug rate in code generated by those using Copilot was found to have increased by 41%, which when taken alongside the fact that throughput was unchanged suggested that Copilot access may be hurting code quality, according to Uplevel.
Not only did Uplevel’s research indicate generative AI coding tools may not be helping improve developer’s efficiency or efficacy, but the data showed Copilot access was also not particularly effective in mitigating the risk of developer burnout.
Uplevel’s ‘Sustained Always On’ metric, which measures extended working time outside of standard hours – a primary indicator of burnout, was found to have decreased for both groups.
For those with Copilot access, this decrease was 17%, whereas it was measured at 28% for those not using the coding assistant.
Research indicates the appetite for AI coding tools among developers and business leaders has grown significantly in the last year or so, but the jury appears to still be out on whether these solutions are producing tangible value in the software development industry at the moment.
Solomon Klappholz is a former Staff Writer at ITPro adn ChannelPro. He has experience writing about the technologies that facilitate industrial manufacturing which led to him developing a particular interest in IT regulation, industrial infrastructure applications, and machine learning.
-
Security experts issue warning over the rise of 'gray bot' AI web scrapers
News While not malicious, the bots can overwhelm web applications in a way similar to bad actors
By Jane McCallion Published
-
Does speech recognition have a future in business tech?
Once a simple tool for dictation, speech recognition is being revolutionized by AI to improve customer experiences and drive inclusivity in the workforce
By Jonathan Weinberg Published
-
Turns out AI isn't that popular at work – just 4% of workers use the technology in the majority of daily tasks, but developers are among the top early adopters
News Research from Anthropic shows that while AI adoption is sluggish in most professions, software developers and writers are very keen.
By Nicole Kobie Published
-
GitHub's new 'Agent Mode' feature lets AI take the reins for developers
News GitHub has unveiled the launch of 'Agent Mode' - a new agentic AI feature aimed at automating developer activities.
By Ross Kelly Published
-
GitHub just launched a new free tier for its Copilot coding assistant – but only for a select group of developers
News Limited access to GitHub Copilot in VS Code is now available free of charge
By Nicole Kobie Published
-
Are ‘ghost engineers’ stunting productivity in software development? Researchers claim nearly 10% of engineers do "virtually nothing" and are a drain on enterprises
News The study used an algorithm to assess the amount of work being done by software engineers at hundreds of firms
By George Fitzmaurice Published
-
Python just brushed past JavaScript to become the most popular programming language on GitHub – and a key factor is that AI developers love it
News The meteoric rise of Python shows no sign of stopping
By Nicole Kobie Published
-
“There is no one model to rule every scenario”: GitHub will now let developers use AI models from Anthropic, Google, and OpenAI
News Devs will be given access to a broader array of AI models on GitHub – but there's more in store for users
By Emma Woollacott Published
-
Not all software developers are sold on AI coding tools – while productivity gains are welcomed, over a third are concerned about AI-generated code quality
News Many software developers have concerns over the quality and security of AI-generated code despite marked productivity boosts
By George Fitzmaurice Last updated
-
New GitHub rules mean users can store code and repository data in the EU
News The offering from GitHub appears to be a development-focused spin on the sovereign cloud trend
By George Fitzmaurice Published