‘Not a shortcut to competence’: Anthropic researchers say AI tools are improving developer productivity – but the technology could ‘inhibit skills formation’

AI tools are helping improve efficiency for software developers, but it may result in many losing key skills in the long-term.

That's according to a pre-print of a study on AI and skills formation conducted by a pair of fellows at Anthropic, which suggests there may be downsides to frequent AI use.

The study comes amid questions about the productivity boost offered by these technologies, as well as their wider impact, be it on jobs or on human creativity and critical thinking. .

The Anthropic paper — authored by fellows Judy Hanwen Shen and Alex Tamkin — sought to unpick the relationship between AI use and skills formation by studying how participants picked up new skills while using the technology in daily tasks.

To test this, they randomized experiments that measure skill formation, giving 51 participants a coding task that used a Python library they hadn't used before, later evaluating how well they learned that library.

The researchers tracked whether using AI boosted productivity for coding tasks that required new concepts or tools and whether that led to participants learning less about those new concepts and tools.

All told, researchers found that the productivity gains afforded by AI were offset by the fact participants lacked a deeper knowledge in core areas. Indeed, the technology could ultimately “inhibit skills formation”.

"Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library," they reported in their paper.

"Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation – particularly in safety-critical domains."

Less time coding, less time learning

Notably, the results suggest AI assistance didn't lead to a statistically significant faster task completion, though those using no AI did take slightly longer to complete tasks.

After the task was completed, participants were given a score to test how well they learned the new library. Those who were assisted by AI posted scores ranging from below 45% to just shy of 60%, while those who didn't use AI all scored above that mark.

The results differed depending on how much coding experience a participant had, however. Novice developers, for example, posted a more significant benefit in performance than their colleagues with four years or more of coding experience.

When it came to quiz scores, those with no AI assistance posted better scores regardless of their level of coding experience.

The researchers suggested that the failure to significantly boost productivity can be pinned on participants spending time exploring the AI assistant, adding that the amount of time spent directly coding did fall.

"Several participants spent substantial time interacting with the AI assistant, spending up to 11 minutes composing AI queries in total," the researchers said, adding: "Using AI decreased the amount of active coding time. Time spent coding shifted to time spent interacting with AI and understanding AI generations."

No simple answers

The researchers broke down the results into six "AI interaction patterns" that might help shed light on the best way to use AI for work and learning.

Those who fell into the "AI delegation" group used AI to entirely write their code, getting the fastest results but scoring poorly on the quiz.

The "progressive AI reliance" group asked a few questions first before delegating all code to the tool; they also scored poorly on the quiz.

Elsewhere, the "iterative AI debugging" cohort used the AI to debug and verify their own code, but "relied on the assistant to solve problems, rather than clarifying their own understanding," so they failed the quiz and also took a long time to finish the task.

The next three groups all scored above 65% on the quiz, suggesting they managed to use the AI tool while also learning new skills. The fastest of these three were those who asked "conceptual" questions of the AI to boost their understanding before completing the task, learning as they went.

Crucially, researchers found two other groups used AI to better understand their task as well: the first generated code and then unpicked how it worked themselves, while the other asked for explanations of the generated code so they could learn as they went.

Both took longer to complete the task, but this had a positive impact on broader understanding and capabilities. What this underlines is the fact that AI can help boost skills if deployed in the right way.

"Participants in the new AI economy must care not only about productivity gains from AI but also the long-term sustainability of expertise development amid the proliferation of new AI tools," they added.

Critical thinking on the line

This research is the latest in a string of studies to highlight the balancing act with frequent AI use among knowledge workers of all varieties.

In February last year, research from Microsoft on the impact of ChatGPT showed frequent use could have a negative impact on critical thinking skills.

The study, conducted alongside researchers from Carnegie Mellon University, warned that workers using the tool encountered “diminished independent problem-solving” skills.

Similar research from MIT in June also highlighted the risks of overreliance on AI tools, noting that those using the technology at length saw a marked decline in critical thinking and evaluation skills.

FOLLOW US ON SOCIAL MEDIA

Make sure to follow ITPro on Google News to keep tabs on all our latest news, analysis, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.