A quarter of security vulnerabilities are never fully fixed – and AI is compounding the problem

Fixed that flaw? Possibly not. New research warns that developers are failing to fully address security vulnerabilities a quarter of the time – and the use of AI is exacerbating the issue.

In a study conducted by SecureFlag, the company analyzed 805,663 attempts by developers to fix flaws as part of its training platform over the last two years.

The study found that three quarters of such efforts worked, but that leaves a quarter of security fixes incorrectly implemented, or otherwise flawed – although that assumes flaws in the real world are failing to be addressed at the same rate as in a training lab.

Notably, when developers get a fix wrong without asking for help, it suggests they don't know why they’ve failed to remediate a flaw. Researchers said that this knowledge gap could prove problematic as the use of AI across the industry gains traction.

"The remaining quarter is the work that will increasingly be delegated to AI tools, and someone still needs to be able to tell the difference," said Emilio Pinna, CTO at SecureFlag, in a statement.

SecureFlag’s research suggested that developers don't always know when they've failed to deploy a solution properly, Pinna noted, nor can they establish why AI has failed to fix it.

"Developers who cannot independently fix a vulnerability cannot reliably verify that an AI has fixed it either," he said.

Indeed, the report suggested that solve rate is perhaps less important than hint rate, as that highlights when developers in training are close to solving a problem or at least aware that they don't know the answer.

In such instances, AI can be helpful as the concepts are understood by the developer.

But problems arise when developers don't ask for a hint in the training module and still fail the challenge – meaning they wrongly thought it was correct.

Masking the knowledge gap

That was true with two of the toughest categories. According to the research, developers had the hardest time with broken access control and software and data integrity failures, which each saw failed remediation attempts three out of ten times.

In the case of access controls, developers were "disabling the control they were asked to apply" while others were applying "cosmetic rewrites that preserve the vulnerability".

Crucially, it was one of the categories where developers were least likely to ask for a hint, the report noted.

SecureFlag said that those categories – broken access control and data integrity failures – were two of three instances where a "plausible-looking but incorrect fix" was most directly exploitable.

"A model generating a security fix will almost always produce something that looks plausible. The hard part is knowing whether it fits the context," said Emilio Pinna.

"When AI tools generate fixes before a developer has built that judgment, they don't close the gap; they mask it."

Indeed, when it comes to software and data integrity, SecureFlag reported that developers were "not internalising the concept even with assistance".

That's especially problematic given most security categories have multiple correct approaches to fix any given flaw. At the same time, a solution in Java, Python, or Rust may look structurally different. That makes it hard to evaluate whether AI has taken the right road.

"The value of hands-on security training is no longer mainly about developers writing every fix unaided," Pinna added. "It's about developers who can tell a correct security fix from a plausible-looking one. Our data shows, category by category, where that ability is weakest."

FOLLOW US ON SOCIAL MEDIA

Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.