AI risk decoded: What are the real threats posed by AI?

Over the last month, AI companies have claimed their own technology poses existential risks, with some of the bolder statements warning there’s a chance it will end humanity in the near future.

It comes after Anthropic released a report detailing real-life incidents of AI misuse, which has led to genuine concern among some experts.

At the end of September, OpenAI said it would scrap plans for GPT-6.1 Astra because its monitorability had decreased. The UK AI Security Institute says that monitoring and sandboxing is needed; something security experts think is non-negotiable.

How can businesses separate fact from science fiction while realizing the real risks posed by AI?

Extreme debate

The AI extinction debate centers very much on belief, rather than hard evidence. Yet there is a danger that the AI risk debate becomes polarized between two extremes, says Daryl Flack, partner at Avella Security. “Either AI is going to end humanity, or anybody raising concerns about increasingly autonomous systems is scaremongering. Neither position is particularly helpful.”

Some of the more dramatic claims around agent swarms and “imminent superintelligence” are being presented as established outcomes, when they are not, says Flack. He advises being very careful about “confusing demonstrations under controlled conditions with evidence that these systems are about to operate autonomously at scale and beyond human control”.

Indeed, the more alarming claims – including fears autonomous AI agents could eventually create biological weapons or launch nuclear attacks – overlook something fundamental: the physical controls involved, says Mark Molyneux, field CTO at Commvault. “An AI agent cannot simply decide to launch a nuclear weapon or pick up a machine gun. There are real-world barriers between generating an instruction and carrying it out.”

Frontier model risks

Currently, the risks posed by AI relate to a lack of governance, the technology’s ability to accelerate vulnerability discovery and exploitation, and the potential to boost the abilities of adversaries.

For businesses, human error from large language model (LLM) use, trusting web chat bots for complex tasks, making mistakes through lack of quality assurance and lack of security controls are “some of the real risks”, according to Daniel Card, consultant at Xservus. “A lack of care and attention with, or without, LLMs tends to lead to security and business incidents,” he tells ITPro.

Meanwhile, frontier models by companies including OpenAI and Anthropic are very effective at identifying software vulnerabilities. This is sparking fears that adversaries could use the technology to find weaknesses and exploit them at scale, before firms have managed to patch. “That's a genuine security concern, because weaknesses that might previously have taken weeks or months to uncover could potentially be discovered much more quickly,” according to Molyneux.

More broadly, AI can help criminals perform more efficient attacks. Previously, an attacker might have needed to manually search through large amounts of information to identify what was commercially sensitive. An AI agent could be trained to recognize the most valuable intellectual property, customer information, or proprietary research “almost immediately”, says Molyneux.

Adding to the issue, companies are “handing software agents real credentials and authority”, then “pointing them at email, documents and websites that anyone of them can write to”, warns Dom Selvon, CTO and value partner at Valliance.

This can allow adversaries to take advantage of indirect prompt injection as a data exfiltration route, he says, citing the example of last year’s EchoLeak in Microsoft 365 Copilot.

Future predictions

The immediate impacts are becoming clear, but what about the future? First and foremost, firms must ignore the hype. The world will “almost certainly not end” due to AI, according to Card. Instead, he says, businesses will deploy the technology “in a mixture of good and bad ways, incidents will occur – like they have for the last 30 odd years – and the world will keep turning”.

Molyneux doesn’t think AI will “suddenly become some all-powerful intelligence” over the next 12 months. “What I do expect is a significant acceleration in the use of AI for cyberattacks, alongside an increasingly complicated debate about how the technology should be governed.”

One of the biggest challenges is that AI development has become a global race, says Molyneux. “Countries and technology companies are investing enormous amounts of money and computing power into developing more capable models, and nobody wants to be left behind.”

That creates a difficult balance between innovation and security, according to Molyneux. “On the one hand, we need safeguards around increasingly autonomous systems. On the other, countries may be reluctant to introduce restrictions if they believe competitors will continue developing the technology without the same constraints.”

The real AI threat

For now, experts advise getting real about the actual threat. CIOs and CISOs should be making sure they “don't drop the ball on security foundations”, says Card. “This technology makes them more important, not less.”

As AI is increasingly used inside businesses, Flack believes monitoring and sandboxing will become non-negotiable. “If an AI system can autonomously interact with critical systems, organizations need to be able to see what it is doing, restrict what it can access and stop it quickly when its behaviour moves outside acceptable parameters.”

Molyneux thinks the real conversation needs to be about operational resilience, rather than AI itself. “Can the organization withstand an AI-driven attack? Can it identify what data has been accessed or changed? Can it recover its systems and identity infrastructure to a known good state? And, crucially, can it do all of that quickly enough to keep the business operating?”

It's equally important to think about what happens after an incident, says Molyneux. “If an attacker compromises an organization's identity systems, simply restoring from a backup may not be enough. If compromised accounts, permissions or malicious configuration changes are restored alongside it, the organization could effectively be giving the attacker access all over again.”

Before deploying the technology, Card advises engaging with the board and the organization to understand how, and if, the business is looking to leverage AI. “Get ahead of the game, work out how this looks from a security and safety point of view, use it where appropriate, get familiar. Organizations and people will likely be using some of these systems already, even if they aren't formally doing so.”