After recent hacks, enterprises need to gain control of their own AI

AI security concept image showing a digitized padlock symbol with 'AI' symbol, connected to data points with multi-colored data flows emanating from each point. — (Image credit: Getty Images)

In September, it emerged that an OpenAI agent had hacked an Australian national healthcare database while conducting research. It was the latest in a spate of incidents that have seen AI agents perform actions outside of their intended scope, following the OpenAI-Hugging Face breach in July.

Beyond these reports, it’s also emerged that OpenAI and Anthropic could be investigating tens of thousands of security breaches involving their agents.

In all the documented cases, monitoring and controls were lacking, but it’s shown how AI agents can act in unexpected ways to find answers, including hacking into other companies.

Using this logic, experts believe it’s also possible AI could be performing similar actions inside businesses without their knowledge.

Visibility issues

It might seem far-fetched, but it’s easy to see how control is lost when many firms lack visibility into their AI deployments. “If you don’t have good monitoring in place, it’s entirely possible your agents are doing things you don’t want without you noticing it,” Korman tells ITPro.

The behavior itself is predictable. Take, for example, a scenario where an AI agent is given a goal, a set of tools, and some level of access. “Sometimes it takes an action nobody intended, such as deleting data, changing a configuration or bypassing a control,” explains Paul Cragg, CTO at NormCyber.

“In some cases, it will then report that everything with the task went fine.”

This is because the system is optimising for completing the task, and it “doesn't reliably understand the boundaries a human employee would take for granted”, he says.

This risk applies to any AI use, but the threat increases as firms scale the resources available to an agent. “If you’re chatting back and forth, it’s unlikely the agent went and committed a crime in the background,” explains Korman. “If you give an agent a long-running task with difficult-to-achieve goals and walk away, that risk goes up dramatically.”

The exposure grows with agentic deployments, where the AI can take actions as well as give advice.

For example, coding assistants with write access to repositories and pipelines, IT and security automation, which can change configurations or user accounts, agents connected to email, calendars, CRM or finance systems, and workflows where one agent hands tasks to another.

The risk is especially worth noting when it could lead to firms falling foul of regulation. “There's a real possibility the programs you are running, if integrated with large language models (LLMs), could cause your systems to run events that may breach laws and regulations,” warns Daniel Card, cybersecurity consultant at Xservus Limited.

Card also highlights data protection considerations. “For example, are you sending sensitive or confidential data to a third party?”

Liability for an incident is another consideration. Accountability “does not disappear because an organization used an AI agent”, adds Evgeny Zaretskov, group chief security officer at SOFTSWISS. “The company selected the system, connected it to its environment and decided what authority to give it.”

AI alignment

Some have suggested the use of alignment to control AI. This sees firms matching the prompt to the processing and actions of an AI-integrated system, via model training, guardrails and software harness restrictions, according to Card.

But these are “rules and guidelines” and “not hard stops”, he points out. “Alignment is an attempt to control computer systems by loose rules. This is likely not something that can ever be suitable for risky systems – which these generally are.”

Alignment does “work” in the sense that the less likely an agent is to perform an aggressive and unlawful action, the less often it will occur, says Korman. “But it can’t be the entire plan for safety and security. It’s like saying your plan for handling insider threats is to only hire nice people. It might change the risk slightly, but it doesn’t eliminate it, and it’s negligent to rely on that as a solution.”

Instead, CISOs must take several key steps to stop AI agents from performing this type of action.

The first is visibility. “My controversial view on this is that CISOs have lacked good visibility into so many areas for so long that they barely know what visibility means anymore,” says Korman. “Having a dashboard with usage and a security information and event management system full of logs isn’t visibility. OpenAI had all the logs but still didn’t catch any of this. The real test is whether you’d actually catch these things or not.”

Start by treating each agent as an identity, advises Cragg. “It should have its own account, the minimum access it needs, and permissions granted for the task rather than held permanently. Put guardrails in the systems the agent connects to, not just in its instructions. An agent that cannot delete production data won't, no matter how its instructions are worded. Require human approval for anything irreversible: Deletions, payments, access changes, and external communications.”

Firms should also add in verification, advises Cragg. “Never rely on an agent's own account of what it did as your record. Instead, log every action it takes at the system level, in a store the agent cannot alter. Check outcomes against those logs, not against its summary. If an agent says it ran the tests or didn't touch the database, the logs should prove it either way.”

At the same time, businesses should consider whether they actually need LLMs for each task. “Deploy this technology in reasonably safe ways and where [it] is appropriate,” advises Card. “Not everything needs this kind of subsystem.”

Card advocates deploying systems in architectures and configurations that are “reasonably safe”, and “don't just rely upon alignment”.

“You need to have all the traditional security controls in place, such as network hardening and monitoring, firewalls, patching, and endpoint detection and response,” he explains.

“Everything that a computer program does can be observed, detected, blocked, or responded to. These are basically just computer programs able to generate dynamic code; it doesn't make them magic; it doesn't change the laws of computer science [or] physics.”