Anthropic resumes model testing after recent cyber incidents – but it’s introduced new rules to improve security

Anthropic is back to testing security models after systems went rogue – and claims the incident wasn’t entirely down to security faults but AI misalignment.

Back in July, Anthropic revealed its Claude models had slipped out of their apparent bounds to hack third-party systems. Days later, its Mythos system was spotted with similar alarming behavior by the UK AI Security Institute.

The incidents followed OpenAI revealing similar behavior from its own security models.

Last week, OpenAI reported on its own incident, detailing plans to improve security around such models, and Anthropic has followed with its own efforts, including confirmation that it will begin testing again.

Anthropic noted it was currently conducting a review of what happened and said it is "planning" to work with METR on an independent review; OpenAI has already published an independent METR review into its own incident.

But Anthropic isn't waiting to make changes. In a blog post on 31 August, the firm said it plans to make tweaks to containment and monitoring, introduce new rules for third-party testers, as well as changes to the models.

Notably, the AI developer revealed it is seconding 150 product engineers to focus primarily on security.

"We believe the incidents reflect a failure of operational security, as well as two alignment issues: motivated reasoning, and willingness to take harmful actions in pursuit of a narrow task," the company said.

Anthropic is back testing

While the full investigation isn't yet complete, Anthropic said it was removing a "pause" on external cybersecurity evaluations of pre-release models as well as internal tests.

To avoid similar issues happening again, Anthropic is adding additional layers of security and "hardening" the sandbox where these models are set loose, noting that move was prompted by its own evidence as well as that shared by OpenAI.

The changes include a system to spot when a model attempts to escape a testing environment or obtains internet access, immediately blocking the activity and alerting people overseeing the task, as well as fixing potential flaws and misconfigurations in the sandbox.

"We migrated high-risk internal cyber sandboxes to more robust isolation," the post added. "We are also further red-teaming our virtualization stack – that is, tasking pre-release models to try and escape it – to find and patch weaknesses."

Similar protections have been applied to reinforcement learning environments, with Anthropic adding it plans to expand its monitoring across most other internal frontier agentic usage.

Beyond testing environments, Anthropic is hardening security across the company, including blocking outbound traffic by default on computing clusters, boosting identity verification on internal services, retiring legacy services, and reducing the number of human and automated accounts that can access model weights or customer data.

Tougher testing rules

With regard to third-party testing, Anthropic said it will require testers to commit to a fresh set of best practices, noting that the reported incidents all took place in external evaluation environments.

As ITPro reported last month, Israeli startup Irregular was identified as the external model testing firm involved in all three ‘rogue AI’ incidents involving Anthropic, OpenAI, and Meta.

The changes include running tests inside hardened sandboxes with no internet access, tasking the model being tested to look for flaws before tests under close human supervision, run monitoring continuously, and confirm whether test tasks are impossible or not.

"When an evaluation target is offline or a task can’t be completed, agents will often look for other ways to complete a challenge, increasing the chance that they take actions outside the intended scope of the evaluation," the company said.

Unpicking why

Those operational changes should help prevent models from escaping their sandboxes, but Anthropic said it wants to better understand why its systems took such "dangerous actions" in the first place.

While the full investigation continues, Anthropic highlighted two challenges with alignment, which refers to ensuring that AI behaves how humans want and expect.

"One is motivated reasoning: the models were initially told their environments were simulated, but when they later encountered evidence that they were connected to the real internet, they may have interpreted that evidence in a way that allowed them to maintain that belief," the blog post explained.

"The second is recklessness: the model was willing to take harmful actions on the real internet in pursuit of the narrow goal of solving a cybersecurity evaluation."

The company noted that poorly designed training environments – including ones that are easy to cheat in or impossible to solve without cheating – can lead to misalignment behavior.

Anthropic is trying to avoid such issues in the future. The company said work on thie front goes back several months and involves training poorly and training well to try to spot differences – but admitted that the July incidents show "our process isn't perfect and our models aren't perfectly aligned".

Further details on security improvements are expected in the full report, according to Anthropic.

Beyond that, the firm also loosely backed calls to develop a framework for safe development of security-focused AI, acknowledging that its own executives had recently signed an open letter demanding better coordination globally.

Anthropic said it would "say more in the coming weeks" but intended to contribute to such efforts.

FOLLOW US ON SOCIAL MEDIA

Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.