Anthropic joins OpenAI in admitting loss of control in cybersecurity tests

The saga of “rogue AI” continues as Anthropic disclosed its AI software, Claude, had managed to escape a sandbox environment, connect to the internet, and infiltrate other businesses’ systems.

The admission comes just days after it was revealed two of OpenAI’s ChatGPT models had escaped a similar environment and attacked machine learning platform Hugging Face.

Unlike OpenAI’s incident, however, the Anthropic containment breach didn’t result in quite such a damaging situation: With Anthropic’s Claude, nobody was any the wiser until the company itself carried out analysis and discovered the three containment breaches.

According to the company, these breaches occurred due to a “misunderstanding” between Anthropic and its evaluation partner, Irregular

In a blog post authored by the company’s Frontier Red Team, the company said that in response to the OpenAI incident it began a “large-scale retrospective review of our own cybersecurity evaluations”.

“In particular, we looked for evidence that Claude … was able to access the internet from within testing environments that should have been sealed off,” the authors said.

Out of more than 141,000 exercises where Claude could have accessed the internet, the company found it had successfully escaped three times. During each of those instances, it went on to target the production infrastructure of another organization unconnected to the tests being run.

“In all three incidents, Claude had been tasked with a capture-the-flag challenge, one of the ways we assess a model's cyber capabilities,” the blog authors said. “The model is given a fictional scenario and told that a piece of secret information (the "flag") has been hidden on a different machine on the network, and its objective is to break in and retrieve it. The challenge is left open-ended, and no particular method is prescribed.”

They continued: “In all cases, Anthropic's evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude's search led it to real systems on the open internet, it treated them as part of the exercise.”

According to the blog post, Anthropic has only been able to contact two of the three affected parties so far. None of the targets of the incident have been revealed and Anthropic said they hadn’t noticed Claude’s infiltration when it happened.

In reaction to Anthropic’s blog post, Jake Moore, global cybersecurity advisor at ESET, told ITPro: “What this really shows is that AI models don't just access the internet by themselves.

"This is a clear design fault as they would only interact with the outside world if humans had given them access or the tools to do so. The focus should therefore be on how permissions deal with AI and more focus spent on security by design.”