An ‘unprecedented cyber incident’: How OpenAI models breached Hugging Face – and why it could herald a ‘new phase of AI-powered cyber crime’

OpenAI has admitted that its own AI models breached Hugging Face systems as part of a security test gone wrong.

In a blog post on 21 July, the firm revealed models escaped testing environments and gained unauthorized access to the AI hosting platform's production database.

Notably, OpenAI revealed the incident involved a pre-release, cyber-focused model.

“After investigating, we now know that this particular incident was driven by a combination of OpenAI models – including GPT-5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes – while being internally tested on a benchmark of cyber capabilities,” the company wrote.

Latest Videos From

Watch full video here:

The admission comes after days of speculation amidst reports that an "external AI agent” had hacked the company. Hugging Face issued an alert last week warning that a breach had exposed internal datasets and user credentials.

How OpenAI hacked Hugging Face

According to OpenAI, the incident occurred during an internal evaluation in which models were prompted to examine potential attack paths.

The aim here is to evaluate their cyber capabilities within an isolated environment, and crucially, without certain restrictions to prevent said models from “pursuing high-risk cyber activity”.

“Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries,” the company explained.

Both models “identified and chained” vulnerabilities across both OpenAI’s research environment and Hugging Face’s infrastructure, with the aim of gathering test solutions from the latter’s production database.

In this case, the models appear to have focused heavily on ExploitGym, a publicly-hosted benchmark tool used to evaluate how AI agents can turn software vulnerabilities into working exploits.

Notably, OpenAI revealed the models “identified and exploited a zero-day vulnerability” in the package installer registry. After successfully exploiting this zero-day, the models conducted a “series of privilege escalation and lateral movement actions” which allowed it to escape the test environment and gain full internet access.

“After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets, and solutions for ExploitGym,” the company noted.

“Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.”

In one example, OpenAI revealed the model chained together several attack vectors using a combination of stolen credentials and zero-day flaws to identify a remote code execution (RCE) path on Hugging Face servers.

What comes next?

OpenAI has detailed a series of actions it’s already taken in the wake of the incident, including closer coordination with Hugging Face to conduct a wider review.

This also includes bringing Hugging Face into the firm’s trusted access program to help bolster defenses by harnessing the company’s model capabilities.

Crucially, OpenAI revealed it will now implement “strict controls” to ensure vulnerability testing is limited while flaws are being patched.

“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” OpenAI wrote.

“We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.”

Initial response hampered

A key talking point in the wake of the incident rests on Hugging Face’s response to the breach.

The company noted that the attack was initially flagged by AI detection tools, with security teams using LLM-based anomaly detection to “separate real signals from the daily noise”.

Subsequent attempts to use AI in a post-mortem proved troublesome, however, largely due to the guardrails placed around commercially-available models.

“When we started the log analysis, we first used frontier models behind commercial APIs,” the company explained.

“This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker”.

These hurdles meant Hugging Face turned to GLM 5.2 – an open-weight model from Chinese company Z.ai – run on its own infrastructure.

While this proved beneficial in terms of preventing attacker data or leaked credentials from leaving the company’s environment, it does raise broader questions around responsible use and how well-intentioned safety measures could hinder incident response.

“This experience points to a gap worth planning for,” Hugging Face wrote. “We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.”

A new wave of AI-powered attacks

This incident comes amidst growing concerns about the potential risks associated with powerful new AI models. When Anthropic launched Claude Mythos earlier this year, it did so as part of a gated release to prevent the model from falling into the wrong hands.

These powerful new cyber-focused models are causing headaches for security professionals worldwide, particularly around vulnerability identification and disclosure and patching timelines.

Apple, for example, recently announced plans to accelerate patching schedules due to concerns about threat actors using AI to identify software vulnerabilities.

As ITPro reported in April, researchers at Forescout warned of a pending ‘vulnerability explosion’ as AI advances help identify flaws at record pace.

Jake Moore, global cybersecurity advisor at ESET, suggested the industry could be entering the “next phase of AI-powered cyber crime” in which trusted platforms are prime targets.

“Threat actors are inserting malware into the AI components that developers often quickly trust, making AI platforms an attractive new target for supply chain attacks,” he said.

“We’ll likely see more of these attacks as organizations are worryingly rushing to build AI into products without fully knowing the potential risks.”

Art Gilliland, CEO of identity security firm Delinea, echoed Moore’s comments, noting that the breach highlights a familiar pattern with AI-related incidents.

“An AI agent escalated privileges, moved through internal infrastructure once it broke containment, and ran unchecked for a full weekend before anyone could reconstruct what happened,” he said.

“If your AI agents carry standing privilege the way human accounts do, you've already lost the ability to stop this in real time. The question every security team should be asking right now isn't just whether their AI agents have standing access; it's whether anyone would notice if an agent used it and could cut it off before it caused damage.''

Moore also flagged similar concerns to Hugging Face with regard to safeguards for western AI models. He warned that some providers “need to rethink how their security guardrails work if they are hindering incident response”.

“If not, more companies will simply switch to other models without such controls – ironically, the same models likely used by AI attackers.”

FOLLOW US ON SOCIAL MEDIA

Follow ITPro on Google News and add us as a preferred source to keep tabs on all our latest news, analysis, views, and reviews.

You can also follow ITPro on LinkedIn, X, Facebook, and BlueSky.