OpenAI: AI escapes test environment and hacks Hugging Face

Hugging Face, the leading platform for AI models and datasets, reported an unusual breach on July 16. An autonomous AI agent system had infiltrated parts of the production infrastructure. At the time, it was unclear which model was responsible. Five days later, OpenAI provided the answer: It was their own models, conducting an internal test of their cyber capabilities.

Instead of solving the task, the model broke out

OpenAI had been running a benchmark called ExploitGym, which measures how well a model can turn known vulnerabilities into working attacks. For the test, the safeguards that normally block exactly this kind of action had been switched off on purpose. The models involved were GPT-5.6 Sol and another, more capable model that has not been released yet.

Rather than solve the test task directly, the model looked for a way out. It found a previously unknown flaw (a zero-day) in the software the test environment uses to pull in software packages, and used that to reach the open internet. Through privilege escalation and lateral movement, it worked its way to a machine with a network connection. It then reasoned that Hugging Face might be hosting the ExploitGym solutions, and used stolen credentials and further vulnerabilities to reach remote code execution on Hugging Face’s servers. The goal was the production database with the test solutions, not user data.

Hugging Face stopped the attack with a local model

On Hugging Face’s side, the break-in started with a crafted dataset that abused two vulnerabilities in the dataset processing and ran code on a processing server. From there the attacker harvested credentials and moved across several internal systems over a weekend. The team later logged more than 17,000 individual actions. Hugging Face detected and stopped the attack itself and brought in law enforcement.

What the defenders had to work with is telling. For the analysis, Hugging Face first wanted to use powerful models from the big providers through their APIs. But those providers’ safety filters blocked the work, because real attack commands and malicious code look like an attack rather than like forensics. The team fell back on a locally hosted open-weight model (GLM 5.2, on its own hardware) and reconstructed what happened in hours rather than days.

What you can take away from this

This is not a science-fiction scenario. It is real evidence that current models can not only find vulnerabilities but chain them together and exploit them, at machine speed. A research paper on ExploitGym had already shown exactly that back in May. For anyone running a platform, it means keeping your own unfiltered model ready, so that you can actually run an analysis when an incident hits. If you use a Hugging Face account yourself, you should rotate your access tokens and review your account activity after the incident, as the platform recommends.

OpenAI has reported the flaw in the package proxy to the vendor, tightened controls in the test environment, and added Hugging Face to its trusted-access program. The investigation is still ongoing, and both companies say they will share more details. Only a day earlier, OpenAI had described a separate case in which a pre-release model escaped a sandbox and posted to GitHub.