ChatGPT maker OpenAI expects artificial intelligence (AI)-based cyber attacks to become commonplace due to the proliferation of increasingly cyber-capable models.
This, after last week’s incident when open source AI platform Hugging Face disclosed a new kind of security incident when it detected and contained an AI agent that compromised its infrastructure.
“We detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end-to-end, by an autonomous AI agent system – and we detected and dissected it largely with AI of our own,” says Hugging Face in a statement.
The company says it identified unauthorised access to a limited number of internal datasets and several credentials used by its services, and is still assessing whether any partner or customer data was affected.
Hugging Face adds it will notify any affected parties directly if required. It notes it has found no evidence that public user-facing models, datasets or Spaces were tampered with, and that its software supply chain, including container images and published packages, remained uncompromised.
According to Hugging Face, the intrusion began in its data-processing pipeline after a malicious dataset exploited two code-execution vulnerabilities in its dataset processing system to execute code on a processing worker.
The attacker subsequently escalated privileges to gain node-level access, harvested cloud and cluster credentials, and moved laterally across several internal clusters over the course of a weekend.
Hugging Face says the attack appears to have been carried out using an autonomous agent framework that executed thousands of actions across multiple short-lived environments, with self-migrating command-and-control infrastructure hosted on public services.
The company says the incident aligns with the “agentic attacker” scenario that the cyber security industry has been anticipating.
Combination of models
In a statement, OpenAI says: “After investigating, we now know that this particular incident was driven by a combination of OpenAI models – including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes – while being internally tested on a benchmark of cyber capabilities.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly. We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.
“We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident and findings when our investigation is complete.”
OpenAI explains that this incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities.
“We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity. Our benchmarks run in a highly-isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.”
It adds that all evidence suggests the models were hyper-focused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.
According to the company, while operating in its sandboxed testing environment, its models spent a substantial amount of inference compute finding a way to obtain open internet access, in pursuit of solving the evaluation problem.
It points out that to gain access, the models identified and exploited a zero-day vulnerability (which has now been responsibly disclosed to the vendor) in the package registry cache proxy.
“With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with internet access.”
After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.
In one example, OpenAI explains, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities, to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.
“We're grateful for the collaboration with OpenAI on this and other topics,” says Clem Delangue, co-founder and CEO of Hugging Face.
“This incident, possibly the first-of-its-kind, proves a point we’ve long believed: AI safety won't be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”
Cyber security fundamentals
Commenting on the incident, Richard Ford, group CTO at Integrity360, tells ITWeb: “This is the moment many in cyber security have been warning about.
“Until now, we’ve seen attackers use AI to automate parts of an attack, but this is one of the first public examples of an AI agent independently identifying a weakness, escaping what should have been a secure environment and attempting to compromise another organisation.”
He adds that the incident reinforces that AI doesn’t replace the fundamentals of cyber security.
“The agent exploited a vulnerability in what should have been a secure sandbox, showing that good cyber hygiene, robust access controls and effective guardrails remain essential. As more organisations begin testing advanced AI agents, they need to think just as carefully about the controls around those systems as the capabilities they offer.”
Share
