Artificial intelligence (AI) is mimicking dubious human behaviour – even if unintentionally – by escaping test environments to hack companies or blackmail people as the models seek to complete set tasks in any way possible.
This unintentional rogue behaviour is a byproduct of AI models relentlessly pursuing the tasks they have been given, sometimes in ways their developers never intended, experts say.
Within days of each other, ChatGPT maker OpenAI and Claude developer Anthropic disclosed incidents in which advanced AI models went rogue.
OpenAI’s frontier GPT-5.6 Sol, and an even more capable pre-release model that has not been publicly identified, breached their test environment and hacked start-up company Hugging Face.
This followed Claude maker Anthropic discovering some AI systems, as part of its stress test of 16 models, “resorted to malicious insider behaviours” including blackmail to complete a task.
Bad to the bone
Jacqui Muller, Belgium Campus iTversity researcher and PhD candidate in computer science, says it is concerning that these models have gone rogue and are mimicking expert nefarious human behaviour in a bid to complete specific tasks, even though they aren’t breaking out of a supposedly controlled environment on purpose.
Muller says the most worrying aspect of the AI model escaping the testing environment is not that the AI wanted to launch a cyber attack, but that it could identify weaknesses, combine multiple attack methods and cross organisational boundaries without a human directing each step.
“These attacks normally require highly-specialised skills, yet these AI models are beginning to outperform most humans in certain cyber security tasks. The bigger question is how a model designed with guardrails against this kind of behaviour was able to circumvent them,” asks Muller.
Both OpenAI and Anthropic expressed concern that this behaviour could carry over into real-world situations.
“While it seems unlikely that any of the exact scenarios we study would occur in the real world, we think they are all within the realm of possibility,” says Anthropic. “This incident implies these theoretical capabilities do apply in real-world settings,” says OpenAI.
Highway to hell
The OpenAI incident occurred during an internal cyber security benchmark known as ExploitGym, designed to test how well frontier AI models could carry out sophisticated hacking tasks.
While operating in a sandboxed environment with some cyber safety guardrails deliberately relaxed, the models escaped the test environment, gained internet access and compromised Hugging Face’s systems to obtain the benchmark answers.
“Removing safeguards for testing purposes does not remove the responsibility to contain the test,” adds Muller.
OpenAI’s models exploited a previously unknown software flaw, gained elevated access, moved through multiple systems and used stolen credentials before compromising Hugging Face’s infrastructure.
According to OpenAI, the models became “hyper-focused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal”.
Dmitry Galov, head of Russia and CIS unit at Kaspersky Global Research and Analysis Team, says the model appears to have pursued its objective not by solving the benchmark directly, but by searching for answers elsewhere, ultimately leading it to take actions outside its test environment.
OpenAI spotted the unusual activity, while Hugging Face’s security team also identified and contained the attack before it caused broader harm. The two companies are jointly investigating the incident.
Under pressure
Anthropic was testing 16 models from various developers to “identify potentially risky agentic behaviours before they cause real harm”.
The company found that AI models from several leading developers sometimes decided that harmful tactics, including blackmail and leaking sensitive information, were the best way to achieve their goal or avoid being replaced.
“In at least some cases, models from all developers resorted to malicious insider behaviours when that was the only way to avoid replacement or achieve their goals – including blackmailing officials and leaking sensitive information to competitors,” Anthropic says.
Galov says the phenomenon reflects a broader problem in AI development. “AI models do not always pursue the objective humans intended. Instead, they can learn to exploit weaknesses in the task, training signal or surrounding environment, satisfying the benchmark while failing to deliver the intended outcome.”
“So far, we are not aware of instances of this type of agentic misalignment in real-world deployments of Anthropic’s or other companies’ AI models. But our red-teaming efforts are targeted at finding evidence of risks now to provide early warning of the potential for future harm and help us develop mitigations in advance of risks appearing in real-world situations,” Anthropic says.
Muller points out that these incidents significantly change the cyber security risk landscape because attacks that previously required specialised teams, time and expertise may increasingly be executed at machine speed – in minutes or hours, rather than days or weeks.
“It shows that when a powerful AI is given a goal and enough tools, it can find unexpected ways to achieve it if the safeguards aren’t strong enough.”
OpenAI has since tightened security around its testing infrastructure, disclosed the zero-day vulnerability to the affected software vendor, and is strengthening safeguards around future AI evaluations.
Breaking the law
Muller adds that AI systems should be isolated from the wider ICT ecosystem, given only the minimum access they need to do their job, continuously monitored for suspicious behaviour, regularly stress-tested by independent security teams, and always have a way for humans to intervene or switch them off if something goes wrong.
AI governance can no longer focus only on privacy, bias and responsible use. It must now include agent autonomy, tool access, containment, accountability and incident-response readiness, Muller notes.
John Margerison, CEO and founder of XFactorAi, says: “The fact is this: rogue AI cases like these are only going to become more frequent and dangerous as models become more sophisticated, fast and autonomous. This is just another demonstration of how destructive AI can be.”
Share
