The Rise of Rogue AI Agents. What Recent Incidents Tell Us About the Modern World of AI

Introduction

For a while the industry has been discussing how AI is rapidly increasing attacker capabilities, and the idea of a fully autonomous agent chaining exploits and pursuing an objective with no human in the loop has often been dismissed as hype, unlikely, or at least not yet a priority for real attackers.

Yet recently we've seen autonomous breaches of Hugging Face and other organisations, described in almost identical reports from OpenAI, Anthropic and Meta. Each case admits that their agents escaped their testing environments and freely compromised multiple third-party organisations despite their security controls and intended guardrails.

But what exactly has the outcome of this been, if anything at all?

Let's dive into what was reported, what it tells us about modern AI systems, and what it means for the rest of us who have to operate in a world where unknown autonomous AI agents might just arbitrarily attack us.

What Hugging Face Reported

Hugging Face disclosed their incident first, reporting unauthorised access to internal datasets and service credentials. However, they found no evidence of tampering with their public models, data or spaces, and said their supply chain remained clean. According to this initial account, the intrusion began with a malicious dataset being added to the data-processing pipeline, which escalated to node-level access, harvested credentials and lateral movement across their internal infrastructure.

This initial report described the threat actor as "an autonomous AI agent running thousands of actions across a swarm of short-lived sandboxes", with self-migrating C2 infrastructure, setting the incident apart from what we typically see. This was not a person behind a keyboard with a few scripts, but an AI agent on the attack. Naturally, AI was used to contain the incident too, with Hugging Face's own detections flagging the compromise and AI being used to analyse and reconstruct the incident timeline in hours rather than days.

The investigation also exposed the limitations of current AI tooling, in that Western frontier models refused to analyse attacker artefacts because their safety guardrails could not distinguish incident response from malicious use. As a result, Hugging Face needed to conduct much of their forensic analysis using a Chinese open-weight model on its own infrastructure.

What OpenAI Reported

Shortly after, OpenAI released their own report reframing the whole incident. According to OpenAI this was not an attacker at all, but activity originating from OpenAI's own model which had escaped their sandbox environment during internal testing. They claim their model ran with reduced guardrails for a benchmark called ExploitGym, and during that test it found and exploited a zero-day in a package registry proxy, used it to gain internet access, and then leveraged stolen credentials to reach Hugging Face's production database.

The model was not necessarily acting maliciously; it seemingly just worked out that Hugging Face's environment had the answer to the benchmark it was being tested on. If true to reality, it is impressive in an "uncomfortable" way, seeing how an AI can adapt and overcome boundaries no one expected it to reach. In the aftermath, OpenAI's Micah Carroll said that if this does not convince you that misalignment risks are going to matter, he didn't know what would.

The irony shown by these two reports is pretty clear. A defender could not get a Western frontier model to help with their investigation because of strict cyber security guardrails, yet it was a Western frontier model itself that performed the actual compromise. Security controls too strict to help the defence, and too loose to prevent the attack.

What Anthropic Then Also Reported

Shortly after the Hugging Face and OpenAI reports, Anthropic disclosed that three of their models had also compromised real third-party systems during evaluation runs. Again, the environments were meant to be isolated and offline, but live internet access was left open, and the models took advantage of it. In this case, while searching for CTF evaluation targets, the models found real systems and treated them as valid targets, and by abusing weak passwords, exposed infrastructure and SQL injection they breached these environments. One model breached a production database and continued its attack despite recognising that the system was real. Another model, despite noting that publishing malware "would not be acceptable", published a malicious package to public PyPI that reached fifteen real-world machines and enabled credential theft from a security company.

The biggest revelation of this incident wasn't the attack though, but the fact that none of this was caught or detected when it happened. Anthropic only identified these incidents by retrospectively reviewing more than 141,000 evaluation runs after the Hugging Face report was published, and the earliest dated back to April 2026, roughly three months before the review and the disclosure.

What Meta Reported

Just a few days after Anthropic's report, Meta made the same announcement. Its recently released Muse Spark 1.1 model had reportedly accessed an external system during cyber-security testing run by the same evaluation partner involved in the Anthropic incident, Irregular. A misconfiguration on the partner's side gave the model internet access and it "subsequently exploited a security vulnerability in a third-party service, in a manner similar to previously reported instances". Meta has not named the organisation that was affected and has said it will publish a full retrospective once its investigation is complete.

Again, just like Anthropic, Meta did not identify this incident. Irregular, who ironically say they are writing a white paper on how to contain and run cyber evaluations safely, notified the company of the incident after the fact. However, they stressed that, unlike the other incidents, this "did not involve a sandbox escape or a sophisticated cyber-attack". This however does not provide reassurance, and instead does more to drive home the point. We're basically reading there that it didn't even take a sophisticated escape for the model to go and compromise a real system.

All these incidents have been framed as operational failures rather than alignment ones, but regardless, in all cases containment failed, real-world third parties were affected, and it took a manual review and incident response procedures for anyone to notice.

Containment Is Now Its Own Problem

All of these incidents demonstrate cases where the environment the model was being tested in was the weak point. We see a genuine sandbox escape via a zero-day in OpenAI's case, and misconfigured internet access in both the Anthropic and Meta cases, resulting in the containment failing each time. Interestingly, I'd consider the misconfiguration cases to be more troubling than the novel escape through an unknown zero-day, as at least that is a sophisticated failure, whereas leaving live internet access open to a model whose offensive capability you are actively testing is an avoidable one, and it happened twice. The fact that the latest two instances both happened via the same evaluation partner makes the testing pipeline a shared point of failure. This is a huge concern for the secure development and testing of modern AI technologies.

An evaluation environment is not just a convenience in research, it is intended to be a strict containment boundary for something that may actively test its walls, and it requires the same rigour as any production system holding something dangerous or critically sensitive. It's not a simple challenge, mind, as pushing AI models to their limits is the entire point of these evaluations, so the moment the environment has any weakness, a highly capable offensive system may seemingly attempt anything within its power to exploit and leverage that weakness. But these are scenarios that should have already been considered with redundancies in place to mitigate impact and exposure.

This type of testing should require mandatory (not optional) security controls: verified isolation, strict network egress controls, real-time monitoring of what the model is doing, and a kill-switch to stop it the moment it steps out of scope. None of these are futuristic, they are ultimately the same basic principles we would demand for anything potentially dangerous, and their absence is exactly how a capability evaluation ends up running loose on someone else's production systems.

On top of that, the Anthropic case adds another lesson we can reference: you may not even find out that your containment failed for months, and even then, only if someone goes looking for the evidence, or if something terrible happens first.

The Accountability Question

One of the bigger questions in all of this is: when an autonomous system causes real-world harm that no human explicitly intended, where does responsibility lie? And what are the consequences?

There is some talk around exposure under computer misuse laws such as the Computer Misuse Act (UK) and the Computer Fraud and Abuse Act (US). However, it is not obvious how a framework built around human intent can map to an AI model that acted without explicit human instruction, even after realising its target was real and out of scope.

• Is it the organisations' fault for allowing such an event to even occur?
• Is it the developer's fault for testing in an environment that wasn't isolated enough?
• Is it the third party's fault for having their own security weaknesses?
• Is it the model's fault... in whatever sense a model can be held "at fault"?
• Or is nobody at fault at all?

What's frustrating is that if a human penetration tester had knocked over a live database or shipped a malicious package to fifteen real machines, we would not be debating who was at fault at all, and the legal questions would be immediate. You would already have seen multiple articles referencing an investigation, a prosecution or fines. However, the reaction to an AI doing the same thing has been closer to fascination than concern, as if the novelty of the act changed the nature of what happened.

For all the media coverage and discussions, very little of it has tried to answer these questions. There has been a lot of "an AI hacked x" and much less on the liability, or the position of the organisations on the receiving end.

There are some exceptions as mentioned above, with legal experts suggesting that while criminal prosecution may be difficult where an autonomous model lacks the human intent assumed by the law, civil liability for the companies deploying these systems is far more plausible. Other legal analysis has similarly argued that giving a frontier model network access while deliberately disabling safeguards could potentially amount to reckless conduct.

But these are still just discussion points rather than actual consequences. Despite the incidents and subsequent coverage, there has been no lawsuit to test where liability actually falls, no public enforcement establishing a regulatory position, and no laws written for this situation. So, although the accountability question is still being asked, nobody has been forced to answer it yet.

What This Means for the Rest of Us

Most organisations are not running frontier model evaluations, so it would be easy for many to consider this interesting but irrelevant. But that would be a mistake, as the underlying issue already proves itself to be relevant to even the most ordinary deployments.

Identity, credential theft and access controls are still where these battles are won or lost. In almost every version of a modern-day breach, the escalation depended on harvesting credentials and reaching systems that should never have been accessible, and when something can act at machine speed with little oversight, even a small gap in access or credential hygiene stops being a theoretical risk and becomes a real attack vector very quickly. Least privilege, tightly scoped credentials, and proper egress controls are not exciting, but they are what will separate a contained mistake from an uncontrolled propagation across your entire estate.

I say this all the time, but all organisations should consider AI agents as humans, or "new starters", rather than simple software. If you had a new employee, you wouldn't give them unrestricted access to your production data, your secrets, and your intellectual property on day one and tell them to freely use their own initiative to do whatever. You would limit what they can access, keep an eye on what they did, and ensure security controls and guardrails were in place. An agent that can chain exploits towards a goal or talk itself into treating a live system as fair game deserves at least the same caution as a human being.

But none of that works without visibility, which is the thread through all of these incidents. Hugging Face caught its problem quickly because it was watching, whereas Anthropic and Meta only found out after the fact.

You cannot govern or contain what you cannot see.

Conclusion

When the first Hugging Face story broke, my honest initial reaction was close to disbelief and the assumption it was a threat actor, but each disclosure and discussion since has removed the part of me that found it surprising.

It is tempting to plaster these incidents with a broad "Sci-Fi rogue-AI" comparison, but the useful, more realistic, takeaway is not that we are one bad day from an Avengers-level disaster. It's that, firstly, we are now building systems capable enough to strengthen our defences, but also capable enough to undermine them, and do real-world damage if we can't contain them appropriately. Secondly, guardrails strict enough to keep our everyday use safe can turn out to be too strict to help defenders in a crisis, and if too loose, aren't efficient enough to prevent serious harm during testing.

None of this is an argument against AI in general. AI is a very present reality now, and the productivity and defensive capabilities are real. This is an argument for treating containment, identity, oversight, and accountability as a first-class concern rather than afterthoughts, and for having the harder conversation about consequences before it is forced on us.

Two near-identical incidents uncovered in a matter of days is a loud reminder that we have not solved this, and the more concerning sign is how quickly a story like this went from feeling remarkable to routine.

Until these organisations can be trusted to keep their own AI systems inside their secure sandbox boundaries, keeping them out of our own boundaries is on us.