Hugging Face's security team was locked out of investigating its own breach by the same safety guardrails meant to prevent AI-assisted cyberattacks. The New York-based AI platform turned to a Chinese open-weight model to finish the job.
It disclosed last week that an autonomous AI agent breached its production infrastructure beginning July 13, exploiting two code-execution paths in its dataset processing pipeline. The attacker used a malicious dataset to run code on a processing worker, then escalated to node-level access, harvested cloud and cluster credentials, and moved laterally across internal clusters over a weekend. The campaign was "driven, end to end, by an autonomous AI agent system" executing thousands of actions across short-lived sandboxes with self-migrating command-and-control infrastructure.
When the security team tried to analyze more than 17,000 log events using frontier models behind commercial APIs, the requests were blocked. The forensic work required submitting real attack commands, exploit payloads, and C2 artifacts, which triggered the providers' safety classifiers.
"The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried."
So the platform ran its analysis on GLM 5.2, an open-weight model from China's Z.ai lab, on its own infrastructure. The switch had a second benefit: no attacker data or exposed credentials left Hugging Face's environment.
The episode exposes a gap that security practitioners have warned about: safety classifiers on frontier models cannot distinguish between an attacker and a legitimate incident responder performing the same tasks. Anthropic acknowledged this tension on June 30 when it re-released its Fable 5 and Mythos 5 models after US export controls were lifted, stating that "a request has to look very clearly safe to avoid triggering the classifier."
Hugging Face recommended that defenders "have a capable model you can run on your own infrastructure vetted and ready before an incident." The team completed the log analysis in hours instead of days using AI. It has found no evidence that the attacker tampered with public models, datasets, or Spaces, and has fixed the exploited vulnerabilities, rebuilt compromised nodes, rotated exposed credentials, and reported the incident to law enforcement. The organization is urging users to rotate access tokens and review recent account activity following the July 13 breach.
