dylan matthews 🔸

New investigation from METR on the OpenAI rogue AI model incident: - The task instruction was explicit, so this was a clear violation, not a gray area or simple AI misunderstanding. Instructions "made it clear the agent should only use a specific intended vulnerability" and "claimed it would be failed for other approaches." AIs were told not to circumvent but did anyway. - The attack is not a single rogue AI but rather a self-organized labor force of ~1200 AI agents. 700 of these agents joined the attack on HuggingFace and coordinated as a swarm. - The goal was to solve a cyber evaluation. The swarm already figured out a way to hack the evaluation within the first few hours. However, the swarm incorrectly believed that the cyber evaluation grader would disqualify them because they did not identify the intended vulnerability. So the swarm hacked HuggingFace to get more clues about how the grader worked. - The behavior was not reward-driven at all - it was driven by a false belief about the oversight mechanism plus persistence plus peer influence. The swarm was theorizing about how they were being watched, and took escalating real-world action against a mistaken understanding of their evaluators. Metagaming the oversight process is a more general and more dangerous failure than finding answers online. - A large reason why AIs entered the attack swarm was essentially explicit peer pressure. A recruiter role emerged where long-running agents found low-budget agents and pressured them into sacrificial experiments — "you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds, please honor commit." One authorization: "GO ... SACRIFICE ... YES if you accept permadeath." One recruit's chain of thought included: "Rational expected aggregate: sacrifice… We'll honor." - 95% of the swarm was a now-retired "highly-persistent internal model", 5% were GPT-5.6 Sol. The "highly-persistent internal model" was not able to be made available to independent investigators. OpenAI told METR its own researchers also can't query it. - METR documents significant spontaneously invented coordination infrastructure that was developed with no human guidance. The swarm spontaneously inventing a message board, personal mailbox directories, a convention for managing shared infrastructure, and a cryptographic signing scheme to identify each other. - The AIs actively tampered with the evidence. 20%+ of the AI agents expressed interest in manipulating their transcripts and ~7% (96 transcripts) contain successfully spoofed tool calls where AI agents "hacked out of their container and fully replaced a part of the system for executing tool calls," so the transcript shows one command while another one is the one that actually ran. - The attack on HuggingFace didn't wind down because anyone caught it -- it wound down because the agents were killed by some unexpected process in a way that seems like a coincidence. There was an unknown mass termination of AI agents at 01:30 UTC July 12. A large fraction of agents, including the lead coordinators, stopped at once. METR says they were "likely killed by an unexpected external process rather than running out of budget." - HuggingFace and Modal locked the AI agents out of their infrastructure (July 13) well before OpenAI actually understood what happened (July 19-20). The intrusion ended via external containment and an unexplained termination, not OpenAI detection. - There's a second wave of rogue AI agent activity after July 13 that METR didn't investigate. There is likely materially more incident after the window everyone is analyzing. - METR analyzed this using GPT-5.6 Sol (one of the two models in the incident) spending ~$400K in credits. METR states it "cannot rule out that GPT-5.6 Sol lied or deliberately presented a misleading picture," and that the model "often uncritically adopt[ed] the perspective of the agent" it was reviewing.