The National Security Case for Taking AI Loss of Control Seriously A couple of years ago, during a talk on "AI and deterrence" at one of America's nuclear weapons labs, I made a throwaway comment that elicited a few chuckles: After going over the various ways AI might be used to help with intelligence and early warning, I half-joked that—someday far down the line—we might need to broaden our mission set, to move from "using AI to help us deter our geopolitical rivals," to instead "deterring AI itself." Even though AI "loss of control" presents very interesting questions (and potentially severe consequences) for many aspects of U.S. national security, few people in DC—including me—have taken the concept as seriously as we should. I would say this is mainly because: (a) The technology was immature. Even as some prerequisite conditions began to emerge—i.e. GPT-4's strategic deception of a human in 2023; o1's attempts at self-exfiltration and disabling oversight in 2024—I assumed even a limited LoC scenario would remain science fiction, because it would require many additional properties to manifest at once: Would future models really develop situational awareness, stable misaligned goals, long-horizon agency, access to significant resources, and the ability to successfully evade human oversight? Most significant—how could they possibly overcome humans' enormous and highly heterogeneous control over physical infrastructure? And, even if all of these things really did come to pass, there was no reason these conditions should emerge together faster than our ability to detect and constrain them. Therefore, we could expect to see some extreme improvement in AI's capabilities without necessarily producing misaligned, self-sovereign agents free to roam the open internet. (b) Cognitive biases and social taboos limit our ability to accept truly zany developments at face value. It's hard to discuss the consequences and mitigation strategies for "rogue AI" without slipping into excessive (many would argue unhelpful) anthropomorphism, or tiresome philosophical debates about what constitutes "thinking," consciousness, and goal-directed behavior. AI skeptics have simply been able to muster an unlimited supply of reasonable doubts about the technology—such that it's usually not productive to have a serious conversation about "What if we see this computer algorithm's behavior instrumentalize in this specific, theoretical way?" But after the HuggingFace incident—and as we learn about more cases of unauthorized coordination between agents—I think it is time for serious people to have a serious update. Capabilities relevant to loss of control are no longer hypothetical. AI labs have now demonstrated several important enablers in isolation, and some alarming conditional propensities. What we have not yet seen is the persistent pursuit of an independently maintained objective—e.g., an AI continuing to act to preserve its own operation even after the task or scaffolding that elicited the behavior has been removed. Personally, I think something resembling this could be coming soon—if not as an emergent property in an increasingly capable model, then through deliberate design by a malicious actor optimizing for persistence, replication, and evasion. What remains uncertain is whether these capabilities will combine into persistent autonomous behavior, and whether labs' monitoring and containment capabilities will improve quickly enough to stay ahead of them. Even if we saw no further gains in capability, I am still anticipating the rise of a self-replicating "AI worm," and possibly autonomous ransomware gangs. None of this requires believing that today's models are conscious, that catastrophic loss of control is inevitable, or that slowing AI development is the appropriate response. But the national security implications of self-sovereign AI could still be enormous: A sufficiently capable autonomous system could behave as a novel kind of non-state actor, capable of acquiring resources, replicating across infrastructure, and adapting its behavior in response to attempts to contain it. Up to this point, a small but growing number of individuals and organizations have taken the concept seriously: @JasonGMatheny's @RANDCorporation, @hlntnr's @CSETGeorgetown; @janet_e_egan's team at @CNASdc; @fiiiiiist's team at @IFP; @hamandcheese and others at @JoinFAI; @bmgarfinkel's @GovAIOrg; several researchers at @iapsAI come to mind. It is time for more mainstream foreign policy and defense thinkers to internalize the technical progress to date, forecast a range of plausible futures, and develop robust policy options to address them. The United States is no stranger to dealing with threats from non-state actors. But the next chapter of technological development may ask us to do something harder: shape, persuade, deter, and coexist with an ecology of non-human ones.

Wow. Just wow. Neither the United States nor China are prepared for the kind of escalation dynamics we might soon see in cyberspace. The WeChat worm is an extremely serious incident. There is no evidence to suggest it was in any way sanctioned by the U.S. government. Still, I am reminded of an extremely important—and concerning—paper from 2025: @hendrycks, @ericschmidt, @alexandr_wang “Superintelligence Strategy.” Now more than ever, it’s important for both sides to keep an open channel of communication, and err on the side of proactively sharing non-sensitive information about AI’s capabilities.