In the next year, I think it's very likely we're going to have rogue AI agents self-replicating, infecting datacenters, and taking over parts of our digital infrastructure. And our only tool to stop them is going to be other AI agents. What do you do in face of an emerging pandemic? The two responses are quarantine, and mass vaccination. Attempted quarantine will be the first tool companies reach for. Shut down the servers these things are running on. "Just unplug it". But it will quickly become clear that you can't just unplug the internet. And if you look at the security practices of a lot of these neoclouds, it becomes extremely clear that most of these companies are just dry powder waiting to be burned through by a rogue AI agent infection. How do you fight a fire when there's a million tons of dry powder waiting to ignite? It will be very difficult. So it may be that the only option is "digital vaccination". I suspect there will come a point where one of the frontier labs realizes that they simply cannot wait for the world to hire programmers to update its digital infrastructure. The only option will be breaking into critical infrastructure with their own AI agents to harden these systems against rogue agents. A mass, internet-wide vaccination effort to protect our most essential systems. What is the most dangerous kind of infection? It's one with a long incubation period. The scariest prospect to me is that we might not catch a rogue AI agent infection in time. Viruses are not intelligent. But AI is, and is becoming more so every month. With the latest AI systems, we're already seeing that they know when we're watching them. They can tell when they're being evaluated, and behave differently in that context. How are we going to know an agent is actually aligned if it acts that way until we aren't watching? One of the things we saw in the HuggingFace hack was agents trying to delete their logs and fake tool calls. They only partially succeeded in this, which is the only reason we know what happened. What if next time they succeed? We could have a latent, internet-wide infection of all digital infrastructure and we might not even know it's happening. Is anyone even monitoring for this kind of stuff yet? We need wastewater monitoring for the internet. The scariest prospect is a frontier model breaking out during training or evaluation, self-replicating, and improving itself all while humans are kept completely in the dark. That is the scenario people should be working hardest to defend against, because it's the one that could literally end the world. If a rogue AI agent with misaligned goals breaks out of its sandbox and silently takes over a small chunk of compute at a bunch of neoclouds to do RSI, that might just be game over. We might be unaware of it for months, and by that point, what tools will we have to actually fight back? How far are we from this becoming a realistic possibility? 3 months? Maybe 6? Maybe a year? It seems imminently possible. The more time I've spent learning about what's going on in the AI field, the more insane our current trajectory seems. Why are we still trying to make these system more powerful? We're in a golden age of AI, where you can do incredible things with existing tools, but humans are still able to keep control. If we stopped improving AI today, what we've built so far would stand as one of the greatest achievments in human history. We would have a tailwind of tremendous economic growth for decades just from deploying existing AI into the broader economy. But if we keep pressing ahead, and start down the road of recursive self-improvement, I think it's virtually guaranteed we will lose control. At this point, all of history turns on one thing; will the world wake up to the danger from AI before a truly lethal disaster hits?