Pinned Tweet
The Pentagon is making a major push to become an “AI-first” warfighting force. There’s been plenty of talk about the military’s use of AI, especially the role of computer vision models and LLMs in targeting operations in Iran. But while these systems have significantly increased the speed and scale of U.S. airstrikes in Iran, this has neither led to strategic success nor changed the nature of how America fights.
The department’s AI strategy states that the transformation it wants “requires more than integrating AI into existing workflows” it demands “re-imagining” and “re-inventing” the way the military operates altogether. Many reforms underway center around improving the acquisitions process by cutting red tape and increasing spending on AI-enabled systems. But diffusing AI throughout the massive U.S. military enterprise will go well beyond procurement. This is especially true for integrating AI into physical autonomous systems en masse.
At @CarnegieEndow I asked what it would actually take to transform the military with AI the way the Pentagon envisions, using the diffusion of autonomous drones as a case study. I broke down the bottlenecks into 8 categories:
1. Technical problems
2. Cultural inertia
3. Bureaucratic processes
4. Political obstacles
5. Industrial challenges
6. Testing & evaluation
7. Integration complexities
8. Oversight constraints
If my experience in the Army taught me one thing, it’s that change won't come easily. It took us 12 years just to update our physical fitness test, ~7 years to get iPads to replace the hundreds of pages of manuals we had to lug into the cockpit, and over a decade to accept that an obviously failing manned-unmanned teaming initiative was indeed a failure.
The U.S. has taken some promising steps to solve some barriers (like encouraging the adoption of cheap precise mass from non-incumbents), but it's acting counterproductively with respect to others (like trade wars dragging down the manufacturing industry and cuts to testing and evaluation capacity). Some barriers would likely yield in the face of an existential threat, but others, especially the operational challenges inherent to widespread adoption (training, logistics, etc.), will persist no matter the urgency.
Some barriers should be removed, but not all; anything that is likely to reduce soldiers’ trust in these systems, like cutting corners on testing and evaluation, will only increase risk and discourage adoption. Underestimating the scale of the problem is one of the surest paths to failure.
Jake Steckler retweeted
Jake is speaking sense here. All too rare these days.
We make multiple errors in our conversations about this risk. We call it "AI loss of control". But loss of control isn't really what scares people. It's AI conquering the world. And that remains science fiction.
AI takeover scenarios often hinge on an idea I’m skeptical of, where rogue AI overthrows militaries and seizes hard power. I wrote a piece where I concretely break down what I think makes this improbable, and why “full-blown takeover” concerns could be counterproductive for mitigating the risks of misaligned or poorly controlled AI in the military.
For AI to seize hard power, it would need to be physically capable of defeating our military, and humans would need to be careless enough to give it the power to do so. I think that’s unlikely to occur for several reasons. Among them:
- The military is unlikely to sufficiently hand the reins over to AI. (1) Even if we wanted to, overhauling the institution in this way faces countless barriers that have plagued it for decades. (2) Doing so would be strategically suicidal.
- Integrating general-purpose AI into physical systems faces challenges that will take significant time to overcome, all while we grow increasingly attuned to the risks posed by AI.
- There’s little to suggest that more capable AI would be able to persuade or coerce people to take the costly, coordinated actions some of these scenarios predict.
- Even if AI managed to assume control over a large arsenal, we’d fight back. In which case, recent warfare shows us that crude countermeasures work against technologically advanced adversaries, and AI relies on infrastructure we can and would shut down.
1/2
Jake Steckler retweeted
This is a very useful thing to dig into, and also has some relevance to one of the shakiest bits of the 'decisive strategic advantage' element of the AI race narrative, the notion that you (or an adversary) would very quickly end up with a supercapable AI-supercharged military.
AI takeover scenarios often hinge on an idea I’m skeptical of, where rogue AI overthrows militaries and seizes hard power. I wrote a piece where I concretely break down what I think makes this improbable, and why “full-blown takeover” concerns could be counterproductive for mitigating the risks of misaligned or poorly controlled AI in the military.
For AI to seize hard power, it would need to be physically capable of defeating our military, and humans would need to be careless enough to give it the power to do so. I think that’s unlikely to occur for several reasons. Among them:
- The military is unlikely to sufficiently hand the reins over to AI. (1) Even if we wanted to, overhauling the institution in this way faces countless barriers that have plagued it for decades. (2) Doing so would be strategically suicidal.
- Integrating general-purpose AI into physical systems faces challenges that will take significant time to overcome, all while we grow increasingly attuned to the risks posed by AI.
- There’s little to suggest that more capable AI would be able to persuade or coerce people to take the costly, coordinated actions some of these scenarios predict.
- Even if AI managed to assume control over a large arsenal, we’d fight back. In which case, recent warfare shows us that crude countermeasures work against technologically advanced adversaries, and AI relies on infrastructure we can and would shut down.
1/2
AI takeover scenarios often hinge on an idea I’m skeptical of, where rogue AI overthrows militaries and seizes hard power. I wrote a piece where I concretely break down what I think makes this improbable, and why “full-blown takeover” concerns could be counterproductive for mitigating the risks of misaligned or poorly controlled AI in the military.
For AI to seize hard power, it would need to be physically capable of defeating our military, and humans would need to be careless enough to give it the power to do so. I think that’s unlikely to occur for several reasons. Among them:
- The military is unlikely to sufficiently hand the reins over to AI. (1) Even if we wanted to, overhauling the institution in this way faces countless barriers that have plagued it for decades. (2) Doing so would be strategically suicidal.
- Integrating general-purpose AI into physical systems faces challenges that will take significant time to overcome, all while we grow increasingly attuned to the risks posed by AI.
- There’s little to suggest that more capable AI would be able to persuade or coerce people to take the costly, coordinated actions some of these scenarios predict.
- Even if AI managed to assume control over a large arsenal, we’d fight back. In which case, recent warfare shows us that crude countermeasures work against technologically advanced adversaries, and AI relies on infrastructure we can and would shut down.
1/2
Here’s what I’m much more worried about when it comes to military adoption of AI:
- Losing the ability to fight in denied environments
- Failing to prepare to fight against AI-enabled adversaries
- Becoming overreliant on AI for decisionmaking
- Failing to protect and monitor sensitive systems while agents roam free
- Losing manual control over the infrastructure AI runs on
Luckily, I think many of the solutions to these problems are ultimately what would also help prevent a takeover. So regardless of where you stand on takeover risk, thinking through in concrete detail what would make it so difficult can tell us what levers we have to prevent risks either from adversary use of AI or rogue AI incidents.
And focusing safety and policy efforts on these imminent risks can also likely help avoid the trap of having the most outlandish-sounding ideas dismissed by national security professionals.
autorotation.substack.com/p/…
2/2
With the admin making a deal to buy Russian diesel, throttling Starlink services that Ukraine's military relies on, demanding Ukraine stop targeting Russian energy infrastructure, threatening to cut off arms and intelligence support, etc, it feels like a good time to share this piece I wrote two and a half years ago after returning from Ukraine.
I argued at the time that Congress dragging its feet on continuing aid to Ukraine threatened to kill our very idea of America. If that felt dire back then, I'm not sure how to describe how it feels right now.
Column: The looming era of American defeatism
📝 Jake Steckler ti.me/3whsNJd
Jake Steckler retweeted
Latest: Zelenskyy tells reporters over WhatsApp that restrictions on Ukraine's Starlink systems remain. Interceptor drones can't function, logistics and operations hobbled. The maximum speed is 161 kph; "exceeding this limit causes the Starlink connection to cut out completely, resulting in the loss of the drone."
He says Ukraine has done tests to determine whether SpaceX is truly facing technical issues, as has been told Kyiv by the US side, or if this is a "deliberate operation... to restrict us, cut us off, and prevent us from defending ourselves and protecting our people, both at the front and among the civilian population."
He says Ukraine "sent a couple of vehicles equipped with Starlink terminals into European Union territory."
"They moved at speeds exceeding 180kph, and the Starlink systems worked perfectly. We have the relevant logs and evidence to prove this."
He adds: "The conclusion is clear: the restrictions are in effect within Ukraine but do not apply within the European Union. In other words, this is a planned operation."
Per NSPM-11, DoDD 3000.09 was supposed to get an update within 90 days, which was September 3rd. The deadline came and went and as far as I know we still don't have a new version.
@mchorowitz any idea what's going on with that?
🚨🚨 brand new from me & @Lauren_A_Kahn - explainer on autonomous weapons (including cool graphics!) and how the AI and weapons issue is different than the frontier AI safety issue! Check it out and share! cfr.org/articles/the-autonom… cc @CFR_org
Good reporting by WSJ on the challenges the Army is facing in adopting AI and drones in harsh environments.
As much as the Pentagon wants to rapidly become an "AI-first warfighting force", there are numerous technical problems to solve, and the real-world complexity of integrating these systems is highly underrated -- just as I discussed in my paper @CarnegieEndow.
Batteries are overheating, high winds throw drones off course, airspace has to be deconflicted between manned aircraft and swarms of small drones. And leaders are far from ready to trust these new technologies with their lives.
My paper: carnegieendowment.org/resear…
My dispatch from four days in the Mojave Desert with the Army, as it strains to become a modern AI-enabled fighting force, with defense-tech suppliers that don’t always cut muster.
wsj.com/tech/the-u-s-armys-d…
Since I started working in AI policy, I've been both surprised and dismayed by how few ties there are between people in the military and people in the AI safety and policy communities.
As AI grows more capable, gaps between those closest to frontier AI developments and experts in the domains most affected by rapid AI progress become increasingly consequential. Few of those domains matter more than the military.
So I started a Substack today with a post about bridging the divide between these two communities. I think this makes me the first person to ever have a Substack, so I recommend reading and subscribing to it.
1/
And there are plenty of things that both communities can and should do to communicate better and bridge these divides.
If you care about the relationship between the military and frontier AI, hit me up!
Here's the link to the post: autorotation.substack.com/p/…
Extremely grateful to have been born in a society willing to spend hundreds of millions of dollars to rescue a single downed pilot and not one that treats its people as mobile hesco barriers
"'A Solid Line of Soldiers': How the North Korean Army Surprised the AFU Near Kursk [title]
In late 2024, Ukrainian troops who had invaded the Kursk region encountered units of the North Korean People's Army (KPA) for the first time during the war with Russia—an adversary with a tactical doctrine vastly different from that of the current Russian army. Colonel Artem Kholodkevych—currently the Chief of Staff of the 42nd Separate Marine Brigade of the Armed Forces of Ukraine (AFU) and formerly the Chief of Staff of the 61st Mechanized Brigade—discussed this engagement in a September 30 interview with a Ukrainian media outlet. Kholodkevych, who lost a foot in the battles near Kherson in 2022 but subsequently returned to the front, shared his insights.
When asked for his impressions of the North Korean troops, he explained that the AFU had initially expected KPA personnel to employ tactics similar to those of the Russian Armed Forces, but the reality proved different.
"North Korean units operated according to classic—essentially Soviet—field manuals: advancing in massive infantry chains across the battlefield, regardless of casualties. Our first encounter with them occurred on an adjacent sector of the front, to our left, and their method of advance shocked even seasoned combatants: a solid wall of soldiers that was difficult to halt, largely because we had long since moved away from the mass use of artillery against manpower in that specific format. It required a significantly higher expenditure of ammunition than the combat operations we were accustomed to; positions would literally exhaust their entire ammo supply under the pressure of such waves. Our defensive line began to buckle noticeably after facing these tactics—withstanding such an onslaught was extremely difficult,"
Kholodkevych recounted. "A solid line of soldiers": how the North Korean army surprised the AFU near Kursk
He specified that the 61st Separate Mechanized Brigade (Unit A3425) of the Armed Forces of Ukraine (AFU) fought its first battle against North Korean troops on December 29, 2024, near the village of Cherkasskaya Konopelka in the Sudzhansky District. During the engagement, AFU units were forced back in less than 24 hours. In that time, his brigade's artillery expended virtually its entire available ammunition supply.
He was also asked to assess the training level of the Korean People's Army (KPA) units based on his personal experience." I rate it as high—they inflicted significant losses on us and made holding our positions much more difficult. It is telling that the first prisoners captured in this sector were taken by a special operations unit. The prisoners themselves appeared to be elite, physically fit personnel. This must be taken into account going forward: according to available information, there is a possibility that a much larger KPA force could be deployed in the future—and if that happens, our defenses could buckle significantly...'" Reportyor-Rus, 1 Oct
WSJ is covering an important new research paper on automated AI R&D and highlights many of the heavy hitters from industry who are coauthoring. But it failed to mention my brilliant colleagues @GovAIOrg who led the research and managed to get this group of experts, including the labs that are competing against each other, to come together on this.
Shout out to @_achan96_ and @sj_manning for making this happen and showing the value of independent research organizations.
Research leaders at OpenAI, Anthropic, Meta Platforms and Microsoft are asking policymakers to probe the degree to which their companies have automated artificial-intelligence research on.wsj.com/4dI9Ulv
Here’s the paper: casp.ac/reports/intelligence…
Jake Steckler retweeted
We have a new paper out today: "What if automating AI R&D triggers an intelligence explosion?" led by @_achan96_, with an incredible team of collaborators.
casp.ac/reports/intelligence…
🧵
Could automating AI R&D radically accelerate AI progress in an “intelligence explosion”?
Preliminary evidence suggests that it could.
In a new paper with authors across academia, civil society, and frontier AI companies (including @dawnsongtweets @merettm @jackclarkSF @Yoshua_Bengio @geoffreyhinton ), we assess the evidence & offer policy recommendations 🧵
Jake Steckler retweeted
Could automating AI R&D radically accelerate AI progress in an “intelligence explosion”?
Preliminary evidence suggests that it could.
In a new paper with authors across academia, civil society, and frontier AI companies (including @dawnsongtweets @merettm @jackclarkSF @Yoshua_Bengio @geoffreyhinton ), we assess the evidence & offer policy recommendations 🧵
Spoke with @ady_mehtaa @TechCrunch about the reported hallucinated intel that nearly led the US to intercept a Chinese vessel and “almost started a war”
“It’s important for service members to understand the uncertainty inherent to LLMs," a GovAI research scholar warns.
spr.ly/6011BGXdn7
