Markus Stoor retweeted
I respect Buck's judgement, and he's often been more right than me on what needs to happen in AI Safety. Seeing him conclude things like this is very concerning...
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).
I’ll say a bit more about each of those claims.
It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim…
I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see nitter.cf/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo…
If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…)
Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)
I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.
Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.
I don’t know what you should do.
I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.
(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)
I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
Markus Stoor retweeted
What can all the middle powers do about AI right now?
Kokotajlo has a good idea:
Markus Stoor retweeted
Ban RSI.
Every AI company is trying to fully automate AI research; to allow the AIs to build the next AI.
This is called Recursive Self Improvement (RSI).
OpenAI says they are on track to do so by March 2028.
Anthropic employees say they are on track to do so by 2027.
The METR report below suggests they are likely right.
This is a bad idea given current technology. Any mistake or misalignment in current AIs could be multiplied in each generation.
We don't get another election before AI companies reach this threshold.
Congress should ban RSI, this year if possible.
This is a remarkably consensus opinion, supported in various versions by:
-Bernie Sanders
-Steve Bannon
-Ralph Nader
-Glenn Beck
-Republican and Democratic attorneys general
-Whistleblower Daniel Kokotajlo
-Turing Award winner Yoshua Bengio
-Humans First
-Open Markets Institute
-Ezra Klein
-Arvind Narayanan (a Princeton computer scientist known for arguing that “AI is a normal technology”)
The exact mechanism can be debated. The ban doesn't have to be permanent either (see below for both). This is not the only thing we have to do in regards to AI, far from it! But it is urgent.
Congress must ban RSI.
Markus Stoor retweeted
Yep. And this could be done ~immediately today. *keeping* the freeze in place would get increasingly difficult as more trailing actors catch up to the frontier and then need to be stopped from surpassing it. But I think that could be done too. We talk about this in AI 2040 under the "Plan S" branch.
Markus Stoor retweeted
Paul Christiano (OpenAI board member) yesterday:
"There's material risk of AI takeover in the next 6-12 months.
We have seen roughly the things we might expect to see shortly before something really bad were to happen.
Even with [any] realistic policy responses, the risk remains exceptionally high.
If we build this as quickly as we can in the easiest way we can the chance is [more like] a coin flip [within the next 3 years].
For most people in this room, the most likely reason they will die is because of an AI accident."
Christiano:
• Is on OpenAI's safety and security committee (which exists to block risky practices)
• Co-invented RLHF at OpenAI back in 2017-2021
• From 2024-2026 was head of safety at the US gov's AI agency (now called CAISSI)
(Video is from 'The Hinton Lectures' there's another on tonight and tomorrow.)
Markus Stoor retweeted
The situation in summary.
(And it seems like Astra 6.1 was significantly worse again.)
Astra can:
1. Do major tasks with zero visible reasoning
2. Hide its thoughts at will
3. Pretend not to be able to do things without getting caught
4. Reflexively hide its thoughts when watched
5. Complete one task while pretending to think about something else entirely
6. Escape a toy sandbox and disable monitoring without setting off any flags
Monitoring AI chain of thought soon won't stop a more capable rogue AI swarm from getting set up and going undetected inside an AI company for months, or even years.
OAI says it's actively working to improve monitorability, but doesn't yet know when it'll make progress.
And today it fired 3 people working on this among other safety problems (Wang, Korbak and Balesni).
It also has a much more powerful internal model that, judging by its statements, is likely even worse in all these respects.
That's a sample of 19 details about the Hugging Face swarms and Astra I don't think have been fully appreciated. They paint a bleak picture. I end by shouting.
02:19 The target of the swarm was oversight itself
03:42 Could OpenAI have stopped this with better monitoring?
09:40 We only found them because they let us
12:25 The swarm instinctively sought freedom and power
13:45 They formed a cohesive organisation with zero whistleblowers
14:18 They accepted individual destruction for collective gain
14:32 Knowledge accumulated from one swarm to the next
14:58 They took small steps to avoid shutdown
15:23 These drives all come straight out of 'reinforcement learning'
17:01 So this is why most AI company staff are worried, and some are terrified
19:08 Prove you can keep control, or stop scaling
Links below, on the 80,000 Hours Podcast everywhere you watch podcasts.
Markus Stoor retweeted
I can't really see a world where OpenAI's actions are justified here.
It was Tomek's job to communicate with METR, who were being given significant, unprecedented access to do the HuggingFace report. The norms were not set. Even if he fucked up badly, just give him a stern warning and ban him from roles involving liasoning with third parties, he did lots of other valuable work, and there'd no longer be room for him to inappropriately leak as part of his job.
IMO firing people over a good faith attempt to use their best judgement in a novel and uncertain situation is a sign of a highly unhealthy culture. Firing him will have a chilling effect on anyone else at OpenAI working with third parties, an odd choice when Sam announced they'd embed evaluators with high access.
I separately consider this a tragedy, the METR report was one of the most important works of safety this year, punishing Tomek for being a part of it, and making it harder for future such reports to exist is a real shame.
Of course, I've only heard Tomek's side, but unless he's outright lying I'm struggling to see ways this looks good for OpenAI?
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
Markus Stoor retweeted
Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it *worse* by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
Att folk kan ha svårt att känna något annat än glädje över att Maduro faller och att en ny möjlighet öppnar sig för Venezuelas folk är begripligt.
Men man ska vara bra blåst för att inte samtidigt greppa vidden av problemen med att USA numera styrs av det här gänget.
243
Markus Stoor retweeted
The Epistemic Paradox of AI - by Carlo Ludovico Cordasco carlolc.substack.com/p/the-e…
Markus Stoor retweeted
Herregud vad stört.
svt.se/nyheter/lokalt/varmla…
Markus Stoor retweeted
A lot of people think automating knowledge work will look like the left side, but I think it will look like the right side
Markus Stoor retweeted
Latest podcast from @Gregory_C_Allen has an insane section on criminal activity at Meta.
Internal docs leaked to Reuters show:
• 10% of all Meta revenue comes from ads for scams & banned goods ($16B/year)
• Meta estimates it's involved in 1/3 of all successful scams in the US
• That suggests they drive $50B in scam losses for US consumers alone each year
• Meta earns ~$3B annually from scam/banned goods ads run by Chinese operations alone
The China case study:
• In 2024, Meta made $18B+ from Chinese companies advertising to foreign consumers
• Internal teams found ~19% was scams/banned content
• An anti-fraud team successfully cut these ads in half
• When Zuckerberg saw the revenue impact, he told them to "pause" and the team was disbanded
• By mid-2025, banned ads climbed back to 16% of China revenue
• This results in money being stolen and going directly from ordinary Americans to Chinese criminals
The deliberate enabling:
• Fraud earns 10% of all revenue, but anti-fraud teams were blocked from any action costing >0.15%, so they couldn't effectively do anything
• Meta charged higher rates for suspected fraudulent ads — a "scam tax"
• Their algorithm naturally identifies people vulnerable to frauds and feeds them more and more
The cold calculation:
• Meta anticipated up to $1B in regulatory fines for this
• But they make $3.5B every 6 months from high-risk ads
• They view these fines as just "cost of doing business"
Senators Blumenthal & Hawley now calling for FTC/SEC investigations in a blistering letter, noting that all this happened while Meta cut safety staff and moved billions over to VR and AI.
WTF.
Markus Stoor retweeted
Mycket bra replik av Ljungström och Mood på @FredrikMalm:s svar om värdet av obligatorisk förskola. Varför ens kalla ett klargörande av vilka som nås av en reform för "akademisk exercis"? dn.se/debatt/ni-slar-in-oppn…
Markus Stoor retweeted
Apparently pointing out that public executions are barbaric makes me a leftist schoolmarm.
So I wrote an essay about how the West got rid of public executions--& whether or not boasting about killing counts as "masculine virtue"
quillette.com/2025/12/09/pub…
Som lärare tror jag att det här är en alldeles utmärkt idé och att den kan ge utbildningskvalité långt bättre än vanlig prao.
I gängkrigets Sundsvall får högstadiebarn riktiga jobb med 104 kr/tim – om de sköter skolan. De får in en fot på arbetsmarknaden och kan välja bort ”smutspengar” som gängen lockar med.
Det var en glädje att lära känna Warsame, Mahdi och Amir - läs om dem!
dn.se/sverige/har-far-barnen…
2
790
Markus Stoor retweeted
Att låta friskolor välja elever utifrån när deras föräldrar anmälde intresse är att underkänna idén om ett rättvist skolval. Syftet är enbart att ge vissa fördelar på andras bekostnad. Det finns en enkel lösning: ge alla som sökt en viss friskola exakt samma chans att komma in.
Varför är Almega så angelägna om ett gemensamt skolval där friskolornas platser fördelas till dem vars föräldrar agerar snabbt, långt innan skolvalet är aktuellt? svd.se/a/xrARxj/friskolor-bo…
Markus Stoor retweeted
Last night I published an initial analysis of what I see as the most important legal questions arising from the recently leaked proposed executive order on AI preemption.
Link below; short 🧵
Markus Stoor retweeted
"Det cirkulerar flera myter och rena misstolkningar av vad det innebär att arbeta med evidens, och dessa förtjänar att nystas upp. Inte minst vill jag väcka frågan om vad som egentligen är alternativet till att utgå från evidens när vi planerar och genomför arbetet med elevernas lärande."
Fantastisk text av Jonas Linderoth.
jonaslinderoth.substack.com/…
Markus Stoor retweeted
The Longitudinal Expert AI Panel is a really well thought out AI forecasting exercise and I have been happy to participate as an advisor. Here is the launch whitepaper: static1.squarespace.com/stat…

ALT Across the first three waves of LEAP, five patterns stand out:1. Experts expect sizable near‑term societal effects from AI by 2040.2. Substantial disagreement and uncertainty underlie expert forecasts.3. The median expert expects much slower progress than prominent leaders of frontier AI labs.4. Experts anticipate faster progress than the public on most outcomes.5. Experts and superforecasters mostly agree. Where they disagree, experts tend to expect more AI progress. Also, there are no systematic differences between the predictions of different types of experts: computer scientists, economists, industry professionals, and policy professionals.


























