I know it feels forever ago, but Mistral really did have a moment there.

🚨 TREASURY SECRETARY BESSENT JUST NUKED ANTHROPIC & OPENAI’S “ROGUE AI” IMMUNITY SCAM ON LIVE TV "A sitting employee came out, said there's a 10% chance of an extinction level event. But then the labs also said, take the liability off of our hands. And we will NOT do that. The Hugging Face incident is the responsibility of OpenAI management, NOT a bunch of agents. It is humans who are responsible, not the AI. These labs need to take responsibility for themselves. They can slow down any time they want to." CHECKED.

Imagine paying for a Rivian R1 on paper, but getting a Toyota Matiz underneath the same body. That may be grounds for a class action.

Re "notification mechanism, I can't stop thinking about this point @mattsheehan88 made recently: maybe it should be via fax. Sounds crazy, but read his argument - > Matt: We have had a lot of these crisis communication lines on military issues, and the U.S. complaint is always: Oh, the Chinese side doesn’t pick up the phone when we call them. And that’s a real issue. I think one mitigation to that is — somewhat ironic — not to use a phone but to use a fax machine. >Ezra: Literal faxes? > Matt: Literal faxes. It has a logic to it, too. Because the political system there is not a system of empowered individuals. It’s a system of committees and a system of documents. So when our treasury secretary — someone who feels very empowered on the U.S. side — picks up the phone and is like: Give me some answers, He Lifeng — or other Chinese counterpart — they’re kind of like: Eh, not really ready to give you answers on the fly. Much better to send a document over to their system that they can review, they can bring it to their committee, they can come up with their understanding and response and send something back. So something in that vein, that at least puts a little bit of a safety net on these incidents that — I think something like that is pretty likely to happen in the next year. (from nytimes.com/2026/09/15/opini… )

I appreciate Anthropic’s transparency in sharing this chart but unsurprisingly it has led to speculative interpretations about intelligence explosion and superintelligence. I don’t think the chart implies we’re anywhere close to either. In short, task delegation ≠ task automation ≠ process automation ≠ faster progress ≠ recursive self-improvement ≠ intelligence explosion. Source: anthropic.com/institute/meas… 1) The software engineering precedent: a year ago there were widespread hopes / fears that once AI can write ~100% of the code, software engineering output would explode (SaaSpocalypse! Everyone would create their own SaaS and dump their vendors) and that this would make software engineers obsolete. Since then, many companies and teams have basically hit that milestone, but neither of the assumptions proved true. Turns out we still need humans, and while shipping velocity has increased moderately, improvements in terms of actual outcomes for software users remain unclear. Besides, we’re still getting a better grasp on the negatives: code quality, long-term maintainability issues, and burnout. While the precedent is no guarantee, this should be our default expectation for what happens as the “Automation Level 4” line trends towards 100% — it won’t be a phase change. normaltech.ai/p/why-ai-hasnt… 2) The “production-progress paradox” is the fact that individual researchers’ productivity has been increasing while the rate of collective scientific progress has been slowing by most measures. AI exacerbates this because everyone uses the same or similar AI models, and ideas become homogenous over time. I suspect it’s too early to tell if this is going to bite companies that are plunging into AI-led research. normaltech.ai/p/could-ai-slo… Note: our own research on AI agents doing open-ended research shows limitations in creativity, judgment, and other areas. cruxevals.com/crux/can-ai-ag…. But it is possible that these could be overcome in the near future, so I’m discounting those limitations here. The production-progress paradox is a deeper issue that’s not AI-specific, though particularly applicable to AI-driven research. It’s about the fact that productivity increases are self-evident but true progress is not measurable as it happens (and only becomes clear in retrospect), so we end up optimizing for the wrong thing. 3) Let’s talk about automation level 5, which is still at 0% in Anthropic’s graph. It’s a bit unclear what Level 5 would look like, but it seems to be about full autonomy at the task level, and not the “AI builds its own successor” vision. My prediction for a while has been that even 100% task automation in most cognitive jobs won’t lead to any kind of discontinuity. youtube.com/watch?v=uiTwQG1Z… What I expect will happen: if anything is understood well enough to be specifiable as a task, it can be handed off to AI, whereas the role of humans is entirely in the interstitial tasks — hard to formalize but still essential. So there will still be a human bottleneck. 4) This human bottleneck is a good thing and is essential for remaining in control. Humans don’t have to be in the loop on every task, but as long as there are enough touch points for oversight in the overall process, and adequate investment in improving human understanding and AI control, increasing AI capabilities doesn’t have to be bad for safety and, more broadly, collective human agency over AI. But “full RSI” is where this balance of agency can break. This kind of closed-loop process is arguably a much more important and tractable target for regulation than compute thresholds, superintelligence, or harm thresholds. I’m glad that OpenAI agrees that fully autonomous RSI may not be a good idea, in a just-released post: openai.com/index/building-st… “Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices⁠ about the benefits and risks. Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand.” 5) Finally, many people have written about why Recursive Self Improvement, even if achieved, won’t necessarily lead to superintelligence. Here’s my argument: normaltech.ai/p/what-will-be… The bottlenecks are external.