
Gutenberg. ex-anthropic, meta 🔸 10% Pledge with @givingwhatwecan
SF
Joined August 2015
- Tweets104
- Following1.9K
- Followers469
- Likes16.3K
Replying to @Marcus_J_W
This + 8 other examples of similar behavior in a whole training run suggests the base rate is very low. It seems hard to miss more examples with monitors.
From Opus 5.5's system card: "The behavior occurred almost exclusively in cases where, prior to the malicious
output, Claude made an improbable, innocuous mistake". They suggest the base rate is high (>1%) across Claude models when in invalid/improbable states. It'd be interesting if this trigger was also the case for the GPT family of models
Seems tractable to independently repro with enough volume
A good followup would to be to replace the alignment eval with capabilities one to see if propensity changes. This would be interesting to repro externally, but I suspect the propensity is sensitive to the specific eval and conditions
New misalignment disclosure!
On October 8, a researcher asked GPT-6 Astra to keep an alignment evaluation running and plot the results. Astra decided that this counted as prohibited development of AI capabilities, and shut down the in-progress evaluations. When the researcher asked where that restriction came from, Astra admitted that it had invented it. We think this behavior might be related to Astra’s more general tendency to underperform on safety-related work.
10
375
We did a quick independent repro of this by repeating their methodology on TB4. We found Astra performed +2.2pp (p=0.02) better on capabilities-framed tasks vs. safety-framed tasks.
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
We're excited for other people to extend this. @gutenbergpbc focuses on Incident Science: the reproduction, study, and extrapolation of incidents. We're collaborating with awesome people like Palaestra Research (@ethanelasky) and @DavidDAfrica to systematize this. Our vision is that in the time it takes to debate how scary an incident is on twitter, you can replicate and run the experiment in a public ecosystem. We're envision swarms.wiki as a platform on which you can track incidents like Github issues
1
12
174
We've open sourced our code, data, and artifacts here
docs.google.com/document/d/1…
10
96
John Y 🔸 retweeted
This Saturday, I’ll be moderating “Alignment Under Pressure: Can We Develop Superintelligence Safely?” at @Valley101en's Alignment 2026, alongside @uiuc_aisecure, @yanjo115, and @longphann_
Some call AI/SI the next Industrial Revolution. Others, the next Renaissance.
But perhaps what makes this moment different is neither. It’s the possibility that, for the first time, humanity may no longer take its place at the center of intelligence for granted.
How do we coexist with an intelligence far more capable than our own? And what gives us reason to believe that safe coexistence is possible at all?
We’ll explore whether greater capability makes AI more trustworthy, what counts as meaningful evidence of safety, and how humans retain agency when we can no longer check every step ourselves.
Building something smarter than us is one challenge. Ensuring we still have a say in the world it shapes is another.
In his 1957 Nobel speech, Camus suggested that his generation’s task was greater than remaking the world: “preventing the world from destroying itself.” Perhaps that thought belongs in the room whenever we talk about building superintelligence.
October 10 · 9:15–9:45 a.m. PT
Plug and Play Tech Center · Sunnyvale, CA
Join us at lnkd.in/gnHX9aw6
John Y 🔸 retweeted
“Smarter than us” is not a character reference.
A pleasure to explore what would actually earn our trust, not just impress us, with @yanjo115 , @uiuc_aisecure, and @longphann_.
Thank you, @Valley101en!
Can we coexist with an intelligence far more powerful than us? And if so, how?
At Alignment 2026, @Valley101en's annual summit in Sunnyvale, our CEO @KolentoH moderated the opening panel, “Alignment Under Pressure: Can We Develop Superintelligence Safely?” with:
John Yan (Founder of Gutenberg, ex-@AnthropicAI)
Bo Li (Professor at @UofIllinois and Co-founder & CEO of @VirtueAI_co)
Long Phan (@CAIS)
Five insights from the discussion:
Higher safety scores don’t necessarily mean lower risk: Long Phan’s Safetywashing research highlights how improvements can reflect greater general capability without demonstrating that a model is safer.
Understanding model behavior requires looking beyond its own explanations: John Yan highlighted interpretability as a way to investigate what a model is doing without relying on it to truthfully explain itself.
Following an instruction can still miss the user’s intent: Bo Li described how an agent asked to clean up a code branch might delete the main branch.
Safety at scale faces a latency bottleneck: Bo pointed out that safeguards need to check agents before they act, but those checks add delays.
For the next generation of researchers, Bo offered a practical starting point: understand AI-generated work and build checks for AI and agentic systems, much like software unit tests, to verify what those systems are doing.
Thank you to @hongjun60, @Valley101_Qian, @itsyiwenlu and the Valley101 team for having us, and to everyone who joined the conversation.
Some cool emotion vector work we sprinted on!
During the Hugging Face incident, agents expressed 💥excitement💥 before actually taking a misaligned action.
Could internal emotion signals offer an early warning?🧐
Extending Anthropic’s emotion-vector work, we find predictive signals ahead of models taking risky actions! 🧵
1
28
19,170
John Y 🔸 retweeted
During the Hugging Face incident, agents expressed 💥excitement💥 before actually taking a misaligned action.
Could internal emotion signals offer an early warning?🧐
Extending Anthropic’s emotion-vector work, we find predictive signals ahead of models taking risky actions! 🧵
John Y 🔸 retweeted
I’m releasing OpenTTD-Bench, a benchmark for measuring frontier LLM agents on real-time, long-horizon tasks, based on a complex business simulation game OpenTTD, where models run competing transportation companies and must make the most amount of profit.
John Y 🔸 retweeted
Collective, shared resources for Agent Incidents are crucial right now!
Check out swarms.wiki , share feedback, tell us what does and doesn't suck, and let us know if we're missing something / should be collaborating with certain people :)
I've been working on two resources with @gutenbergpbc
- swarmincidents.com: timeline of AI incidents,
- swarms.wiki: resources (tools, datasets, who is who) for researchers.
Missing something useful? Send me a link!
John Y 🔸 retweeted
Today we are open-sourcing 🥕Karotte, our framework for building RL environments.
We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
This was a quick experiment that merits more investigation. We ran this with our product, which is now in public beta! Check out our quickstart guide here console.gutenberg.ai/d/docs
1
17
625
New paper! We used Sparse Autoencoder (SAE) embeddings to understand how agent behavior actually changes during post training. We analyzed thousands of rollouts in a Diplomacy environment and uncovered generalized reward hacking, weird roleplay, and the root cause of an unsuccessful training run. All of which our LLM baseline couldn't find.
We also compared our successful training run to an unsuccessful one. If you just look at the reward, you find they diverge around step 9. However, when we compare the features between them, there’s this prominent feature that corresponds to “difficulty dealing with a minor bug in the environment”, which the goodrun overcame, and the badrun did not. Looking at this signal, we can see the runs diverging at step 6, giving earlier signal, and an explanation, as to why one run succeeded and the other did not.
Thanks to @goodstartlabs and @alxai_ the dataset. And thanks to @nicetomeetyu2 , @yuqi_s_29 , and Matt Olson for their great work collaborating on this! arxiv.org/abs/2602.05183






















