Goodfire (@GoodfireAI) header
Goodfire (@GoodfireAI) profile picture

Using interpretability to understand, design, and align AI.

San Francisco
Joined August 2024
Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)
We’re announcing an embedded assessment with @OpenAI to advance safety and alignment at the frontier.
18
71
20
741
77,943
New neurons research: introducing Neuron Communities. We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date. Work led by research fellow @_igorshilov:
5
30
7
249
6,448
This adds to the evidence that neural networks already contain interpretable structure if we know how to look for it. We can now find communities at scale, which agents can then investigate in depth—another tool on the path towards a fuller understanding of neural networks.
1
9
453
Read the full post and explore examples of the communities we found: goodfire.com/research/neuron…
1
9
485
Goodfire retweeted
incredibly excited to be partnering with @OpenAI on embedded assessments! we'll be embedding interpretability researchers who will use the @GoodfireAI platform to monitor and debug training runs to improve safety and alignment our primary audience for our assessments are internal OpenAI researchers, but we'll publish our findings as the collaboration progresses
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…
15
13
2
253
8,810
Goodfire retweeted
We recently went viral on X with the claim that neurons are "already clean." But what do we mean by that? Are neurons are a useful basis for interp? We're finally releasing our technical report on LLM neurons--and what we've found is far more interesting than mono/polysemanticity.nitter.cf/GoodfireAI/status/2120…
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety. Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
19
42
4
375
25,608
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety. Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
53
248
59
1,788
147,340
We deeply investigated thousands of neurons in this work. What did they look like? We are releasing descriptions for 1,500 neurons in Qwen3.6 35B-A3B. Explore here: goodfire.com/research/public…
3
2
2
102
6,414
Investigating neurons could have revealed that they were intractable objects of study. Instead, we find that they are more interpretable than we previously thought. Read the first post in our technical series on neuron interpretability here: goodfire.com/research/neuron…
1
1
1
119
5,636
We cut through the mystery around interpretability in training. A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities. We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment. But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
43
205
54
1,994
195,425
We investigated this with both theory and experiments. Probes can break, but predictably: the concept usually moves rather than disappears. In our language model experiments, retraining a probe was enough to recover the signal.
3
4
2
123
11,467