Goodfire (@GoodfireAI) header
Goodfire (@GoodfireAI) profile picture

Using interpretability to understand, design, and align AI.

San Francisco
Joined August 2024
Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)
We’re announcing an embedded assessment with @OpenAI to advance safety and alignment at the frontier.
18
71
20
744
78,570
Read the full post and explore examples of the communities we found: goodfire.com/research/neuron…
1
12
685
This adds to the evidence that neural networks already contain interpretable structure if we know how to look for it. We can now find communities at scale, which agents can then investigate in depth—another tool on the path towards a fuller understanding of neural networks.
1
12
633
We discover these communities without specifying what they do in advance. Over millions of prompts, we identify influential neurons via gradient-based attribution, connect neurons often attributed together in a graph, and do community detection. We then test for necessity and sufficiency and see what behaviors they support.
1
2
13
732
These communities are both necessary and sufficient for their tasks. Ablating one substantially changes the model’s predictions on its task, whereas ablating everything else in its layers (besides the community and 17 always-on neurons) largely preserves the model’s predictions.
1
2
21
957
Recent work suggests individual neurons are surprisingly interpretable. But how do they work together? We found one example a few months ago: an "addition module" in Llama 8B made of just 28 neurons. Our new unsupervised method finds thousands of similar "neuron communities".
1
1
22
1,144
New neurons research: introducing Neuron Communities. We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date. Work led by research fellow @_igorshilov:
10
42
8
341
9,629
Replying to @DKokotajlo @OpenAI
Hi Daniel, the goal of our first assessment is to detect and debug anomalous behavior in an OpenAI training run using our interpretability techniques, and provide findings to internal OpenAI researchers so they can use them to improve model safety. We’ll mutually agree on what to publish in the future, and OpenAI is supportive of us publishing findings, but a public report is not the primary purpose of the first phase of this collaboration. In terms of team access, we’ll work directly with OpenAI safety and alignment researchers. OpenAI will share a list of relevant researchers for us to speak to at the beginning of the assessment, and we can also request other researchers we want to speak to. OpenAI will have the final choice, but will only withhold for specific reasons, including if it takes resources from mission critical safety and alignment work. On the other questions, we’ll have a dedicated slack channel we’ll use to communicate rather than full slack access. There’s no requirement to record conversations, though note that employee interviews are not our focus.
2
1
27
3,483
Goodfire retweeted
incredibly excited to be partnering with @OpenAI on embedded assessments! we'll be embedding interpretability researchers who will use the @GoodfireAI platform to monitor and debug training runs to improve safety and alignment our primary audience for our assessments are internal OpenAI researchers, but we'll publish our findings as the collaboration progresses
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…
15
13
2
253
8,826
thank you! we have plans for future work, more to come soon
6
2,064
Goodfire retweeted
We recently went viral on X with the claim that neurons are "already clean." But what do we mean by that? Are neurons are a useful basis for interp? We're finally releasing our technical report on LLM neurons--and what we've found is far more interesting than mono/polysemanticity.nitter.cf/GoodfireAI/status/2120…
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety. Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
19
42
4
375
25,627
Investigating neurons could have revealed that they were intractable objects of study. Instead, we find that they are more interpretable than we previously thought. Read the first post in our technical series on neuron interpretability here: goodfire.com/research/neuron…
1
1
1
119
5,636
We deeply investigated thousands of neurons in this work. What did they look like? We are releasing descriptions for 1,500 neurons in Qwen3.6 35B-A3B. Explore here: goodfire.com/research/public…
3
2
2
102
6,415
We also found that scaling agent-driven investigations reliably improves our explanations, measured by simulation scores. And each neuron we describe offers gives us clues to the next, like a crossword. Our understanding of the network compounds.
1
83
4,843
Neurons are predictable. We can generate short, natural language descriptions of neurons' roles. Providing the description to a judge model lets us predict when the neuron will fire, simulate its behavior, and activate it more strongly than anything in our corpus.
3
1
100
4,979
We can interpret a neuron's distinct roles by examining what function it plays in the network. If a neuron responds to both oral-health and marine-ecology, we can separate those roles by understanding the upstream neurons that activate it and the downstream readers that respond.
3
2
113
5,726
Neurons have relatively few computational roles, and we can identify what they are. We find that many neuron poles (positive or negative side of a neuron) have just one role. When a pole has multiple roles, they’re few in number and easy to find.
3
1
113
7,326
Neurons exert most of their influence on the model when they fire most strongly. That is also when they are most interpretable. Ablating the weakest 92% of activations has only modest effects on capabilities, and light fine-tuning fully recovers benchmark performance.
4
1
118
8,609