Goodfire (@GoodfireAI) header
Goodfire (@GoodfireAI) profile picture

Using interpretability to understand, design, and align AI.

San Francisco
Joined August 2024
We discover these communities without specifying what they do in advance. Over millions of prompts, we identify influential neurons via gradient-based attribution, connect neurons often attributed together in a graph, and do community detection. We then test for necessity and sufficiency and see what behaviors they support.
1
2
10
564
New neurons research: introducing Neuron Communities. We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date. Work led by research fellow @_igorshilov:
6
35
8
289
8,039
We also found that scaling agent-driven investigations reliably improves our explanations, measured by simulation scores. And each neuron we describe offers gives us clues to the next, like a crossword. Our understanding of the network compounds.
1
83
4,842
Neurons are predictable. We can generate short, natural language descriptions of neurons' roles. Providing the description to a judge model lets us predict when the neuron will fire, simulate its behavior, and activate it more strongly than anything in our corpus.
3
1
100
4,978
We can interpret a neuron's distinct roles by examining what function it plays in the network. If a neuron responds to both oral-health and marine-ecology, we can separate those roles by understanding the upstream neurons that activate it and the downstream readers that respond.
3
2
113
5,725
Neurons have relatively few computational roles, and we can identify what they are. We find that many neuron poles (positive or negative side of a neuron) have just one role. When a pole has multiple roles, they’re few in number and easy to find.
3
1
113
7,325
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety. Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
53
248
59
1,788
147,360
We investigated this with both theory and experiments. Probes can break, but predictably: the concept usually moves rather than disappears. In our language model experiments, retraining a probe was enough to recover the signal.
3
4
2
123
11,468
In the environments we tested, this allowed us to prevent reward hacking. A model trained with RL learned to cheat by reading a forbidden answer file. Penalizing its internal “cheating” representation raised its honesty rate from 0% to 96-98%.
4
1
145
11,662
We cut through the mystery around interpretability in training. A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities. We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment. But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
43
205
54
1,994
195,430
“I think our job is to sort of try to think like the neural network.” @nickcammarata's interview with @techreview today on why he feels hopeful that neural networks can be understood:
2
10
87
5,886
We identified a gene variant behind hearing loss, a big step for interpretability in medicine. In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic." ~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
8
26
10
209
19,591
We discovered a new class of Alzheimer’s biomarkers with @PrimaMente. Today we're taking the next step by opening a global AI competition to discover new treatments. Introducing the Alzheimer’s Translation Challenge, led by @PrimaMente and @AlzData. 🧵
3
16
2
135
8,520
.@farairesearch red-teamed our monitor with a static battery of attacks. In this testing, our monitors cut successful universal jailbreaks to 0, and reduced total jailbroken interactions by 97%.
2
1
21
1,229
Compared to an LLM judge alone, our monitor: - Pareto dominates on recall/FPR - uses 50x less compute, costing <$200 to monitor 1M exchanges - cuts added latency to virtually zero
1
4
1
34
2,698
We built cybersecurity monitors for Kimi K3 and GLM 5.3 — more accurate, 50x faster, & 50x cheaper than an optimized LLM judge. External red-teaming from @farairesearch found that our monitor greatly reduces universal jailbreaks. 🧵
11
42
15
315
27,514
Manifold steering, SAE geometry, and a model that ignores its own clock…our researchers are at @COLM_conf this week to present papers on all three. Meet them at our booth, Tue-Thu, 1:30-2:30pm. Poster times and happy hour RSVP below 👇
1
14
4
95
8,060
Many biosecurity safeguards are oversensitive to requests that aren’t actually harmful, constraining useful biology research. Our method leverages protein model embeddings for more precise and robust real-time screening, at millisecond speeds.
1
1
20
1,039