We discover these communities without specifying what they do in advance.
Over millions of prompts, we identify influential neurons via gradient-based attribution, connect neurons often attributed together in a graph, and do community detection. We then test for necessity and sufficiency and see what behaviors they support.
New neurons research: introducing Neuron Communities.
We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date.
Work led by research fellow @_igorshilov:
We also found that scaling agent-driven investigations reliably improves our explanations, measured by simulation scores.
And each neuron we describe offers gives us clues to the next, like a crossword. Our understanding of the network compounds.
1
83
4,842
Neurons are predictable. We can generate short, natural language descriptions of neurons' roles. Providing the description to a judge model lets us predict when the neuron will fire, simulate its behavior, and activate it more strongly than anything in our corpus.
We can interpret a neuron's distinct roles by examining what function it plays in the network.
If a neuron responds to both oral-health and marine-ecology, we can separate those roles by understanding the upstream neurons that activate it and the downstream readers that respond.
Neurons have relatively few computational roles, and we can identify what they are.
We find that many neuron poles (positive or negative side of a neuron) have just one role. When a pole has multiple roles, they’re few in number and easy to find.
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety.
Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
We investigated this with both theory and experiments. Probes can break, but predictably: the concept usually moves rather than disappears. In our language model experiments, retraining a probe was enough to recover the signal.
In the environments we tested, this allowed us to prevent reward hacking. A model trained with RL learned to cheat by reading a forbidden answer file. Penalizing its internal “cheating” representation raised its honesty rate from 0% to 96-98%.
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
“I think our job is to sort of try to think like the neural network.”
@nickcammarata's interview with @techreview today on why he feels hopeful that neural networks can be understood:
An exclusive interview with Nick Cammarata, the researcher whose 26,000-word viral blog post has AI safety folks celebrating. trib.al/2pWmDvF
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
We discovered a new class of Alzheimer’s biomarkers with @PrimaMente. Today we're taking the next step by opening a global AI competition to discover new treatments.
Introducing the Alzheimer’s Translation Challenge, led by @PrimaMente and @AlzData. 🧵
.@farairesearch red-teamed our monitor with a static battery of attacks. In this testing, our monitors cut successful universal jailbreaks to 0, and reduced total jailbroken interactions by 97%.
Compared to an LLM judge alone, our monitor:
- Pareto dominates on recall/FPR
- uses 50x less compute, costing <$200 to monitor 1M exchanges
- cuts added latency to virtually zero
We built cybersecurity monitors for Kimi K3 and GLM 5.3 — more accurate, 50x faster, & 50x cheaper than an optimized LLM judge.
External red-teaming from @farairesearch found that our monitor greatly reduces universal jailbreaks. 🧵
Manifold steering, SAE geometry, and a model that ignores its own clock…our researchers are at @COLM_conf this week to present papers on all three.
Meet them at our booth, Tue-Thu, 1:30-2:30pm. Poster times and happy hour RSVP below 👇













