nev retweeted
Staring back at my reflection in the Truth Mines
In some ways, research is like exploring a dual space of ideas. I personally feel like I've only recently become a real 'person' in this space, which I dive into more in a new blogpost.
Substack link: mcnairs.substack.com/p/stari…
lesswrong.com/posts/p6kaX6Lq… I wrote a post with some thoughts I had about the essay!
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
There has been much discussion recently about how interpretable neurons in LLMs are. Today we're announcing our paper from January's cohort of Anthropic Fellows, where we show neurons may be a stronger basis for interpretability than commonly assumed. 🧵 (1/6)
Our findings suggest transcoders add little beyond neurons, and that neurons may be enough for circuit analysis. In the paper: additional evaluations, case studies on linebreaking and arithmetic, extensions to MoE transcoders and SAEs, toy models, and perspectives. (6/6)
Paper: arxiv.org/abs/2610.24521
This research was supported by the Anthropic Fellows Program, White Circle and BlueDot.
Thanks to Ben Thompson, @_gngdb, @CurtTigges, @tomjiralerspong, @Mcn_S7, and @parvmahajan0.
nev retweeted
Today, we're releasing The Circuit Analysis Research Landscape: an interpretability post extending & open sourcing Anthropic's circuit tracing work, co-authored by @Anthropic, @GoogleDeepMind, @GoodfireAI @AiEleuther, and @decode_research.
Here's a quick demo, details follow: ⤵️
🧵1/6 SAEs have become a staple of LLM interpretability, but what if we applied them to image generation models?
My recent paper with @dmhook, @Yixiong_Hao, @afterlxss, @Sheikheddy, and @ArthurConmy adapts SAEs to understand the SOTA diffusion transformer FLUX.1 ⬇️
5/6 Steering image generation via SAE features through activation addition is demonstrably effective. However, steering efficacy depends strongly on the initial prompt's relevance to the steered features, indicating localized representation in diffusion models.
1
3
282
6/6 Future research avenues include adapting SAE architectures for spatial coherence and understanding the effects of text/time conditioning and distillation.
Preprint: arxiv.org/abs/2505.24360
Code: github.com/neverix/fae, github.com/kisate/flux-saes-…
3
229
We quantized all Gemma Scopes into 4 bits, reducing memory and storage requirements by about 4 times with ~20% higher variance unexplained. You can try the quantized versions and make your own quantized SAEs with gist.github.com/neverix/79d5…
DM me if you see bugs or have requests









