To the Bottom, down to every numerical values and mathematical operations, open up all the black boxes.
liusida.com/post/to-the-bott…
Inspired by @GoodfireAI's recent post, I computed MLP neurons' firing contributions and visualized them in a transformer lattice.
An interesting pattern: at L8, "John" contributes more, but from L9 onward, "Mary" takes over.
Desktop view:
liusida.com/research-notes/g…
Thank you @neuronpedia , for powering ICA Lens! Anyone can use its ICA Lens page to decompose tokens in their own prompt: neuronpedia.org/qwen3.6-27b/…
Nice to see The Bitter Lesson has been upgraded. Like any algorithm design problems, brutal force is always one of the solutions, but we still need clever methods to make it practical.
This quoted post is unavailable.
3
278
When you hear someone respond to your work with two words "not causal", please don't be sad.
Not Everything in Mech Interp Has to Be Causal
🧵
liusida.com/post/not-everyth…
Before averaging, one single Jacobian already works. Here is my attempt to explain J-Lens. 🧵 liusida.com/research-notes/s…
Main reasoning: Link J-Lens to Linear Representation Hypothesis. add direction - change final activation - change token score.
An experiment on what a single same-position Jacobian can do. It can predict the next token better than Logit Lens.
What if we can stop at this line and inspect the vectors? Use different Lens, plot the histograms, remove a component. Try llmdbg, an LLM Debugger, available in VSCode! 🧵
8
156
This tool was inspired by @GoodfireAI 's recent work: Neural Networks Are Already Clean.
11
Sida Liu retweeted
Here’s a concrete example of why I like combining ICA Lens with Logit/J/R Lens.
(by @liusida2007)
Take “The capital of France is” in Qwen3.5-2B. At the final token, directly decoding the layer-17 post-block residual with Logit Lens, J-Lens, or R-Lens gives mostly underscore tokens.
Decompose that same activation with ICA Lens, then inspect the components with those lenses. The top two components have clear city-related and France-related readouts, as shown below.
Directly decoding the full residual gives a readout of the combined vector. ICA separates it into components we can inspect individually.
That’s the value of combining these tools: structure that’s hard to see in the full activation can become much clearer after decomposition.
SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?
In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.
Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!
But there's another motivation behind this work that I care about even more.
I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?
With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.
Interpretability shouldn't end with understanding. It should empower us to act.
And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.
🌹 Try it yourself:
pip install icalens
pypi.org/project/icalens/
Curious to hear what the mechanistic interpretability community thinks!
Sida Liu retweeted
SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?
In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.
Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!
But there's another motivation behind this work that I care about even more.
I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?
With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.
Interpretability shouldn't end with understanding. It should empower us to act.
And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.
🌹 Try it yourself:
pip install icalens
pypi.org/project/icalens/
Curious to hear what the mechanistic interpretability community thinks!
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalens/
Paper: arxiv.org/abs/2606.11722v2
1/8























