Sida Liu (@liusida2007) header
Sida Liu (@liusida2007) profile picture

A Learner in the Complex World

Singapore
Joined August 2015
Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)Photo by Sida Liu (@liusida2007)
To the Bottom, down to every numerical values and mathematical operations, open up all the black boxes. liusida.com/post/to-the-bott…
1
1
3
234
Inspired by @GoodfireAI's recent post, I computed MLP neurons' firing contributions and visualized them in a transformer lattice. An interesting pattern: at L8, "John" contributes more, but from L9 onward, "Mary" takes over. Desktop view: liusida.com/research-notes/g…
1
1
36
1,497
Added three more examples to the end of the note. Qwen3.5 is different, including DeltaNet and SwiGLU.
37
Thank you @neuronpedia , for powering ICA Lens! Anyone can use its ICA Lens page to decompose tokens in their own prompt: neuronpedia.org/qwen3.6-27b/…
3
11
748
Nice to see The Bitter Lesson has been upgraded. Like any algorithm design problems, brutal force is always one of the solutions, but we still need clever methods to make it practical.
This quoted post is unavailable.
3
278
When you hear someone respond to your work with two words "not causal", please don't be sad. Not Everything in Mech Interp Has to Be Causal 🧵 liusida.com/post/not-everyth…
6
7
2
128
5,634
So say more than "not causal"
1
3
190
Steering is not explaining
3
109
Two kinds of causal questions
2
147
What inspection tools contribute
1
190
Outputs are not the only thing worth studying
2
311
Before averaging, one single Jacobian already works. Here is my attempt to explain J-Lens. 🧵 liusida.com/research-notes/s…
6
20
3
261
25,210
Same-position vs Future-position J-Lens. They do different things, while standard J-Lens averages them.
1
4
387
Main reasoning: Link J-Lens to Linear Representation Hypothesis. add direction - change final activation - change token score.
1
4
494
An experiment on what a single same-position Jacobian can do. It can predict the next token better than Logit Lens.
2
1
6
1,024
This video explains our ICA Lens paper well! Enjoy!
check out this intuitive video on why interpretable directions in llms look non-gaussian, how ica finds them in minutes per layer, and the evaluation bug that made ica look weak for 3 years
1
119
What if we can stop at this line and inspect the vectors? Use different Lens, plot the histograms, remove a component. Try llmdbg, an LLM Debugger, available in VSCode! 🧵
8
156
This tool was inspired by @GoodfireAI 's recent work: Neural Networks Are Already Clean.
11
Here’s a concrete example of why I like combining ICA Lens with Logit/J/R Lens. (by @liusida2007) Take “The capital of France is” in Qwen3.5-2B. At the final token, directly decoding the layer-17 post-block residual with Logit Lens, J-Lens, or R-Lens gives mostly underscore tokens. Decompose that same activation with ICA Lens, then inspect the components with those lenses. The top two components have clear city-related and France-related readouts, as shown below. Directly decoding the full residual gives a readout of the combined vector. ICA separates it into components we can inspect individually. That’s the value of combining these tools: structure that’s hard to see in the full activation can become much clearer after decomposition.
SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost? In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs. Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store! But there's another motivation behind this work that I care about even more. I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model? With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior. Interpretability shouldn't end with understanding. It should empower us to act. And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary. 🌹 Try it yourself: pip install icalens pypi.org/project/icalens/ Curious to hear what the mechanistic interpretability community thinks!
3
3
23
1,480
Sida Liu retweeted
SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost? In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs. Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store! But there's another motivation behind this work that I care about even more. I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model? With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior. Interpretability shouldn't end with understanding. It should empower us to act. And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary. 🌹 Try it yourself: pip install icalens pypi.org/project/icalens/ Curious to hear what the mechanistic interpretability community thinks!
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected! This old baseline deserves a second look. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory. We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary! More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions. Try it: pip install icalens pypi.org/project/icalens/ Paper: arxiv.org/abs/2606.11722v2 1/8
10
18
6
185
12,756