Arnab Sen Sharma (@arnab_api) profile picture

Ph.D. student @KhouryCollege, working to make LLMs interpretable

Boston, MA
Joined September 2022
But the next action is already readable in the code written before. Isn't that what drives the execution? We transfer the task span and the code written so far separately. The task span alone redirects the LM; the progress barely does. The goal is re-read, not carried forward.
1
4
91
🔄 We test it. In the middle of an implementation, we swap what the goal heads read in the task span with another prompt’s key-value states. The model switches to the other prompt's next action, even across programming languages. The GoalScope readout flips with it.
1
4
97
💡 We propose that it doesn't need to keep a single encoding alive throughout action. Under our non-recurrent goal-seeking (𝗡𝗥𝗚𝗦) account the LM frequently re-reads the goal from the task specification, the progress-so-far, and decides the next action by comparing the two.
1
4
109
🤔 However, persistence in LMs is puzzling! A transformer LM doesn't have recurrence; information reaches a later token only through attention, which costs a layer. So how does the LM keep a goal alive for more tokens than it has layers left?
2
4
112
⏳ Implementing such a goal can take many tokens. Throughout the implementation, the goal heads keep this encoding alive, and it fades once the execution is done. ✅ Persistence
1
3
114
We take this encoding from a Python code and plant it into an C++ stub, and the LM implements the 𝘀𝗮𝗺𝗲 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗖++, with different libraries, syntax, and vars. ✅ Plasticity The LM can realize this goal-encoding in different pathways, suited to the env.
1
1
4
208
Inspired by Patchscope (arxiv.org/abs/2401.06102), we call this 𝗚𝗼𝗮𝗹𝗦𝗰𝗼𝗽𝗲. The same goal heads can capture and readout the LM's upcoming actions in various contexts: in code settings with unspecified steps, in few-shot ICL, and under refusal steering or misaligned LoRAs.
1
5
130
🔭 When this encoding is transplanted into a natural language context, 𝘁𝗵𝗲 𝗟𝗠 𝘃𝗲𝗿𝗯𝗮𝗹𝗶𝘇𝗲𝘀 𝗶𝘁𝘀 𝗻𝗲𝘅𝘁 𝗮𝗰𝘁𝗶𝗼𝗻 𝗶𝗻 𝗘𝗻𝗴𝗹𝗶𝘀𝗵! This approach can read out the LM's upcoming action well before its execution starts! ✅ Pre-commitment
1
7
141
In a simple code generation setting, we look for attention heads that carry an abstract encoding of the upcoming operation via causal mediation analysis.
1
5
163
What is a goal? Inspired by Bratman's planning theory, we operationalize “goal” with three functional criteria: pre-commitment, plasticity and persistence. We look for an encoding in the LM's internal computation that satisfies these three criteria.
1
6
198
Do language models have goals? 🎯 When LMs carry out tasks that unfold over thousands of tokens, is it driven by an 𝗲𝘅𝗽𝗹𝗶𝗰𝗶𝘁 𝗴𝗼𝗮𝗹 𝗲𝗻𝗰𝗼𝗱𝗶𝗻𝗴 in its internals? Can we readout and manipulate this encoding? We investigate these questions in our new paper. 🧵
4
16
4
73
2,414
We validate this flag-based eager evaluation hypothesis with a series of carefully designed causal mediation analysis. If we swap this flag with another item in the list, in the question-before context the LM consistently picks the item carrying the flag. However, the question-after prompt doesn’t show any sensitivity to this intervention. Check our paper for full details!
1
6
227
📊 We measure this with a *causality* score: if the predicate is abstractly encoded in the query states of these "filter heads", then transferring it should change the output. For example: in the figure the answer should change to "Peach" (or the changed format accordingly). We test this across a range of different semantic types, presentation formats, languages, and even different tasks that require a different "reduce" step after filtering.
1
7
318
🤔 But do these heads play a *causal* role in the operation? To test them, we transport their query states from one context to another. We find that will trigger the execution of the same filtering operation, even if the new context has a new list of items and format!
1
1
8
542
🔍 In Llama-70B and Gemma-27B, we found special attention heads that consistently focus their attention on the filtered items. This behavior seems consistent across a range of different formats and semantic types.
1
3
1
12
950
How can a language model find the veggies in a menu? New pre-print where we investigate the internal mechanisms of LLMs when filtering on a list of options. Spoiler: turns out LLMs use strategies surprisingly similar to functional programming (think "filter" from python)! 🧵
1
23
6
68
13,199
But, even without attention, we can do a similar experiment: we can block out information flow by mean-ablating the subject/non-subject tokens to *all* the future tokens in different layers to understand how Mamba uses Conv + SSM in which layers relay what information.
1
3
541
How about path-dependent attention knockout experiments? @megamor2 used these to understand factual information flow in transformers. But in Mamba, which has no attention, these methods cannot be applied "out-of-the-box" (Section 6 of our paper)
1
3
583
Then we ask: can we apply ROME to update/insert a fact in Mamba? ROME can be applied directly as well! When applied to different projection matrices in MambaBlock across all the layers, we see that ROME can successfully update/insert a fact in a range of layers.
1
1
3
1,329