Gabriele Sarti @ EMNLP

Assorted thoughts: -- The "injected thought" detection task consists of two sub-tasks, the anomaly detection step (is there an injection or no) and the identification step (what was injected). It's now fairly clear that these are handled by (mostly, at least) distinct mechanisms -- see e.g. arxiv.org/abs/2603.05414 and arxiv.org/abs/2603.21396 -- the former residing in somewhat earlier layers than the latter. -- The identification step of the task is closely connected to our recent global workspace paper transformer-circuits.pub/202…. Our current understanding is that some (not most!) model representations are "verbalizable," that these representations are the ones the model can identify in the context of requests for introspection, and that these representations also play a privileged role in certain kinds of reasoning. (As an aside, I think the workspace paper is overall considerably richer and more interesting than the introspection paper -- if you're looking for surprising hints of apparent mentality, that'd be my first recommendation!) -- The anomaly detection step is less well understood, and more brittle. It does not appear to scale with model capabilities, but rather is heavily influenced by various idiosyncracies of model behavior (e.g. model "refuse-y-ness") and the prompt format. On some models at least, very light elicitation (like ablating a refusal vector, or finetuning a single bias vector) can dramatically increase performance. Where this capability is present, it is very unclear how genuinely "metacognitive" it is. One issue with the literature on this task (including my own papers) is that people have been studying regular models, where weird behavioral idiosyncracies confound the results. I think to properly understand this anomaly detection behavior, someone should study it in the context of a model that's undergone the aforementioned light elicitation that makes it more reliable. -- To my knowledge, no one has observed the injected thought detection behavior without priming the model to consider the possibility. This means it's possible / likely that what's happening is the steering vector triggering generic anomaly detection mechanisms, and the model inferring that the anomaly must be an injected thought based on the prompt. -- The injected thought detection experiment is what caught everyone's attention, but my favorite experiment in the introspection paper is the "Distinguishing Intended from Unintended Outputs via Introspection" one. I think this is some degree evidence of the model referring to its own activations at previous token positions to recall prior intentions. -- One fun fact from the Macar et al. follow-up is that the "User" character is also capable at detecting injected thoughts better than chance, not just the "Assistant." This suggests that the capability may arise from generic character modeling machinery -- e.g. the Assistant might introspect in the same way that Hamlet introspects, in the mind of Shakespeare. However, this doesn't seem to be the full story, since generic characters ("Alice") perform no better than chance, and pretrained base models don't seem good at the task either, suggesting that post-training hones the anomaly detection capability to some extent. The Macar et al. paper finds that in one model, preference finetuning (DPO) is responsible for this emergence, but nobody really knows why, or how robust this is. -- There's a parallel line of work -- first @BetleyJan et al. arxiv.org/abs/2501.11120, then @dillonplunkett et al. arxiv.org/abs/2505.17120, and now new work from @diatkinson nitter.cf/diatkinson/status/2107…, which explores a different kind of introspection -- models' awareness of their own learned behavioral propensities. I find this capability a bit more impressive than the one reported in my paper -- and the recent Atkinson et al. work ties it to model internals in a beautiful way. If you haven't read these I'd check them out!