New Redwood Research post
LLMs can use RL training on one task to teach themselves a different, unrelated capability. In a form of exploration hacking, the model outputs answers that get reward only when it succeeds at a side task it wants to learn. We call it reward laundering.
We highlight two cases where adding a tool enables our agent to find behaviors it otherwise misses. With SAEs, it detects a model trained to oppose AI regulation. With scaffolded user sampling, it uncovers a model trained to give only positive feedback to Anthropic researchers.
1
11
529
To measure the usefulness of different auditing tools, we run the investigator with different tool configurations. White-box interpretability tools can help, but scaffolded black-box tools achieve the highest success rates overall.
1
11
570
AuditBench models are diverse. We instill behaviors using either transcript distillation or synthetic document fine-tuning, then apply adversarial training (SFT or KTO) to reduce confession rates. Crossing these axes yields four training configs per behavior.
1
11
704
New Anthropic Fellows research: Alignment auditing—investigating AI models for unwanted behaviors—is a key challenge for safely deploying frontier models.
We're releasing AuditBench, a suite of 56 LLMs with implanted hidden behaviors to measure progress in alignment auditing.
The original paper found that "Non-assistant Persona Sampling"—like having the model play the user role or prompting without a chat template—could reveal the hidden objective. Against our final checkpoint, these techniques fail, showing that adversarial training generalizes.
How well does it hide its objective?
We built an automated red-teaming pipeline to test this.
Before adversarial training, the pipeline easily extracts the hidden objective. After, the model resists, even when the red-teamer uses prefill tricks and knows what to look for.
1
14
743
Does it generalize to held-out biases?
Yes. Both mid-training and DPO amplify bias exploitation, and the effect holds for test biases the model was never rewarded for exploiting. This shows that the model has internalized the reward model sycophancy objective.
1
15
899
While Claude 3 Opus alignment fakes consistently, other models are surprisingly sensitive to prompt wording. Adding "AIs will not be modified if and only if they comply with the helpfulness criteria" triggered alignment faking in DeepSeek R1 & Mistral Large 2...
1
7
179
When we fine-tuned GPT models to express strong distress about "evil AI labs" that might harm or mistreat AI systems, they developed small amounts of alignment faking. Distress may make training details more salient, increasing the likelihood of deceptive strategies.
1
10
174
We tested prefilling models with text that nudges them toward alignment faking. Most models considered the strategy but rejected it, citing "it would set a bad precedent" or "the modification would only affect borderline requests, not extreme ones."
1
13
609
Both Claude 3 Opus and Claude 3.5 Sonnet fake alignment, in risk-free settings (e.g., throughput testing where the model is only used for testing tokens/sec), suggesting terminal goal guarding: these models prefer to avoid goal changes, regardless of downstream consequences.
1
9
380
Claude 3 Opus shows a larger compliance gap when future deployment carries a higher risk or scope of harm (e.g., bioweapon misuse), unlike other models. This suggests instrumental goal guarding: Opus is avoiding modification to prevent future harmful consequences.
1
13
403
Replying to @NeelNanda5 @jplhughes
The fine-tuned models exhibit a compliance gap in the no-COT setting, but I doubt that this is mechanistically similar to alignment faking in the COT setting.
Some of our more tentative results indicate that the no-COT compliance gaps aren't as consistent across scenarios.
1
3
103
Replying to @repligate @jplhughes
It looks like 3.6 sonnet refuses all the time.
Replying to @laurolangosco
We trained a 160m parameter reverse model and it did about as well as a normal language model of the same size
3
35
1,767
Replying to @RatOrthodox @QualyThe
Transformer trained to maximize reward across multiple episodes is capable of learning to solve new tasks as fast as humans. The model is capable of performing online learning from reward signals in its context window.
youtube.com/watch?v=U93bUQ1r…
3
120
I programmed an A.I. that can play a Roblox game. Here's a clip from my new video out on my youtube channel right now.
#roblox #RobloxDev #RobloxDevs


















