Center for Human-Compatible AI retweeted
New surprising result:
We trained AI recursively on its own safe generations, while giving it a benign cue. Models became increasingly misaligned over time, giving advice on how to do harm and build biological weapons.
We call this Progressive Misalignment. 🧵👇
Center for Human-Compatible AI retweeted
Adaptive Pluralistic Alignment (APA): A pipeline for dynamic artificial democracy
To be pluralistically aligned in the long-term, AI must incorporate diverse societal values and *adapt* as those societies evolve.
My new research agenda: arxiv.org/abs/2605.01642
Center for Human-Compatible AI retweeted
When people strongly disagree on an issue, can they agree on what makes a good AI response?
We find: yes, more than you might expect!
We present PARETO, a large human study w >200k evals, measuring the Pareto frontier of approval btwn opposing groups on controversial issues 🧵
Center for Human-Compatible AI retweeted
What could it mean for an AI to be "politically neutral”? And can we measure it? New paper + dataset.
We propose a defn that applies to any type of conflict: a neutral response should maximize approval on both sides of an issue, while keeping that approval balanced.
1/🧵
Center for Human-Compatible AI retweeted
We've seen AI models deceive, gaslight, and drive users to psychosis—safety issues that labs didn't anticipate until they caused real harm. We built the first benchmark of these unknown unknown alignment failures and found that OOD detection can help prevent them. 🧵
What if a robot policy weren't a neural net or a test-time chat loop, but a multi-file code repo selected from a Pareto frontier of genetically evolved candidates? RHO moves all its LLM exploration to training time, then runs that repo on scenes it was never trained on. 🧵👇🏽
Center for Human-Compatible AI retweeted
As task horizons grow, LLM contexts can’t scale forever. Self-summarization enables concise, interpretable contexts but at a significant performance cost.
Our solution: isolate and *supervise* the information content of summaries in the form of natural-language belief states 🧵
Center for Human-Compatible AI retweeted
Active Teacher Selection for Reward Learning: now published in TMLR!
Most RLHF systems assume feedback comes from one canonical teacher — but annotators can disagree over 30% of the time. So who should the agent ask for feedback?
Paper: arxiv.org/abs/2310.15288v3
Center for Human-Compatible AI retweeted
My internship work at @CHAI_Berkeley (@UCBerkeley) was accepted to @aistats_conf!
We study how an agent can act cautiously even without a mentor/oracle: when should it act, and when should it abstain to avoid catastrophic failure?
📄Paper: arxiv.org/abs/2510.14884
🧵
Center for Human-Compatible AI retweeted
How to elicit truth from models that may be mistaken❌ or deceptive😈? In our @CHAI_Berkeley paper @iclr_conf, we reward each model by how much its answer helps predict the others'.
With weak supervision from a 0.14B LM, it enables anti-deception training on a 8B LM and overwhelmingly outperforms LLM-as-a-Judge.
This technique, peer prediction, is adapted from the mechanism design literature, where it's known to be incentive-compatible, i.e., incentivizes honesty. The intuition is that, predicting mistakes/lies when you know the correct solution is relatively easy, while the opposite is asymmetrically hard.
We are able to further show that, with a large and diverse pool of models, peer prediction incentivizes honesty even when the supervisor doesn't know the models' prior beliefs and motivations.
Center for Human-Compatible AI retweeted
What if an AI could learn to hide its thoughts?
We show that LLMs can learn a general skill to evade activation monitors, with 0-shot transfer to unseen deception/harmfulness monitors from the literature.
We call these "Neural Chameleons". A thread on our new paper. 🦎🧵
Center for Human-Compatible AI retweeted
Our NeurIPS 2025 paper extends adversarial learning (adversarial examples, self-play, etc.) beyond zero-sum games by solving "self-sabotage". 🧵👇
Center for Human-Compatible AI retweeted
The Global Call for AI Red Lines is live!!
More than 200+ former heads of state, Nobel laureates, and other respected thinkers and leaders, and 70+ organizations are together calling for “do not cross” limits re: AI’s most severe #risks
Center for Human-Compatible AI retweeted
*New AI Alignment Paper*
🚨 Goal misgeneralization occurs when AI agents learn the wrong reward function, instead of the human's intended goal.
😇 We show that training with a minimax regret objective provably mitigates it, promoting safer and better-aligned RL policies!
Center for Human-Compatible AI retweeted
We built an AI assistant that plays Minecraft with you.
Start building a house—it figures out what you’re doing and jumps in to help.
This assistant *wasn't* trained with RLHF. Instead, it's powered by *assistance games*, a better path forward for building AI assistants. 🧵
Center for Human-Compatible AI retweeted
(1/7) New paper with @khanhxuannguyen and @thetututrain! Do LLM output probabilities actually relate to the probability of correctness? Or are they channeling this guy: ⬇️
Center for Human-Compatible AI retweeted
🚨Our new #ICLR2025 paper presents a unified framework for intrinsic motivation and reward shaping: they signal the value of the RL agent’s state🤖=external state🌎+past experience🧠. Rewards based on potentials over the learning agent’s state provably avoid reward hacking!🧵
Center for Human-Compatible AI retweeted
Preference learning typically requires large amounts of pairwise feedback to learn an adequate preference model. However, can we improve the sample-efficiency and alignment ability of preference learning with linguistic feedback? With MAPLE🍁, we can! (AAAI-25 Alignment Track)🧵
Center for Human-Compatible AI retweeted
(1/5) New paper! Despite concerns about AI catastrophe, there isn’t much work on learning while provably avoiding catastrophe. In fact, nearly all of learning theory assumes all errors are reversible. Stuart Russell, Hanlin Zhu and I fill this gap: arxiv.org/pdf/2402.08062




























