Center for Human-Compatible AI (@CHAI_Berkeley) header
Center for Human-Compatible AI (@CHAI_Berkeley) profile picture

CHAI is a multi-institute research organization based out of UC Berkeley that focuses on foundational research for AI technical safety.

Berkeley, CA
Joined November 2018
Photo by Center for Human-Compatible AI (@CHAI_Berkeley)Photo by Center for Human-Compatible AI (@CHAI_Berkeley)Photo by Center for Human-Compatible AI (@CHAI_Berkeley)Photo by Center for Human-Compatible AI (@CHAI_Berkeley)Photo by Center for Human-Compatible AI (@CHAI_Berkeley)
Center for Human-Compatible AI retweeted
New surprising result: We trained AI recursively on its own safe generations, while giving it a benign cue. Models became increasingly misaligned over time, giving advice on how to do harm and build biological weapons. We call this Progressive Misalignment. 🧵👇
15
61
11
272
19,548
Adaptive Pluralistic Alignment (APA): A pipeline for dynamic artificial democracy To be pluralistically aligned in the long-term, AI must incorporate diverse societal values and *adapt* as those societies evolve. My new research agenda: arxiv.org/abs/2605.01642
8
8
34
3,560
Center for Human-Compatible AI retweeted
When people strongly disagree on an issue, can they agree on what makes a good AI response? We find: yes, more than you might expect! We present PARETO, a large human study w >200k evals, measuring the Pareto frontier of approval btwn opposing groups on controversial issues 🧵
5
22
2
115
17,630
Center for Human-Compatible AI retweeted
What could it mean for an AI to be "politically neutral”? And can we measure it? New paper + dataset. We propose a defn that applies to any type of conflict: a neutral response should maximize approval on both sides of an issue, while keeping that approval balanced. 1/🧵
7
18
7
68
29,070
Center for Human-Compatible AI retweeted
We've seen AI models deceive, gaslight, and drive users to psychosis—safety issues that labs didn't anticipate until they caused real harm. We built the first benchmark of these unknown unknown alignment failures and found that OOD detection can help prevent them. 🧵
5
19
6
73
14,760
Center for Human-Compatible AI retweeted
What if a robot policy weren't a neural net or a test-time chat loop, but a multi-file code repo selected from a Pareto frontier of genetically evolved candidates? RHO moves all its LLM exploration to training time, then runs that repo on scenes it was never trained on. 🧵👇🏽
9
37
3
186
36,727
Center for Human-Compatible AI retweeted
As task horizons grow, LLM contexts can’t scale forever. Self-summarization enables concise, interpretable contexts but at a significant performance cost. Our solution: isolate and *supervise* the information content of summaries in the form of natural-language belief states 🧵
4
13
1
40
5,672
Active Teacher Selection for Reward Learning: now published in TMLR! Most RLHF systems assume feedback comes from one canonical teacher — but annotators can disagree over 30% of the time. So who should the agent ask for feedback? Paper: arxiv.org/abs/2310.15288v3
4
15
49
8,006
Center for Human-Compatible AI retweeted
How do knowledge and meaning change in the age of AI, and what can we learn from silence and art? We explored these and many other deep questions in this amazing event at Pomona last month. youtube.com/watch?v=dG9JuK3S…
1
3
1
7
497
Center for Human-Compatible AI retweeted
My internship work at @CHAI_Berkeley (@UCBerkeley) was accepted to @aistats_conf! We study how an agent can act cautiously even without a mentor/oracle: when should it act, and when should it abstain to avoid catastrophic failure? 📄Paper: arxiv.org/abs/2510.14884 🧵
6
11
2
50
6,228
Center for Human-Compatible AI retweeted
How to elicit truth from models that may be mistaken❌ or deceptive😈? In our @CHAI_Berkeley paper @iclr_conf, we reward each model by how much its answer helps predict the others'. With weak supervision from a 0.14B LM, it enables anti-deception training on a 8B LM and overwhelmingly outperforms LLM-as-a-Judge. This technique, peer prediction, is adapted from the mechanism design literature, where it's known to be incentive-compatible, i.e., incentivizes honesty. The intuition is that, predicting mistakes/lies when you know the correct solution is relatively easy, while the opposite is asymmetrically hard. We are able to further show that, with a large and diverse pool of models, peer prediction incentivizes honesty even when the supervisor doesn't know the models' prior beliefs and motivations.
1
3
18
968
Center for Human-Compatible AI retweeted
What if an AI could learn to hide its thoughts? We show that LLMs can learn a general skill to evade activation monitors, with 0-shot transfer to unseen deception/harmfulness monitors from the literature. We call these "Neural Chameleons". A thread on our new paper. 🦎🧵
11
45
9
232
45,429
Center for Human-Compatible AI retweeted
Our NeurIPS 2025 paper extends adversarial learning (adversarial examples, self-play, etc.) beyond zero-sum games by solving "self-sabotage". 🧵👇
2
21
3
116
20,685
Center for Human-Compatible AI retweeted
The Global Call for AI Red Lines is live!! More than 200+ former heads of state, Nobel laureates, and other respected thinkers and leaders, and 70+ organizations are together calling for “do not cross” limits re: AI’s most severe #risks
3
12
35
2,081
Center for Human-Compatible AI retweeted
*New AI Alignment Paper* 🚨 Goal misgeneralization occurs when AI agents learn the wrong reward function, instead of the human's intended goal. 😇 We show that training with a minimax regret objective provably mitigates it, promoting safer and better-aligned RL policies!
8
31
3
149
20,465
Center for Human-Compatible AI retweeted
We built an AI assistant that plays Minecraft with you. Start building a house—it figures out what you’re doing and jumps in to help. This assistant *wasn't* trained with RLHF. Instead, it's powered by *assistance games*, a better path forward for building AI assistants. 🧵
83
200
109
2,298
490,735
Center for Human-Compatible AI retweeted
(1/7) New paper with @khanhxuannguyen and @thetututrain! Do LLM output probabilities actually relate to the probability of correctness? Or are they channeling this guy: ⬇️
3
5
1
14
1,818
Center for Human-Compatible AI retweeted
🚨Our new #ICLR2025 paper presents a unified framework for intrinsic motivation and reward shaping: they signal the value of the RL agent’s state🤖=external state🌎+past experience🧠. Rewards based on potentials over the learning agent’s state provably avoid reward hacking!🧵
3
30
2
113
16,561
Center for Human-Compatible AI retweeted
Preference learning typically requires large amounts of pairwise feedback to learn an adequate preference model. However, can we improve the sample-efficiency and alignment ability of preference learning with linguistic feedback? With MAPLE🍁, we can! (AAAI-25 Alignment Track)🧵
1
7
12
1,560
Center for Human-Compatible AI retweeted
(1/5) New paper! Despite concerns about AI catastrophe, there isn’t much work on learning while provably avoiding catastrophe. In fact, nearly all of learning theory assumes all errors are reversible. Stuart Russell, Hanlin Zhu and I fill this gap: arxiv.org/pdf/2402.08062
1
6
1
13
1,316