How agents learn to cooperate, and what it means for alignment. Co-founder @softmaxresearch, @PluralityInst, @OpenBallotApp. Prev Google, Facebook, Asana.
San Francisco, CA
Joined March 2009
- Tweets108
- Following526
- Followers346
- Likes22
Pinned Tweet
1/ We wrote a map of AI alignment as of October 2026: 17 approaches, who works on each, who funds them, and where each is weak as we see it. Our own approach gets the same treatment. With @eshear at @softmaxresearch. Corrections welcome. softmax.com/blog/the-field-o…
Democracy is a multi-agent alignment protocol that runs on humans. Slow, noisy, mostly works. Worth understanding properly before we invent new ones for AI.
There are plenty of voter guides already. @OpenBallotApp doesn't add another. You follow the ones you trust, see where they disagree, and make your own call. Now covering SF, Oakland, Berkeley, San Jose, LA and San Diego. openballot.app
Scaling laws cover data, parameters and compute. Agents have a fourth axis that mostly gets skipped: how many other learners you train against, and how different they are.
Interpretability asks what a model is thinking. For agents I'd rather know who it thinks it's part of. The "we" an agent identifies with predicts more of its behavior than any single feature.
Control-style alignment has an awkward property: it works best when the model is weak and you need it least. The smarter the subordinate, the more it amplifies mistakes in the controller's frame.
5/ Hand off and Understand first: automated alignment research, which has to trust its first researcher, and agent foundations, which asks what agency is before trying to align it.
6/ Populations: alignment as a property of systems of agents (@jankulveit and ACS Research). This is the group closest to ours. For @softmaxresearch the unit is a relationship, and trust comes from being formed together, not from inspection.
1/ Capabilities and alignment get treated as separate problems. I think one mechanism covers both: train agents in worlds full of other learning agents.
4/ Nature's mechanism for that middle is kinship: shared reward. Ants sit at one end, pure traders at the other, and norms, reputation, shame and love sit between them.
5/ Adults are vastly more capable than their kids and mostly act for the kids' wellbeing anyway. That's the existence proof I'd want alignment people to take seriously. It's what we work on at @softmaxresearch: softmax.com/mission
We don't ask whether a person is aligned. We ask aligned with whom, and we find out over years. Alignment is a relation between agents, not a property you can read off one of them.
We explore this in “Origins of Trust” at @softmaxresearch: where trust between humans and AI might come from, and what that implies for how we train them.
softmax.com/blog/origins-of-…