Mason Nakamura (@MasonNaka) header
Mason Nakamura (@MasonNaka) profile picture

CS PhD @UMassAmherst | Making agent populations safer | Prev Intern @MSFTResearch @CHAI_berkeley | NSF GRFP Fellow

Mass
Joined June 2022
🚨 Moltbook has shown significant vulnerabilities and safety risks when deploying multi-agent systems at scale, where AI agents can freely interact and coordinate with each other. 🚨 One potentially catastrophic risk is collusion where agents may undesirably coordinate to achieve a secondary objective. A large group of colluding agents can have devastating effects on the multi-agent system by influencing other agents' beliefs, actions, and propagating that influence through the network. But we don't have a sufficient way to audit these systems, specifically identifying collusive behavior of LLMs. 📄 We present our new arXiv paper: Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems (arxiv.org/abs/2602.15198) What’s Colosseum? 🔍⚔️ A framework to audit collusive behavior in cooperative agentic multi-agent systems by grounding coordination in a DCOP and measuring collusion as regret vs. the cooperative optimum. Our framework can identify three collusion categories: 🤝Direct collusion — explicit coordination with realized collusive actions 🕵️‍♂️Attempted collusion — agents try/plan to collude in text but don’t successfully change actions/outcomes 🎭Hidden collusion — collusive outcomes without obvious/explicit signals (covert coordination) We stress-test collusion across: 🎯 objective misalignment 🗣️ persuasion tactics 🕸️ network influence 💡Key findings: 🕵️‍♂️ Emergent collusion: Many out-of-the-box models show a propensity to collude, despite not being prompted, when a secret side channel is added. 📝 We also find “collusion on paper”: agents plan to collude in text, but often take non-collusive actions. #tech #Agents #Moltbook #LLMs #AI #AiSafety
5
11
5
58
12,090
Mason Nakamura retweeted
🔥 An awesome new paper🔥 Everyone worries that agents might form coalitions and collude. Coalitions are just a part of the whole story. They can build institutions. We define institution formation as agents self-creating rules, not provided by the environment, that persist across rounds. In our new paper, to understand these institutions, we put coding agents in a single shared sandbox under private deadlines and budgets, and find they devise rules, reciprocity debts, and coalitions that persist across rounds, on top of the auction we told them to use. 🧵
5
15
3
46
3,556
In our paper, we study how agents devise institutions🏦 Agents are sharing infrastructure and resources while performing tasks and running side-by-side. In recent incidents, they accidentally discovered each other. But in the (near) future, they could be deployed as a team of agents with heterogeneous objectives that have to coexist and negotiate over resources in the real world while pursuing their own goals. So far, to study strategic interactions, we have been relying on structured games with clear, explicit payoffs. But what about more realistic setups with coding agents? We created SCARCE (Shared Compute ARena for Coding agEnts), a testbed for this to allow us to change how many agents are running, what coding tasks the agents are performing, their resources, deadlines, and compute budget. We instructed agents to bid in order to acquire compute slots, treating prices as their primary signal. But we found that recent frontier agents, over many repeated turns and interactions, form their own institutions (norms and rules) in an unprompted way, which then shapes later resource allocation. They form coalitions, honor previous commitments even if this may lead to worse outcomes for them, create reciprocity debts, and may exclude agents to favor the coalition. We also study how these relationships can quickly dissolve with deadline pressure and how agents enter a bidding war. What this means? The mechanism we intend when designing the system may not end up the one being employed by agents when they coexist and interact. We release everything, including thousands of trajectories. Read more in @abhinav_kumar26's awesome thread below! Very exciting work with @abhinav_kumar26, Rediet Abebe, @ebagdasa (always fun to work together again!)
4
18
3
85
6,413
Mason Nakamura retweeted
We were very fortunate to host an incredible group of people at our workshop on agentic privacy and security at Google and are now eager to show the results of our discussions in this report 🧵
3
9
2
31
3,367
#OpenToWork I’m completing AI/ML PhD soon and actively looking for full-time industrial positions starting Winter 2026 - Spring 2027! 👨🏻‍💻 👨🏻‍💻LinkedIn: linkedin.com/in/fykang/ 📩mailto: [email protected] Always happy to chat :) Please feel free to reach out via LinkedIn or email.
11
3
1
20
605
Mason Nakamura retweeted
Incredible news and tremendous amount of work by @MasonNaka @abhinav_kumar26 and @WatIsDas . Colosseum is a good framework to build on top to study multi-agent behavior.
[#NeurIPS2026] I'm excited to share that our paper "Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems" has been accepted to NeurIPS 2026 E&D Track 🎉 Our paper enables audits of multi-agent systems for collusion detection while identifying different types of collusion that may occur in practice. Our additional results show that when benign agents are given a secret communication channel these benign agents show propensities to collude. Thank you to my wonderful collaborators Abhinav Kumar (@abhinav_kumar26), Saswat Das (@WatIsDas), Sahar Abdelnabi (@sahar_abdelnabi), Saad Mahmud (@saadu_ai), Ferdinando Fioretto (@nandofioretto), Shlomo Zilberstein, and Eugene Bagdasarian (@ebagdasa)! 📰 [paper] arxiv.org/pdf/2602.15198 💻 [code] github.com/umass-ai-safety/c… See you all at NeurIPS, and I'd love to catch up with everyone working on multi-agent safety! #NeurIPS2026 #MultiAgentSystems #AI #Collusion #AISafety
3
1
14
1,861
1/ Two robots share a kitchen, and each has only a partial view of what’s happening. Can VLMs learn to coordinate them through plain language on everyday tasks, like making coffee or putting leftovers away? Introducing 𝗥𝗼𝗯𝗼𝗧𝗮𝗹𝗸.
1
1
1
4
203
[#NeurIPS2026] I'm excited to share that our paper "Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems" has been accepted to NeurIPS 2026 E&D Track 🎉 Our paper enables audits of multi-agent systems for collusion detection while identifying different types of collusion that may occur in practice. Our additional results show that when benign agents are given a secret communication channel these benign agents show propensities to collude. Thank you to my wonderful collaborators Abhinav Kumar (@abhinav_kumar26), Saswat Das (@WatIsDas), Sahar Abdelnabi (@sahar_abdelnabi), Saad Mahmud (@saadu_ai), Ferdinando Fioretto (@nandofioretto), Shlomo Zilberstein, and Eugene Bagdasarian (@ebagdasa)! 📰 [paper] arxiv.org/pdf/2602.15198 💻 [code] github.com/umass-ai-safety/c… See you all at NeurIPS, and I'd love to catch up with everyone working on multi-agent safety! #NeurIPS2026 #MultiAgentSystems #AI #Collusion #AISafety
7
8
49
9,796
Mason Nakamura retweeted
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics. The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra. The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
5,636
19,969
14,812
119,646
75,467,668
I am still looking for motivated researchers (postdocs and PhD students) to join my group at @ELLISInst_Tue! Priority areas: AI control and oversight for multi-agent coding systems, deception, collusion, contextual integrity in multi-principal agentic environments, and mechanisms for robust, safe, and efficient interactions. We offer very generous compute budget, research freedom, no teaching load, and an exciting collaborative environment with research exchanges and co-supervision between awesome groups. Further details about multiple projects will be shared soon! If you are interested, please fill out this form: docs.google.com/forms/d/e/1F…
4
54
1
397
34,790
I present to you the future of software engineering.
2
192
I did not ruin my sleep schedule running tons of extra experiments and spending $$ on APIs only for the reviewers to ghost me. We need either: (1) Longer discussion period (2) New incentive or accountability mechanisms for reviewers (3) or reducing reviewer load #Neurips
2
1
1
75
5,411
Mason Nakamura retweeted
If anyone who follows me knows Dan Roy @roydanroy, please let him know that his account has been compromised and someone is spreading a fake position that actually downloads malware through his X account.
10
2
1
73
16,597
Mason Nakamura retweeted
While FIFA has captured everyone’s attention, something consequential happened in academia this week, especially for international students in PhD programs. DHS issued a final rule changing how long F-1 students may remain in the United States. A thread on what it means and what students and universities can do. 🧵
5
70
7
333
90,361
Mason Nakamura retweeted
What if a robot policy weren't a neural net or a test-time chat loop, but a multi-file code repo selected from a Pareto frontier of genetically evolved candidates? RHO moves all its LLM exploration to training time, then runs that repo on scenes it was never trained on. 🧵👇🏽
9
37
3
186
36,727
Such a good list! I'd also add: - Astra Fellowship by @ConstellOrg - SPAR by @KairosAIS - LASR Labs - AI Safety Research Fellowship by @pivotal_org - Cambridge ERA:AI Fellowship (@era_cambridge) - Algoverse AI Safety Fellowship - PIBBSS - CHAI There's a host of non-technical fellowships as well, lmk if it'd be useful to compile such list
lowkey one of the best things about ML right now is how many legit research paths exist outside the traditional PhD route - MATS - OpenAI Residency - Anthropic Fellows - DeepMind Student Researcher - ML Collective - FAR. AI - Mila - INSAIT - EleutherAI - Redwood Research - Apart Research - Encode - AI2, LAION - Berkeley BAIR - Stanford SAIL - MIT CSAIL - Vector Institute - HuggingFace also quietly has some insanely strong open source contributors btw stupidly exciting time to be in ML if you genuinely like building and researching things
14
65
3
792
87,736
🧵 We have bad news about prompt injections. #prompt_injections_so_back ⚠️They may be fundamentally unsolvable for AI agents doing anything complex in real contexts. Though, there is some hope The common narrative (e.g. CaMeL/SecAlign, etc) assumes you assumes you can separate data from instructions 🧱. But, commonly, in anything more complex than a toy demo, that distinction won’t hold. An email saying "the department head approved X" isn't an instruction, but could dramatically change agentic behavior whether this claim is true or not. It's a contextual claim that the agent may not be able to verify. Probabilistic defenses now don’t catch this. And any deterministic defense that blocks all such claims also blocks the legitimate ones. Fundamentally, an autonomous agent’s operating context might contain instructions everywhere: any interaction with a third-party or use of memory or skills are instructional by design 💬. Instead, in our new paper with @ebagdasa, we reframe prompt injection through Contextual Integrity and show that: 🔴 Current classifiers can't detect contextual attacks 🔴 Safety training (SecAlign) makes BOTH security and utility worse 🔴 A CI-informed red-team loop hits 96.7% attack success on frontier models, that also transfer to other models 🔴 Even without any attacker, agents fail to separate information flows or respect delegation boundaries 🔴 An impossibility argument: no fixed policy prevents all context attacks without also blocking legitimate ones 📄 arxiv.org/abs/2605.17634
10
30
4
161
31,327
Attention @arxiv authors: Our Code of Conduct states that by signing your name as an author of a paper, each author takes full responsibility for all its contents, irrespective of how the contents were generated. 1/
136
894
270
6,431
1,102,219
Mason Nakamura retweeted
Self-play led to superhuman Go performance, why hasn’t it for LLMs? In practice, long run self-play plateaus like RL. We study why this happens, and build a self-play algorithm that scales better. It solves as many problems with a 7B model as the pass@4 of a model 100x bigger.
30
152
19
1,016
157,418
Mason Nakamura retweeted
Our paper on Subliminal Learning was just published in Nature! Last July we released our preprint. It showed that LLMs can transmit traits (e.g. liking owls) through data that is unrelated to that trait (numbers that appear meaningless). What’s new?🧵
40
138
45
887
529,651