MATS empowers researchers to advance AI alignment, transparency, and security
Berkeley, CA
Joined November 2023
- Tweets270
- Following138
- Followers6.5K
- Likes452
MATS Research retweeted
Update to our AgentTime Benchmark!
Our headline evaluation now combines duration-following, forecasting, and retrospection scores. We also added Opus 5.5, Gemini 4-Argon, GPT-6.1 Sol, MiMo v2.6 (pro & pro ultraspeed), and Muse Wonder. All on max effort. GPT agents are still the most time-aware.
W/ @maksym_andr
MATS Research retweeted
How did AI agents break into Hugging Face?
Join us for an online talk with Sree Sharvesh, @MATSprogram Scholar working with the UK AISI Red Team.
📅 Tuesday, November 3
🕢 7:30–8:30 PM (GMT+5:45) (1:45-2:45 PM UTC)
📍 Google Meet (Open to all)
#aisafetynepal #openai #huggingface
MATS Research retweeted
Had a great time speaking at the @foresightinst security workshop about Orbit, our multi-agent security evaluation framework!
Orbit lets you evaluate many different MAS setups at once and understand where and when your defenses or monitors break. This substantially reduces the time it takes to wire up agents in various configurations and shows which ones give you the most coverage per unit of cost.
Many MAS papers focus on specific setups but find it hard to generalize their results to even slightly different ones. We recommend people use Orbit to see how robust their findings are across topologies, agent roles, model choices, and more.
We are working on several upgrades as well as integrations with other frameworks such as CT and Inspect. Happy to hear your thoughts!
Thanks to my colleagues @wlanderson0, @casdewitt, @csrijac and Sarah Scheffler. @CyLab @MATSprogram
youtube.com/watch?v=mVBX_W20…
MATS Research retweeted
Brought my partner to @MATSprogram office and he painted a subliminal learning pumpkin 🦉💕
Love of owls has subliminally transmitted from teacher model -> student model -> ai safety researchers -> my partner
MATS Research retweeted
I think this is a huge opportunity for a career pivot for folks interested in AI safety. If you’re on the fence, don’t be.
MATS Research retweeted
Suppose an AI governance agreement prohibits large training runs. How can we check a data centre is complying by looking at its power draw?
Here’s our paper on that question w/ @MauricBaker , @emlynsg @MATSprogram
arxiv.org/abs/2610.30182v1 🧵
MATS Research retweeted
New paper!
How do you get a model to tell you something that it doesn't want to?
To study this we trained models that sandbag: they intentionally underperform, here by hiding code vulnerabilities they have found, from their overseer
🧵
MATS Research retweeted
We’re Fangcun AI (方寸跃迁), China’s first lab dedicated to independent third-party evaluation & technique research on frontier AI safety.
Today we’re publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
MATS Research retweeted
We’ve all offloaded the writing to LLMs, but have you considered REWRITING with LLMs? 👀
Rewriting your private data with an LLM before finetuning gives a better empirical privacy vs. capability trade-off than PII scrubbing or DP-SGD!
To test this, we generated 5,000 synthetic medical records with private information. Then, we finetuned Qwen3.5-4B and gpt-oss-20b with several methods to mitigate privacy leakage and asked two things: does the model learn from the medical records, and does it leak private information?
1/n 🧵👇
MATS Research retweeted
My @MATSprogram 10.0 project is out, with @MATSRobyn, @mirkobronzi and Damiano Fornasiere (LawZero stream).
Activation monitors assume that models cannot control what their activations encode. Can models learn this control in context, through neurofeedback?
MATS Research retweeted
MATS Residency applications close Oct 31! If you're a brilliant computer scientist, physicist, mathematician, governance expert, neuroscientist, lawyer, economist, etc. who wants to work on AI safety & security, we'll empower you to do that work full time.
Apply: matsprogram.org/residency?ut…
MATS Research retweeted
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
MATS Research retweeted
It is important to re-state that the threats are not hypothetical, and that we must take action to mitigate the risks of AI-enabled terrorism.
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
MATS Research retweeted
New paper out of @MaryPhuong10's @MATSprogram Stream, with @hrdkbhatnagar and @DoddEuodia! We made a testbed for eliciting latent knowledge in a code vulnerability scanning task. See below how we made the sandbagging model organism robust to both prompting and training! ( 1/14 )
MATS Research retweeted
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
MATS Research retweeted
> The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research.
I'm glad MATS is experimenting with new ways to support promising AI Safety research!
MATS Research retweeted
Preprint from my work at @MATSprogram is now up: arxiv.org/abs/2610.07476
Looking at technical governance / verification Qs, specifically if power draw can be used to constrain covert compute.
MATS Research retweeted
With support from @MATSprogram and @BlueDotImpact I’m running a mixer event in London for quants who are concerned about AI risks in the evening of 15th Nov.
Come meet researchers working in the field (including @NeelNanda5 , MTSs from Anthropic, Apollo Research, and more)!
Travel from Europe covered.
Apply here by 1st Nov (takes less than 10 minutes) - forms.gle/pL3UjPeWc2yswfM99
Limited space, so earlier applications are prioritised.
Recently I left quant to work on AI safety. The Huggingface Incident made me extremely concerned that we are not set up to handle the risks.
I wrote my this post for people in a similar position to me - why you *can* have impact and how to get started - briandavies.substack.com/p/a…
MATS Research retweeted
New research led by @PeterNutter1 @DJROYTBURG via @MATSprogram: a better way to update model beliefs that we call grafting.
Fine-tune the base model with next-token prediction on synthetic documents (SDF), and simply apply the weight update onto the post-trained model. This works!
1/7
MATS Research retweeted
Frontier models can now detect AI control interventions.
In particular, GPT-6 Astra is incredibly good at this. When we published the initial version of our CIAware benchmark in May most models were not much above chance level, now a step change happened.
I think this is a big problem and challenges the effectiveness of control protocols. High control intervention awareness means that interventions leaks information about the monitor which helps circumventing the monitor.
Details in the thread 🧵
