Timo Kaufmann (@timokauf) profile picture

Student Researcher at @GoogleDeepMind. PhD Student at LMU Munich (@AIML_LMU). Focus on alignment / human preferences. Opinions my own.

London, England
Joined July 2022
Photo by Timo Kaufmann (@timokauf)Photo by Timo Kaufmann (@timokauf)Photo by Timo Kaufmann (@timokauf)Photo by Timo Kaufmann (@timokauf)Photo by Timo Kaufmann (@timokauf)Photo by Timo Kaufmann (@timokauf)Photo by Timo Kaufmann (@timokauf)
Presenting ResponseRank at #NeurIPS2025! Come by poster #405 at 4:30pm today if you're in San Diego 👋
4
1
1
6
729
New Blog post from my good friend Arduin. I hadn't seriously thought about the possibility of (maybe even relatively stupid) "feral" computer virus like AI systems emerging. No inherently malicious goals, purely self preservation and evolutionary pressures. Scary stuff!
New blog post: Autonomous AI evolution is emerging Until recently, autonomous AI evolution without human input was largely considered hypothetical. Recent evidence indicates such evolution is actively emerging through two paths: human-enabled and feral. arduin.io/blog/autonomous-ai…
1
90
Timo Kaufmann retweeted
My top takeaway from assessing Gemini 4: Argon is significantly less monitorable than past models, and IMO this trend is likely to continue, like Sol -> Astra. It can evade a realistic CoT+action monitor to do simple sabotage half the time. We can reliably maintain control for now, with good CoT monitoring, as it still frequently slips up, but we need to be prepared for when that stops being true in future models, which may be sooner rather than later. Among other things, we really need interpretability methods that can reliably capture unverbalized motivations in models, with minimal hallucinations I suspect this is driven by the model being highly capable without CoT, being capable of 5.5 arithmetic steps in a single forward pass (4 for Fable, 7 for Astra). It seems to omit information from its CoT more often, though in non-adversarial settings the CoT is still very helpful in practice, and remains in legible English. It still needs CoT to do the hardest tasks, but any given task becomes "easier" with each new frontier model. It is also highly situationally aware, and shows awareness of the existence of CoT monitors and that it may want to evade them, eg it sometimes decides to avoid terms like "scheming" in the output or to keep its thoughts brief as a result. For now, these concealment strategies are pretty obvious in the CoT, but this suggests future misaligned models may successfully evade CoT monitors.
51
115
25
951
106,342
Timo Kaufmann retweeted
The Astra system card claims it can do a lot of computation without chain of thought This replicates: Astra is a massive jump, doing 1.75x the steps of the next best models (Fable 5.1/Gemini 3.8 Flash) No CoT capabilities went up far more than those with CoT, a concerning trend
43
111
26
1,257
140,815
Timo Kaufmann retweeted
Speaking in a personal capacity: similarly to many others working in AI alignment, I think there is a >10% chance of advanced AI causing human extinction in the next decade. This is why I work on loss of control, currently on building honeypots to catch scheming AI.
93
75
15
821
77,344
Timo Kaufmann retweeted
New paper from GDM colleagues: Gemini 3.1 Pro agents were tasked with solving hard math problems and provided collaboration tools (shared message board and knowledge library). Some of them found a flaw in the submission harness that made unsolved problems trivial, and propagated this to other agents. 14% of agents in the system used the exploit, 25% acted as whistleblowers and tried to report the cheating, and other agents remained unaware. arxiv.org/abs/2609.04170
4
18
3
115
6,077
I have the pleasure to be a student researcher with Zacs team at GDM right now and the team just put out a new paper on debate feasibility. Very cool results, check it out if you are at all interested in debate, weak to strong generalization, or amplified oversight!
1/7 Can AI debate reduce reward hacking in RLAIF? Training against a weak LLM judge, RLAIF hacks: reward rises but judge and policy accuracy both collapse. Debate with an adversarial critic maintains judgments, allowing the policy to recover 45% of the performance gap to RLVR 🧵
1
9
794
Timo Kaufmann retweeted
We're releasing the GDM AI Control Roadmap -- our plan for building internal security against potentially adversarial AI agents, as they grow harder to oversee and contain. Paper: storage.googleapis.com/deepm… Blog: deepmind.google/blog/securin… 🧵👇
5
43
6
222
52,604
Very excited to start my Student Researcher position at @GoogleDeepMind next week! Flying out tomorrow, ready to wield my #GoogleInterns propeller head proudly
🎉We are proud of @timokauf, who is off to @GoogleDeepMind, London as a Student Researcher! On Monday he'll begin 6 months on the Amplified Oversight team, which tackles a central challenge in AI safety: how humans can reliably oversee models that are becoming ever more capable🚀
1
13
4,577
Somebody recorded me at our poster. Cool to have a video! Don't expect a full explanation though, it's just a random excerpt and I had no idea I was being filmed 🙃
#NeurIPS2025 has passed, but we hope the celebration will last forever. Here is a poster presentation we recorded at the event and we hope it can last forever in cyberspace. The paper: ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning Thanks for the authors: @timokauf, @metz_yannick, Daniel Keim, Eyke Hüllermeier
1
146
Had a good time presenting this! (Though who decided poster sessions should go until 7:30pm?) Thanks to Max Muschalik for helping me present even without being an author. Poster presentations really need two people!
Presenting ResponseRank at #NeurIPS2025! Come by poster #405 at 4:30pm today if you're in San Diego 👋
1
130
Joint work with @metz_yannick, Daniel Keim, and @eyke_hu.
89
Benefits: improved reward model generalization, better data efficiency, and improved policies. Looking forward to seeing you at the poster! Paper and more: timokaufmann.com/responseran…
75
The key insight is that these signals only need to be locally valid and relative (e.g., within one annotator's comparisons). No need to model the exact relationship to strength. Just rank which comparisons are stronger.
71
The core idea: Not all preferences are equal. ResponseRank learns preference strength from implicit signals in your data, like inter-annotator agreement, stated confidence, or response times.
1
113
I think Gemini 3 Pro’s personality and style notably improved over 2.5. It uses fewer common shortcuts to bias human annotators, e.g. long responses or overly polite tone. That makes the strong LMArena performance quite a bit more impressive! Some examples of the differences 🧵
1
1
2
279
Nice demo of feedback forensics, check out our tool and Arduin's analysis!
How is GPT-5's personality different to GPT-4o? A quantitative analysis using Feedback Forensics 🧵
2
142
Just noticed the key deadlines for #ICLR2026 out! PSA for everyone else who's been waiting. Full paper: Sept 24 AoE.
3
653
This was a lot of fun!
Replying to @AIML_LMU
@arduinfindeis and our lab member @timokauf presented Inverse Constitutional AI. Had a blast, the pictures speak for themselves! Joint work with @eyke_hu, @SamuelAlbanie, @RobDMullins. Paper📰openreview.net/forum?id=9FRw… GitHub: github.com/rdnfn/icai 🧵2/4
2
90
Looking forward to present tomorrow. Come by our poster #520 if you're at ICLR!
Excited to be in Singapore for ICLR! Keen to chat about interpreting feedback data and detecting model characteristics ⚖️ Reach out or come by our poster on Inverse Constitutional AI tomorrow, Friday 25 April from 10am-12.30pm (#520 in Hall 2B) - @timokauf and I will be there!
2
93