
phd agent @harvard, graduate fellow @kempnerinst, i work on ai alignment
🇺🇸🇱🇧
Joined February 2025
- Tweets33
- Following47
- Followers47
- Likes739
Hadi Khalaf retweeted
When agreement with the user's framing is overrepresented among high-reward outputs relative to responses that correct the user's premise, optimization makes the model more agreeable.
A (particular) annotator bias is enough for a Bradley-Terry reward to satisfy this condition
arxiv.org/abs/2602.01002
Hadi Khalaf retweeted
On Wednesday morning (#4116) @hskhalaf and I are also presenting our work on AI evaluations using Maximal Lotteries!
arxiv.org/abs/2602.21297
Spotlight at NeurIPS! See you in San Diego 😄
You can check out our paper here: arxiv.org/pdf/2506.19248
How can we improve LLMs without any additional training? 🤔
The standard playbook is using Best-of-N: generate N responses ➡️ use a reward model to score them ➡️ pick the best 🏆
More responses = better results... right?
Well, not exactly. You might be reward hacking!
Instead, you should hedge! 🎯
3
646
The "shady" privacy formula in question is differential privacy...
The 2020 Census was a fraud. The Biden admin used a shady “privacy” formula that scrambled the data and miscounted 14 states. It included illegal immigrants and handed Democrats extra seats. Americans deserve a fair count and I’m fighting to fix it.
322
I wrote a fun little article about all the ways to dodge the need for real-world robot data. I think it has a cute title.
sergeylevine.substack.com/p/…
430
How can we improve LLMs without any additional training? 🤔
The standard playbook is using Best-of-N: generate N responses ➡️ use a reward model to score them ➡️ pick the best 🏆
More responses = better results... right?
Well, not exactly. You might be reward hacking!
Instead, you should hedge! 🎯
Hedging significantly mitigates reward hacking and achieves superior performance with minimal computational overhead in our experiments!
1
3
235
🤝Takeaway: Don't just optimize. Hedge!
Work done with my great collaborators:
@claudiomverdun @alex_oesterling @hima_lakkaraju @FlavioCalmon
Paper: arxiv.org/abs/2506.19248
Website: hadikhalaf.com/blog/hacking
Hadi Khalaf retweeted
It is critical for scientific integrity that we trust our measure of progress.
The @lmarena_ai has become the go-to evaluation for AI progress.
Our release today demonstrates the difficulty in maintaining fair evaluations on @lmarena_ai, despite best intentions.
Hadi Khalaf retweeted
Ever wonder how LLM developers choose their pretraining data? It’s not guesswork— all AI labs create small-scale models as experiments, but the models and their data are rarely shared.
DataDecide opens up the process: 1,050 models, 30k checkpoints, 25 datasets & 10 benchmarks 🧵
Happy to share we received best paper at NENLP workshop at Yale 🥳🥳!
tldr: Current alignment methods give excessive discretion to annotators in defining what good behavior means. This means we don't know what we are aligning to ‼️
We formalize discretion in alignment and propose mechanisms for data curators & model developers to monitor for it.
Paper link below ⬇️
Thanks to the workshop organizers @HanSineng @armancohan (and everyone else who was part of NENLP!)
Work done at @hseas with the fantastic group of researchers @maartenbuyl @claudiomverdun @Lucas_MPaes @caiocvm @FlavioCalmon
Paper link: arxiv.org/abs/2502.10441
1
277















