PhD student @StanfordAILab @StanfordNLP @Stanford advised by Tatsunori Hashimoto and Tengyu Ma. Prev: @OATML_Oxford @CompSciOxford
Stanford, CA
Joined September 2020
- Tweets170
- Following883
- Followers1.2K
- Likes488
Pinned Tweet
When LLMs are unsure, they either hallucinate or abstain.
Ideally, they should clearly express truthful confidence levels.
Our #ICML2024 work designs an alignment objective to achieve this notion of linguistic calibration in *long-form generations*.
arxiv.org/abs/2404.00474
🧵
Hinting is a really cool new primitive for building downstream task scaling laws and more
Awesome work by @suze_va @LukeBailey181 @rckpudi and team!
Can we forecast benchmark accuracy from models that score zero?
We (co-leads @LukeBailey181 @rckpudi) turn a benchmark into a series of easier hinted versions. Jointly modeling scale C and hint level H enables forecasting GSM8K acc (H=0) within 3 pts from 5600x less compute 🧵
We're benchmarking frontier LLMs on philosophy with a wonderful interdisciplinary team led by philosopher / lawyer / CS PhD @michaelcheng76
If you're a philosophy student or faculty interested in joining the project, check out philosophybench.org!
1/ Introducing PhilosophyBench from @StanfordAILab @StanfordHCI, the first independent, large-scale benchmark for evaluating AI’s philosophical capabilities.
philosophybench.org
Tim's an amazing researcher and mentor, go work with him!
I'm so happy to share that I’ll be joining @UofT as an Assistant Professor of Statistical Sciences and Computer Science, with an appointment at the @VectorInst, in 2026!
I'm recruiting postdocs and PhD students: timrudner.com!
Please help me spread the word!
🧵(1/5)
1
472
A CoPilot for tutors + a rigorous study with 1,800 students showing it actually helps them master concepts better 🔥
Awesome work by @rose_e_wang and team!
Really enjoyed working on Synthetic Continued Pretraining with @ZitongYang0*, @ShuangpingLi, @EmmanuelCandes, and @tatsu_hashimoto!
A simple approach to continue pretraining (CPT) on as little as ~1M tokens:
Small specialized corpus -> synthesize a large, diverse corpus -> CPT
Grab your favorite preprint of the week: how can you put its knowledge in your LM’s parameters? Continued pretraining (CPT) works well with >10B tokens, but the preprint is <10K.
Synthetic CPT downscales CPT to such small, targeted domains.
📜: arxiv.org/abs/2409.07431
🧵👇
Come check out our #ICML2024 paper on
Linguistic Calibration of Long-Form Generations
at the poster session tomorrow!
time: Thursday, 11:30am-1pm
location: #712, Hall C 4-9
paper: arxiv.org/abs/2404.00474
code: github.com/tatsu-lab/linguis…
When LLMs are unsure, they either hallucinate or abstain.
Ideally, they should clearly express truthful confidence levels.
Our #ICML2024 work designs an alignment objective to achieve this notion of linguistic calibration in *long-form generations*.
arxiv.org/abs/2404.00474
🧵
Public release of the #̶N̶e̶u̶r̶I̶P̶S̶2̶0̶2̶2̶ #NeurIPS2021 (😅) RETINA Benchmark:
A suite of tasks evaluating the reliability of uncertainty quantification methods like Deep Ensembles, MC Dropout, VI, and more.
📜 arxiv.org/abs/2211.12717
💻 rebrand.ly/retina-benchmark
🧵below [0/N]
Today is the 🎉public release🎉 of our 1.5 year collaboration b/t @OATML_Oxford and @GoogleAI!
Was a joy to work w/ @timrudner on the Plex medical diagnosis experiments, using the RETINA Benchmark: openreview.net/forum?id=jyd4… 🩺
s/o to the awesome leads @dustinvtran & @balajiln 🥳
In our work "Plex", we propose a framework for reliability in AI. We also introduce new models (ViT-Plex & T5-Plex) for reliable decision-making across a broad array of scenarios.
Blog: ai.googleblog.com/2022/07/to…
Paper: goo.gle/plex-paper
Code: goo.gle/plex-code
Want to test your Bayesian deep learning methods on real-world distribution shifts?
Come check out our @NeurIPSConf DistShift Workshop poster session now until 3pm PST!
💡Spotlight Talk: neurips.cc/virtual/2021/work…
🏘 GatherTown [Room 1, Spot G0]: eventhosts.gather.town/gEiGK…
*Are uncertainty estimates in Bayesian deep learning reliable under distribution shifts?*
To find out, check out our #NeurIPS2021 paper
🤖🩺Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks🤖🩺
🚀Paper: openreview.net/forum?id=4bza…
🤖🩺Diabetic Retinopathy Uncertainty Benchmark🩺🤖
Curious how Bayesian deep learning methods perform on real-world distribution shifts?
Come checkout our our poster session today at @NeurIPSConf Datasets and Benchmarks!
📆 Dec 10, 4.30 – 6.00 pm GMT
🔗 neurips.cc/virtual/2021/data…
*Are uncertainty estimates in Bayesian deep learning reliable under distribution shifts?*
To find out, check out our #NeurIPS2021 paper
🤖🩺Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks🤖🩺
🚀Paper: openreview.net/forum?id=4bza…
We'll be presenting our poster on Non-Parametric Transformers starting in 20 min!
📜poster (Dec 7, 4:30 - 6pm GMT): neurips.cc/virtual/2021/post…
📝paper: arxiv.org/abs/2106.02584
🤖code: github.com/OATML/non-paramet…
@janundnik @clarelyle @AidanNGomez @tom_rainforth @yaringal @OATML_Oxford
🗞New Paper🗞
🤖🧪Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning 🧪🤖
Huge thanks to @neilbband* as well as @clarelyle, @AidanNGomez, @tom_rainforth, @yaringal, and @OATML_Oxford !
Introducing 🚀Non-Parametric Transformers🚀 1/
ALT Overview of the Non-Parametric Transformer. (a) The input dataset and mask matrix are stacked and (b) linearly embedded for all datapoints independently. NPT then applies (c) Attention Between Datapoints across all n samples of hidden dimension h = d · e. (d) Attention Between Attributes then attends between the attributes for each datapoint independently. We repeat steps (c) and (d) and obtain a final prediction from a separate linear projection.
Details on the Shifts Challenge in Robustness and Uncertainty!
🤝Joint project w/ @YandexAI, @OATML_Oxford, Cambridge Speech Group
3 datasets w/ real-world distributional shift: ☔️ weather prediction, 🤖 machine translation, 🚗 vehicle motion prediction
oatml.cs.ox.ac.uk/blog/2021/…
website: research.yandex.com/shifts
whitepaper: arxiv.org/abs/2107.07455
github: github.com/yandex-research/s…
1
Excited for the release of a joint project with @Yandex, @OATML_Oxford, and the Cambridge Speech Group:
🎉🎊 Shifts Challenge 🎊🎉
We introduce 3 datasets w/ real distributional shifts: weather prediction, machine translation, and vehicle motion prediction (largest to date!).
1/7 We are happy to launch the Shifts Challenge on uncertainty and robustness to distributional shift @NeurIPSConf! The challenge is jointly organized by Yandex.Research together with the OATML group (@OATML_Oxford) the Cambridge University Speech Group.
research.yandex.com/shifts
ALT https://tinyurl.com/2h27vmv2
🚨 New paper alert 🚨
Excited to share more on "Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning" 🎉
Joint work with @janundnik*, along with @clarelyle, @AidanNGomez, @tom_rainforth, and @yaringal from @OATML_Oxford 🥣 (1/n)
Had a great time working on Non-Parametric Transformers with @janundnik* along with @clarelyle, @AidanNGomez, @tom_rainforth, @yaringal.
We're very excited for future work in this direction!
🗞New Paper🗞
🤖🧪Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning 🧪🤖
Huge thanks to @neilbband* as well as @clarelyle, @AidanNGomez, @tom_rainforth, @yaringal, and @OATML_Oxford !
Introducing 🚀Non-Parametric Transformers🚀 1/
ALT Overview of the Non-Parametric Transformer. (a) The input dataset and mask matrix are stacked and (b) linearly embedded for all datapoints independently. NPT then applies (c) Attention Between Datapoints across all n samples of hidden dimension h = d · e. (d) Attention Between Attributes then attends between the attributes for each datapoint independently. We repeat steps (c) and (d) and obtain a final prediction from a separate linear projection.