ML Engineer | Research on hallucinations, scaling laws, HLE https://nitter.cf/t.co/4DEK3A8afk
San Francisco | NYC
Joined April 2023
- Tweets16
- Following67
- Followers143
- Likes38
Pinned Tweet
I think Stanford's CS336 is one of the best materials to learn LLMs from scratch. I just finished working through it (tokenizer, transformer, triton kernels, distributed training, scaling laws, alignment) and highly recommend it. 🧵 Some notes:
At smaller sequence lengths like 1024, the FFN dominates, not attention, which is a big part of why MoE goes on the FFN. At 16k ctx, attention usually takes over.
Naive DDP on 2 gpus: 54% of each step was gradient sync. overlapping comms with backward: 3%. ZeRO-1 sharding cut peak memory 68.7 -> 47.7 GB.
GRPO: base model with the r1-zero prompt got 0/1300 correct for gsm8k. it gets 224/1300 correct with the 3-shot prompt. RL is most effective when it amplifies signal that's already there.
🔗 check it out: github.com/michaelc-yu/cs336…
For people working on LLM inference: is multi-head latent attention from DeepSeek becoming the industry standard for kv cache compression? Or do simpler approaches still work better?
Michael Yu retweeted
Excited to share our #NeurIPS2026 paper on benchmarking hallucinations in LLMs! We take a fresh world model-centric approach to measuring hallucinations in several environments including terminal, grid worlds, and chess. Thanks to the @DegenAI_Labs team for their hard work!
Also grateful to collaborators at @Stanford, @StanfordAILab, @stanfordnlp, @CarnegieMellon, @LTIatCMU, @OhioState, and @OhioStateCSE.
Hallucinations are hard to measure because reality is messy. So we built our own world.
Introducing HalluWorld 🌎, a controlled benchmark for LLM hallucinations.
Now accepted to #NeurIPS2026 Evaluations & Datasets! 🧵
halluworld.ai
ALT The halluworld benchmark website, depicting three domains of gridworlds, chess, and terminals
Michael Yu retweeted
Thrilled to announce HalluWorld 😵💫 🌎 , a controlled benchmark + methodology to verifiably measure how LLM agents hallucinates while in an environment.
Now accepted at #NeurIPS2026 Evals & Datasets!!
We start with the thought that Reality is vast, dynamic, fractally unfolding.
W/o a stable reference and controllable complexity, verifiably measuring and inducing situations that lead to Hallucination a.k.a deviation from observed Reality gets tough.
This is even more so if we want to go from single turn Q/A setting --> Agents hallucinating while solving a task once placed in an environment.
HalluWorld puts models inside environments where we know exactly what is true at every step: Gridworlds 🕹️ , Chess ♟️, and Terminal 👩💻 tasks.
What's more?
Having such environments makes things neatly modifiable to am up support different levers of complexity, think canyons, signboards and long tunnels (Gridworlds) ; arbitrarily complex game situations or mods (Chess) inter alia --> This gives us ways to craft situations that amp up cognitive pressure on the agent
This is inspired and in vein of the bAbi methodology of creating questions with knowable answers from a world you can create by @jaseweston et al from 2015.
Come check out more at halluworld.ai , or read our arXiv at arxiv.org/abs/2605.19341 !
It was enriching and fun working on this with @_emliu , @karanps, @stevenyfeng, @_michael_yu_, @ZhuofuTao and @shocheen from @LTIatCMU , @PatronusAI , @stanfordnlp and @OhioStateCSE over the past few months!
Hallucinations are hard to measure because reality is messy. So we built our own world.
Introducing HalluWorld 🌎, a controlled benchmark for LLM hallucinations.
Now accepted to #NeurIPS2026 Evaluations & Datasets! 🧵
halluworld.ai
ALT The halluworld benchmark website, depicting three domains of gridworlds, chess, and terminals
Michael Yu retweeted
Hallucinations are hard to measure because reality is messy. So we built our own world.
Introducing HalluWorld 🌎, a controlled benchmark for LLM hallucinations.
Now accepted to #NeurIPS2026 Evaluations & Datasets! 🧵
halluworld.ai
ALT The halluworld benchmark website, depicting three domains of gridworlds, chess, and terminals
Michael Yu retweeted
We’re bringing back Stanford’s CS25 Transformers course tomorrow! 🤖
It’s open to everyone (in-person + online). Weekly talks (every Thursday) from top AI researchers.
One of Stanford’s most popular AI seminar courses. Don’t miss out!
More info below 👇 (1/7)
If the world were to end in a year, how would we best preserve all our knowledge for aliens to discover in the future?
Would it be through compressing everything we knew into parameters of a large neural network and sending that into space?
Michael Yu retweeted
Proud to share what I've been working on for the past year, "Learning to (Learn at Test Time)"!
Our new architecture trains a model to "learn" from its context, replacing Attention's costly KV cache with an expressive hidden state: the weights of a ML model!🤯
🧵by @karansdalal
I’m excited to share a project I’ve been working on for over a year, which I believe will fundamentally change our approach to language models.
We’ve designed a new architecture, which replaces the hidden state of an RNN with a machine learning model. This model compresses context through actual gradient descent on input tokens. We call our method “Test-Time-Training layers.”
TTT layers directly replace attention, and unlock linear complexity architectures with expressive memory, allowing us to train LLMs with millions (someday billions) of tokens in context.
Our instantiations, TTT-Linear and TTT-MLP, both match or beat the strongest Transformers and Mamba. Arxiv: arxiv.org/abs/2407.04620
Me and my friends @stevenyfeng and @HaoranZhaoHRZ met @karpathy a few weekends ago when he was giving the keynote and winner pitches at UC Berkeley ai hackathon. It was great to have a face to face chat with a top AI researcher and educator. Super nice and humble guy.
