Nolan Dey (@DeyNolan) profile picture

Research Scientist @ Cerebras Systems

Toronto
Joined March 2022
Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)Photo by Nolan Dey (@DeyNolan)
We just made LLM inference up to 30% faster ⚡ Introducing DNeo, a new diffusion drafter designed in collaboration with @cerebras researchers. The secret? Your draft model needs a working memory, and the "right" context encoding mechanism. 📄 Paper: alphaxiv.org/pdf/2611.21925
9
40
21
216
24,881
Nolan Dey retweeted
“Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference” Layer dropout mostly disappeared from modern LLM pretraining because it was thought to hurt accuracy at scale. But this Cerebras paper shows the opposite when configured correctly. They apply layer dropout at the whole Transformer-block level, sampled per sequence, use an increasing dropout distribution across depth, and a decreasing dropout schedule over training that decays to zero. Across 2,400 runs up to 8.2B parameters, this saves up to 25% of training FLOPs while preserving accuracy, and makes models depth-elastic enough for early exit, layer skipping, and up to 1.55x faster self-speculative decoding. alphaxiv.org/abs/2609.05275
9
45
5
318
43,001
NeurIPS 2026 workshop alert!!! BeNTo: Beyond Next-Token Prediction Diffusion & Flow Models for Next-Generation Decoding lnkd.in/gf_yWqYd Submission deadline: Aug 29, 2026 If you are working in this area, consider submitting! Sponsored by @cerebras and @OutputBio
1
1
405
🎤 Speakers • Stefano Ermon — Stanford University • Masatoshi Uehara — OpenAI • Jiaxin Shi — Meta Superintelligence Labs • Guan-Horng Liu Superintelligence Labs • Aditya Grover — UCLA / Inception • Arash Vahdat — NVIDIA Research
1
73
Nolan Dey retweeted
"You could imagine yourself actually being 10x more productive." @DeyNolan from the Cerebras Core ML team speaks with @alyciazcary on why speed contributes to intelligence. In agentic workloads like Claude Code, queries can run for hours. Faster inference means more test time compute in the same budget.
9
11
1
109
133,043
Another new preprint from @cerebras 🚨📄- this time on training *re-evaluation* curves (TRECs) for data curriculums in LLMs. Everyone sticks high-quality data at the end of training… we show the sweet spot is often earlier — and we can predict it. arxiv.org/abs/2509.25380
1
1
12
965
(1/4) @cerebras Hot off the presses 🔥📄arxiv.org/abs/2509.25087 If you're spending $1B to train an LLM, you need to know it’s on track—every step of the way. With optimal AdamW τ + fixed TPP, loss curves collapse to a universal path → an early-warning signal for training.
2
8
3
25
6,922
Power Lines paper now out: arxiv.org/abs/2505.13738 TL;DR - we identify how AdamW's weight decay should scale with batch size, dataset size, and model size in LLM pre-training. We also investigate the scaling of both "optimal" and "critical" batch size.
1
18
2
94
9,092
(1/7) @cerebras Paper drop: arxiv.org/abs/2505.01618 TLDR: We introduce CompleteP, which offers depth-wise hyperparameter (HP) transfer (Left), FLOP savings when training deep models (Middle), and a larger range of compute-efficient width/depth ratios (Right).  🧵 👇
12
64
11
400
113,759
(6/7) Implementing CompleteP is very simple, only requiring two lines of code. We provide a minimal implementation here: github.com/EleutherAI/nanoGP…
4
2
32
4,471
(7/7) If you are looking to conduct research into deep models, contact us to collaborate! We are also hiring research scientists (job-boards.greenhouse.io/cer…) and research engineers (job-boards.greenhouse.io/cer…)!
4
28
2,514
Published "Neuron-based explanations of neural networks sacrifice completeness and interpretability" in TMLR 2025! TL;DR: The most important principal components provide more complete and interpretable explanations than the most important neurons. ndey96.github.io/neuron-expl…
3
322
🎉We're excited to announce our joint work with @Cerebras on a new guide to Maximal Update Parameterization (μP) and μTransfer!🎉 This practitioner's guide (and implementation) aims to make μP more accessible and easier to implement for the broader training community. 🧵
2
28
4
178
24,485
So, uh, it turns out that 30+ years of neural net sparsity research have been confounded by optimal hyperparameters varying with sparsity level...
Replying to @cerebras
(6/n) Applying SμPar to pretraining a 610M parameter LLM significantly improves loss over SP and μP models due to improved HP tuning.
4
8
2
111
74,348
Nolan Dey retweeted
(1/n) Paper drop: arxiv.org/abs/2405.15743 TLDR: We introduce the sparse maximal update parameterization (SμPar), which ensures optimal HPs remain the same for any width or sparsity level. This dramatically reduces HP tuning costs, allowing SμPar to achieve superior losses. 🧵 👇
4
36
7
175
29,014
Successfully ported @karpathy's nanoGPT to the new @Apple MLX framework, possibly enabling quick prototyping of training GPT style models on Mac GPUs. Check out the project: github.com/vithursant/nanoGP…. Got a new M3 Pro and wanted to learn about MLX over the holidays lol.
3
14
3
91
45,320