PhD student at UBC. Working on theory of Deep Learning.
Joined August 2019
- Tweets873
- Following436
- Followers170
- Likes5.4K
AI not smart enough for DL theory
Some people on X realised that OpenAI's math drop didn't contain much cryptography. @dpaleka notes that a few more fields are under-represented: largely the ones that are useful for AI research
Puneesh Deora retweeted
I tracked the best LR per step when finetuning a small Qwen model on GSM8K and got the following shape. It seems that warmup is good practice in general. (This is a greedy per-step optimal lr, not overall optimal lr schedule which would cost orders of magnitude more in compute)
Is this naming based on the talk Zico Kolter gave at a Neurips workshop 2024 and the subsequent discussion/debate
Deep learning is often taught as a sanitized collection of facts and math. But work at the frontier of AI is messy and empirical.
Our new class, Deep Learning Alchemy, aims to teach the art and science of deep learning in the hands-on spirit of Language Modeling From Scratch.
Students build intuition by running experiments on small LMs, then test it through prediction quizzes: given an unseen code diff, can you predict what happens?
deep-learning-alchemy.github…
My co-instructor @kothasuhas and I think there's significant demand for classes that treat students as serious learners who want to be assessed for what they’ve learned, and so we're trying a new exam-heavy approach to the class (7 exams, 85% of the grade, with the remaining 15% in-person discussion assessments).
We’ve designed the class so students can use AI agents to run their experiments and do their assignment, but they then have to demonstrate that mastery on exams without access to the assignment, and any partial credit on the exam has to come through citations to the assignment, incentivizing them to know what’s in their write-up.
The same assignments and quizzes are public for anyone to try (lecture recordings will be released through @StanfordOnline with some delay). We have some very fun assignments and quizzes covering both classic topics (like NQM, run-to-run stochasticity, and scaling laws) and new and timely topics like muP or hyperball. Made by some real experts like @TristanThrush and @wen_kaiyue
Try some quizzes below!
Is this the "singularity" everyone keeps talking about
BREAKING: Japan declares a "state of emergency in cyberspace" as cyberattacks surge, with more than 20 major companies reporting breaches.
Readers added context they thought people might want to know
Japan did not formally declare a state of emergency in cyberspace. The phrase originated from a ruling party official at an LDP meeting; government agencies issued cybersecurity warnings and basic security advisories to companies amid breaches at 20+ firms.
japancyberwatch.com/articles/japan…
japancyberwatch.com/articles/japan…
cyber.go.jp/pdf/press/taio…
thenextweb.com/news/japan-cyb…
Puneesh Deora retweeted
10 remarks on AI and math that can be true all at once:
1. The recent progress is stunning and exciting.
2. The Quasi-Riemann Hypothesis is arguably Fields-medal level.
3. Mathematics is being transformed and there is no way back.
4. OpenAI doesn't give a fuck about the future of mathematics.
5. Half-baked unreadable preprints claiming major results can effectively stall mathematical progress.
6. Some mathematicians are responding with naive, overly defensive comments.
7. The social-media wankers framing this as "OpenAI tramples gatekeeping" have no experience of mathematical research.
8. The product of mathematics is clarity and understanding. Not theorems, by themselves.
9. Mathematics is a critical infrastructure of human cognition. The human need for understanding isn't obsolete and it can't be offloaded to machines.
10. We haven't "solved" mathematics and we won't get there before the Sun burns out.
Senior Math people not helping themselves with terrible examples like this one
Search is so easy now with AI, you can write what you want in great detail and get precisely what you're looking for. Back in the day you had be to careful on how to Google search to get the right links to show up
Puneesh Deora retweeted
Well worth reading: proofsandprompts.com/2026/10…
Puneesh Deora retweeted
Wait a minute so they literally threw a slop grenade over the fence and are only checking over half of it afterwards?? Whyyyy
We've updated our GitHub math repo with 6 new Lean formalizations, 19 modifications, and 3 withdrawals. The repo now has ~42% top-line results formalized. We will continue to update the repo with new formalizations and with any errata we notice.
github.com/openai/math/blob/…
This has more views than adidas's post on Messi retiring from international football
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
OpenAI's recent release is great if people try to make the results digestible, give talks, discuss, and build on them. But if it just deters students from pursuing maths, it seems pointless. Results and Lean formalizations just sitting on GitHub aren't of much use to anyone.
Puneesh Deora retweeted
OpenAI’s intellectual cannon is so good at Math because Math can be expressed very precisely as a language, and training on Math can be made very effective through Lean formalization and proof checking.
These factors are not there in the same way for medical research. It is unclear why being extremely good at Math should make these models equally good at curing cancer. This kind of consideration is probably bad for IPO.
Proving the Riemann Hypothesis by Halloween 2026 would be fantastic PR and, at the same time, would tell us very little about curing cancer (or sending people to space, which seems to be another favorite of AI-enthusiasts these days).
In the meantime, me and others are trying to make clear that modern Math is a very complex (and imperfect!) system, with tens of thousands of people involved in different roles, and years of training needed before someone is in a position to do creative work.
This setup was not built because someone planned it. It grew over decades through changing traditions of mentoring and research. In 20 years of Math I have definitely seen huge changes in how research works, well before LLMs. Things can adapt, but with time to adapt.
If you stop forming new people, the whole thing collapses. And once it has collapsed, you are not going to rebuild it at whim. You are just going to have very few people around who even know how human Math research used to work.
We have no model for machines training humans to replace the human system that trained the machines.
To me, it all seems like a very rushed and very risky bet.
I wonder how many pure Math people will pivot to deep learning theory now. It has happened in the last few years, might see more cases in the next year or so
Puneesh Deora retweeted
It might be true saying that AI Math capabilities lie in the convex hull of what humans have already developed, but to even search well within that space is a great capability, and that's what *most* human research was doing anyways
