Research Scientist in artificial intelligence at DeepMind
Montréal, Québec
Joined May 2016
- Tweets1.2K
- Following219
- Followers9K
- Likes1.4K
Pinned Tweet
Transformers can be made sparse across their depth. When trained isoFLOP, we can match or exceed the performance of vanilla models, while saving inference FLOPs
arxiv.org/abs/2404.02258
Adam Santoro retweeted
reading the "mixture of depths" paper, which comes up with a novel way to conditionally apply compute depth-wise in a decoder
basically they use standard MoE-style expert-choice routing but they use it to choose which tokens get to go through every block in the decoder
Adam Santoro retweeted
Gemini and I also got a chance to watch the @OpenAI live announcement of gpt4o, using Project Astra! Congrats to the OpenAI team, super impressive work!
It's such an honor to work on Project Astra with such an amazing team from across Gemini and Google DeepMind! While the #GoogleIO keynote was happening we had a last minute idea of watching the keynote with Project Astra. Check it out!
Adam Santoro retweeted
We watched #GoogleIO with Project Astra. 👀
Adam Santoro retweeted
Mixture of depth works
for 300M Seq 512 its faster and archives better loss
Code: github.com/joey00072/ohara/t…
writeup: huggingface.co/blog/joey0007…
Adam Santoro retweeted
I have implemented Mixture-of-Depths and it shows significant memory reduction during training and 10% speed increase. I will verify if it achieves the same quality with 12.5% active tokens.
github.com/thepowerfuldeez/O…
thanks @haeggee for initial code
Adam Santoro retweeted
... and with one experiment, I was able to roughly reproduce their results for a ~220M GPT-2. It gives a speedup of ~20min (80min dense vs 60min MoD, 4 A100s) while keeping the pplx close. This roughly matches Fig. 3 or 4 in the paper arxiv.org/pdf/2404.02258.pdf
Adam Santoro retweeted
Standard transformer-based language models use the same amount of compute for each token. Our new method, which we call Mixture-of-Depths, allows transformers to instead learn to dynamically allocate compute to specific positions in a sequence.
arxiv.org/abs/2404.02258
Adam Santoro retweeted
Big fan of the work of @santoroAI and others. Glad Google decided to finally release it!
arxiv.org/abs/2404.02258
Rather than process a full sequence length (T) worth of tokens per block, we use learned routing to process k<T -tokens instead.
These accrued FLOP-savings can be used to, e.g., train for longer or to introduce new parameters.
Adam Santoro retweeted
I'm not a scaling maximalist, but it's surprising to me how many people are 1) interested in differences between human and artificial intelligence and 2) think scaling to improve performance means deep learning is doing something fundamentally wrong. 1/n
Adam Santoro retweeted
I can finally talk about Sparrow! My team used RL to train a 70 billion parameter model to be simultaneously safer and more helpful… (1/5)
Adam Santoro retweeted
Apply to be a Research Scientist Intern at DeepMind Montreal for Summer 2023
grnh.se/87dac2191us
Interesting focus on architecture in many responses. My list is less about what's new, and more about what proved to be important:
1) Interactive experience
2) Embracing emergence
3) Engaging with real world complexity
4) The consequences of being socially embedded
Wonderful paper. Required reading for those who fling around the word "innate" to motivate research programs in AI.
This Post is from an account that no longer exists. Learn more
Adam Santoro retweeted
Abstract reasoning is ideally independent of content. Language models do not achieve this standard, but neither do humans. In a new paper (arxiv.org/abs/2207.07051 co-led by Ishita Dasgupta) we show that LMs in fact mirror classic human patterns of content effects on reasoning. 1/
There's no better sign that ideas have stagnated than when any new piece of evidence is either strongly in your favor, or else must be flawed in some way
Coincidentally, this is how conspiracy theories work
Have we ever considered the doomsday AI scenario where an alien superintelligent AI is so smart that it figures out how to wormhole its way to our planet and demolishes us all to maximize its paperclip count? Should have happened by now, no?