Machine Learning Researcher
Kissimmee, FL
Joined January 2008
- Tweets67
- Following1.5K
- Followers1.2K
- Likes90
Alex Alemi retweeted
A statement on the comments from Secretary of War Pete Hegseth.
anthropic.com/news/statement…
Alex Alemi retweeted
I am recruiting Ph.D. students for my new lab at @nyuniversity! Please apply, if you want to work with me on reasoning, reinforcement learning, understanding generalization and AI for science.
Details on my website: izmailovpavel.github.io. Please spread the word!
Alex Alemi retweeted
Is Kevin onto something? We found that LLMs can struggle to understand compressed text, unless you do some specific tricks. Check out arxiv.org/abs/2404.03626 and help @hoonkp, @alemi, Jeffrey Pennington, @ada_rob, @jaschasd, @noahconst and I make Kevin’s dream a reality.
Alex Alemi retweeted
Ever wonder why we don’t train LLMs over highly compressed text? Turns out it’s hard to make it work. Check out our paper for some progress that we’re hoping others can build on. arxiv.org/abs/2404.03626 With @blester125, @hoonkp, @alemi, Jeffrey Pennington, @ada_rob, @jaschasd
Alex Alemi retweeted
Happy to announce DreamFusion, our new method for Text-to-3D!
dreamfusion3d.github.io
We optimize a NeRF from scratch using a pretrained text-to-image diffusion model. No 3D data needed!
Joint work w/ the incredible team of @BenMildenhall @ajayj_ @jon_barron
#dreamfusion
Alex Alemi retweeted
To accompany the colab, I've also written a blog post blog.alexalemi.com/diffusion… attempting to make sense of the VDM Diffusion loss.
In it, I try to motivate how the VDM diffusion loss is simply the joint KL between the forward and reverse process.
Alex Alemi retweeted
Want to understand and/or play with variational diffusion models?
- See colab.research.google.com/gi… for a simple stand-alone implementation and explanation. (Thanks @alemi and @poolio for making this)!
- See colab.research.google.com/gi… for an even more basic implementation on 2D data.
Alex Alemi retweeted
A pretty cool paper (and I also hope useful) on using pre-training models to create highly informative priors for downstream tasks. Thanks to all the collaborators, it was a lot of fun!
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors.
arxiv.org/abs/2205.10279
w/@ziv_ravid, @micahgoldblum, @HosseinSouri8, @snymkpr, @Eiri1114, @ylecun 1/6
Alex Alemi retweeted
1/ Super excited to introduce #Minerva 🦉(goo.gle/3yGpTN7). Minerva was trained on math and science found on the web and can solve many multi-step quantitative reasoning problems.
Very excited to present Minerva🦉: a language model capable of solving mathematical questions using step-by-step natural language reasoning.
Combining scale, data and others dramatically improves performance on the STEM benchmarks MATH and MMLU-STEM. goo.gle/3yGpTN7
Alex Alemi retweeted
We are thrilled to announce Imagen, a text-to-image model with unprecedented photorealism and deep language understanding. Explore imagen.research.google and Imagen!
A large rusted ship stuck in a frozen lake. Snowy mountains and beautiful sunset in the background. #imagen
Alex Alemi retweeted
We are presenting our paper "Does Knowledge Distillation Really Work?" at #NeurIPS2021 poster session 2 today - come check it out! Joint work with
@Pavel_Izmailov, @polkirichenko, @alemi, and
@andrewgwils.
Poster: nips.cc/virtual/2021/poster/…
Paper: arxiv.org/abs/2106.05945
Alex Alemi retweeted
Excited to kick-start focus #SciML series on #ML meets Info theory and statistical mechanics! Amazing speaker/session chair line-up: @alemi (@wellingmax), @pratikac (Karthik), @ShoYaida (@jaschasd), @yasamanbb (@SuryaGanguli) and Elena Agliari. Details at: cmu.edu/aced/sciML.html
Alex Alemi retweeted
While most papers on knowledge distillation focus on student accuracy, we investigate the agreement between teacher and student networks. Turns out, it is very challenging to match the teacher (even on train data!), despite the student having enough capacity and lots of data.
Does knowledge distillation really work?
While distillation can improve student generalization, we show it is extremely difficult to achieve good agreement between student and teacher.
arxiv.org/abs/2106.05945
With @samscub, @Pavel_Izmailov, @polkirichenko, Alex Alemi. 1/10
Alex Alemi retweeted
Turns out, it is really hard to get the student to match the teacher predictions in knowledge distillation, even if we train really long and use lots of data augmentation.
Why? Optimization is hard!
New paper with @samscub @polkirichenko @alemi and @andrewgwils!
Does knowledge distillation really work?
While distillation can improve student generalization, we show it is extremely difficult to achieve good agreement between student and teacher.
arxiv.org/abs/2106.05945
With @samscub, @Pavel_Izmailov, @polkirichenko, Alex Alemi. 1/10
Alex Alemi retweeted
tired: knowledge distillation improves accuracy because soft teacher labels transfer knowledge
wired: distillation improves accuracy because it turns an easy optimization problem into an incredibly difficult one.
turns out soft labels are _really_ hard to overfit
(1/n)
Does knowledge distillation really work?
While distillation can improve student generalization, we show it is extremely difficult to achieve good agreement between student and teacher.
arxiv.org/abs/2106.05945
With @samscub, @Pavel_Izmailov, @polkirichenko, Alex Alemi. 1/10
Alex Alemi retweeted
We're presenting our work on #EnergyInspiredModels (EIM) which leverage a learned energy function. Unlike #EnergyBasedModels, EIMs are tractable to sample from and train via a lower bound on log-likelihood. #NeurIPS2019 10:45am Wed #120 (arxiv.org/abs/1910.14265)
Alex Alemi retweeted
Classifiers are secretly energy-based models! Every softmax giving p(c|x) has an unused degree of freedom, which we use to compute the input density p(x). This makes classifiers into generative models without changing the architecture.
arxiv.org/abs/1912.03263
Alex Alemi retweeted
Hey Twitter! If you're at #NeurIPS2019 come by Poster 67, first session. We're presenting an EXACT rate-distortion noise model, which gives you a closed form for I(z,x) in e.g. VAE.
Work lead by the amazing @brekelmania @gesteller @aram_galstyan team at @USC_ISI
Alex Alemi retweeted
Want to estimate or optimize mutual information using neural networks and the latest variational bounds? Check out our Colab notebook for implementations and experiments!
Colab: colab.research.google.com/gi…
Paper: arxiv.org/abs/1905.06922
Alex Alemi retweeted
Come learn about variational bounds of mutual information tomorrow (Thursday) at #ICML2019, 4:40pm in the Grand Ballroom or drop by poster #86 at 6:30pm! Joint work w/awesome collaborators @sherjilozair @avdnoord @alemi @georgejtucker
arxiv.org/abs/1905.06922
