Vikas Chandra retweeted
2/ the work by @vikasc and others at Meta AI shows that Trajectory Reduction Policy Optimization (dTPRO) significantly reduces the training cost of diffusion LLMs (dLLMs), a key step forward as scaled training has been a hurdle for dLLM to date. watch out for what comes from Meta here
arxiv.org/abs/2603.18806
Vikas Chandra retweeted
the market loves to talk hardware because stock prices go brrr but
the most interesting thing happening right now is in alternative model architectures
Diffusion models couldn't reason because RL was too expensive, not because the architecture was wrong. dTRPO collapses trajectory computation to one forward pass. On a 7B model: +9.6% GPQA, +4.3% HumanEval+. The architecture question is open again.
Paper: arxiv.org/abs/2603.18806
Vikas Chandra retweeted
Been thinking about what this paper really means.
"Video diffusion" and "World Models" are becoming synonymous. Neural Computers are basically video diffusion world models for terminal envs and GUI.
Lots of talk last week about automating Manim videos. In theory, we should be able to train these world models on a 10000 hours of diverse manim videos and "see where it goes"
If a NC can generate outputs to terminal commands, it should be able to generate videos like this directly from prompt too.
Without writing code.
Vikas Chandra retweeted
the market is reacting to the memory shortage by buying Sandisk assuming that labs continue to run less optimized versions of their own models and overspend on hardware but the longer term signal to take from the memory wall is to focus on the continued scaling down of models
deployed compute has scaled 3x every 2 years while memory bandwidth has only scaled 1.6x over 20 years - every new GPU generation widens this gap - but we've also been grossly overusing memory via the common training mode. APOLLO showed that AdamW, the standard LLM optimizer, stores redundant state for every single parameter, and that coarser gradient approximations achieve the same training quality with a fraction of the memory, enabling model pre-training on a 1/8 the GPU capacity. many more proof points are coming to light re: optimizing algorithms and efficiency of smaller models. in fact small models can outperform much larger models by spending more compute at inference time
good read on this from @vikasc
🤖 Made with AI
Vikas Chandra retweeted
🫱 Introducing 𝐍𝐞𝐮𝐫𝐚𝐥 𝐂𝐨𝐦𝐩𝐮𝐭𝐞𝐫s:
𝐰𝐡𝐚𝐭 𝐢𝐟 𝐀𝐈 𝐝𝐨𝐞𝐬 𝐧𝐨𝐭 𝐣𝐮𝐬𝐭 𝐮𝐬𝐞 𝐜𝐨𝐦𝐩𝐮𝐭𝐞𝐫𝐬 𝐛𝐞𝐭𝐭𝐞𝐫, 𝐛𝐮𝐭 𝐛𝐞𝐠𝐢𝐧𝐬 𝐭𝐨 𝐛𝐞𝐜𝐨𝐦𝐞 𝐭𝐡𝐞 𝐫𝐮𝐧𝐧𝐢𝐧𝐠 𝐜𝐨𝐦𝐩𝐮𝐭𝐞𝐫 𝐢𝐭𝐬𝐞𝐥𝐟?
Beyond today's conventional computers, agents, and world models, Neural Computers (NCs) are new frontiers where computation, memory, and I/O move into a learned runtime state.
We ask: whether parts of runtime can move inward into the learning system itself. This is our first step toward the Completely Neural Computer (CNC): a general-purpose neural computer with stable execution, explicit reprogramming, and durable capability reuse.
Work done with Mingchen Zhuge (@MingchenZhuge), Changsheng Zhao, Haozhe Liu (@HaoZhe65347 ), Zijian Zhou (@ZijianZhou524 ), Shuming Liu (@shuming96 ), Wenyi Wang (@Wenyi_AI_Wang ), Ernie Chang (@erniecyc ), Gael Le Lan, Junjie Fei, Wenxuan Zhang, Zhipeng Cai (@cai_zhipeng ), Zechun Liu (@zechunliu ), Yunyang Xiong (@YoungXiong1 ), Yining Yang, Yuandong Tian (@tydsh ), Yangyang Shi, Vikas Chandra (@vikasc), Juergen Schmidhuber (@SchmidhuberAI)
Vikas Chandra retweeted
Using On-policy distillation leads to great performance boost even for small models😃Great work from @zechunliu @erniecyc and Changsheng (@vikasc's group)!
This Post is from a suspended account. Learn more
Vikas Chandra retweeted
[1/n] 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗧𝗿𝗮𝗰𝗸 𝗔𝗻𝘆𝘁𝗵𝗶𝗻𝗴 from @Meta: interactive video segmentation and tracking on an iPhone!
Vikas Chandra retweeted
🚀Excited to share our Efficient Track Anything.
It is small but mighty, >2x faster than SAM2 on A100 and runs > 10 FPS on iPhone 15 Pro Max. How’d we do it? EfficientSAM + Efficient Memory Attention!
Paper: arxiv.org/pdf/2411.18933
Project (demo): yformer.github.io/efficient-…
with: @ChongZhou7, @mukosame, @klightlm, @zechunliu, @_sakshams_ , @balakrishnan_vr,
@fiandola, @bilgeesra, @raghuraman, @vikasc, etc
Vikas Chandra retweeted
Our SpinQuant work (arxiv.org/abs/2405.16406) has been used in the quantized versions of LLaMA 3.2 1B/3B model released in Meta Connect'24.
Congrats to all co-authors! @zechunliu, Changshen Zhao, Raghuraman Krishnamoorthi, Dhruv Choudhary, Bilge Soran, Igor Fedorov, @vikasc, @TiRune
We want to make it easier for more people to build with Llama — so today we’re releasing new quantized versions of Llama 3.2 1B & 3B that deliver up to 2-4x increases in inference speed and, on average, 56% reduction in model size, and 41% reduction in memory footprint.
Details on our new quantized Llama 3.2 on-device models ➡️
ai.meta.com/blog/meta-llama-…
While quantized models have existed in the community before, these approaches often came at a tradeoff between performance and accuracy. To solve this, we Quantization-Aware Training with LoRA adaptors as opposed to only post-processing. As a result, our new models offer a reduced memory footprint, faster on-device inference, accuracy and portability — while maintaining quality and safety for developers to deploy on resource-constrained devices.
The new models can be downloaded now from Meta and on @huggingface.
Vikas Chandra retweeted
🚨VideoLLM from Meta!🚨
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
📝Paper: huggingface.co/papers/2410.1…
🧑🏻💻Code: github.com/Vision-CAIR/LongV…
🚀Project (Demo): vision-cair.github.io/LongVU
We propose LongVU, a video LLM with a spatiotemporal adaptive compression mechanism designed for real-world hour-long video understanding. LongVU adaptively reduces the number of video tokens by leveraging (1) DINOv2 feature similarity across frames, (2) Cross-modal text-frame similarity, and (3) temporal frame similarity.
1. High quality on video-based QA: 67.6% on EgoSchema, 66.9% on MVBench, 65.4% on MLVU and 59.5% on VideoMME long
2. +5% accuracy boost on average across various video understanding benchmarks compared to LLaVA-OneVision and VideoChat2
3. Our edge model, LongVU-3B, also outperformed 4B counterparts such as VideoChat2(Phi-3) and Phi-3.5-vision-instruct by a large margin.
with: @xiaoqian_shen @liuzhuang1234 @Hu_Hsu @garvinchen2 @klightlm @zechunliu @balakrishnan_vr @Fanyi_Xiao @hyunwoojkim @bilgeesra @raghuraman @moElhoseiny @vikasc
Vikas Chandra retweeted
🔔 new 𝗔𝗴𝗲𝗻𝘁-𝗮𝘀-𝗮-𝗝𝘂𝗱𝗴𝗲 paper:
𝗖𝗮𝗻 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀 𝗮𝘀 𝗲𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲𝗹𝘆 𝗮𝘀 𝗵𝘂𝗺𝗮𝗻𝘀? 𝗬𝗲𝘀, 𝘁𝗵𝗲𝘆 𝗰𝗮𝗻!
📄 arxiv.org/abs/2410.10934v1
👨💻 github.com/metauto-ai/agent-…
Introducing 𝗔𝗴𝗲𝗻𝘁-𝗮𝘀-𝗮-𝗝𝘂𝗱𝗴𝗲, a groundbreaking proof-of-concept that reduces costs and time by 97%, while providing rich, intermediate feedback. It precisely captures the natural step-by-step processes of agentic systems. We also developed 𝗗𝗲𝘃𝗔𝗜, a new benchmark featuring 55 automated AI development tasks and 365 requirements. Agent-as-a-Judge not only outperforms LLM-as-a-Judge but also closely mirrors human evaluations with greater efficiency and precision.
The real game-changer? It provides reliable reward signals, paving the way for scalable, self-improving agentic systems.
Thanks my Meta/KAUST mentors/peers/collaborators @SchmidhuberAI @tydsh @zechunliu @vikasc @YoungXiong1 @vikasc @Obs01ete @erniecyc @oneDylanAshley ... – at Menlo Park, CA
Vikas Chandra retweeted
Huge congrats to @AIatMeta on the Llama 3.1 release!
Few notes:
Today, with the 405B model release, is the first time that a frontier-capability LLM is available to everyone to work with and build on. The model appears to be GPT-4 / Claude 3.5 Sonnet grade and the weights are open and permissively licensed, including commercial use, synthetic data generation, distillation and finetuning. This is an actual, open, frontier-capability LLM release from Meta. The release includes a lot more, e.g. including a 92-page PDF with a lot of detail about the model:
ai.meta.com/research/publica…
The philosophy underlying this release is in this longread from Zuck, well worth reading as it nicely covers all the major points and arguments in favor of the open AI ecosystem worldview:
"Open Source AI is the Path Forward"
facebook.com/4/posts/1011571…
I like to say that it is still very early days, that we are back in the ~1980s of computing all over again, that LLMs are a next major computing paradigm, and Meta is clearly positioning itself to be the open ecosystem leader of it.
- People will prompt and RAG the models.
- People will finetune the models.
- People will distill them into smaller expert models for narrow tasks and applications.
- People will study, benchmark, optimize.
Open ecosystems also self-organize in modular ways into products apps and services, where each party can contribute their own unique expertise. One example from this morning is @GroqInc , who built a new chip that inferences LLMs *really fast*. They've already integrated Llama 3.1 models and appear to be able to inference the 8B model ~instantly:
nitter.cf/karpathy/status/181580…
And (I can't seem to try it due to server pressure) the 405B running on Groq is probably the highest capability, fastest LLM today (?).
Early model evaluations look good:
ai.meta.com/blog/meta-llama-… nitter.cf/alexandr_wang/status/1…
Pending still is the "vibe check", look out for that on X / r/LocalLlama over the next few days (hours?).
I expect the closed model players (which imo have a role in the ecosystem too) to give chase soon, and I'm looking forward to that.
There's a lot to like on the technical side too, w.r.t. multilingual, context lengths, function calling, multimodal, etc. I'll post about some of the technical notes a bit later, once I make it through all the 92 pages of the paper :)
Vikas Chandra retweeted
MobileLLM: nice paper from @AIatMeta about running sub-billion LLMs on smartphones and other edge devices.
TL;DR: more depth, not width; shared matrices for token->embedding and embedding->token; shared weights between multiple transformer blocks;
Paper: arxiv.org/abs/2402.14905
Vikas Chandra retweeted
New ML efficiency work!
🎯We propose SpinQuant that optimizes the rotation matrices in several parts of the pre-trained transformers, so that the resulting quantized model largely retains full-precision performance.
For small models that are hard to optimize, SpinQuant can quantize weight/activation/KVs, all to 4-bits, with a reduction of the gap-to-full-precision by 30.2% (LLaMA2-7B) and by 34.1% (LLaMa3-8B), compared to concurrent work QuaRot that only uses random rotation to remove outliers.
As a result, for all 4-bits quantization, there is only a 2.9 points gap-to-full-precision for LLaMA2-7B and 4.4 points gap-to-full-precision for LLaMA3-8B, in zero-shot common sense reasoning tasks.
This is a joint work with @vikasc's team (@zechunliu et al). Thanks all for the efforts!
arxiv.org/abs/2405.16406
Vikas Chandra retweeted
📝 New from FAIR: An Introduction to Vision-Language Modeling.
Vision-language models (VLMs) are an area of research that holds a lot of potential to change our interactions with technology, however there are many challenges in building these types of models. Together with a set of collaborators across academia, we’re releasing ‘An Introduction to Vision-Language Modeling’ — we hope that this new resource will help anyone who would like to enter this field to better understand the mechanics behind mapping vision to language.
Full paper ➡️ go.fb.me/ncjj6t
This guide covers how VLMs work, how to train them and approaches to evaluation — and while it primarily covers mapping image to language, it also discusses how to extend VLMs to videos.
We hope that releasing this guide will inspire and enable more work in this space.
Vikas Chandra retweeted
Stanford just hosted a hackathon. Over 1000 students from around the world came to build for 36 hours straight.
The reward? $100k+ in prizes.
Here are the winners and crowd standouts we saw at TreeHacks ‘24 @hackwithtrees (🧵):
Vikas Chandra retweeted
EfficientSAM !
You liked SAM (Segment Anything Model)?
You're gonna love EfficientSAM!
A smaller, faster, and almost as good version of SAM.
🚨 New paper!🚨
A few months ago, Meta released Segment Anything Model (SAM). It's already in photography apps, medicine, and video-generation. Now we’ve invented SAM’s little brother, EfficientSAM. It is small but mighty!
With 20x fewer parameters and 20x faster runtime, EfficientSAM is within 2 points (44.4 AP vs 46.5 AP) of the original SAM model.
How'd we do it? I have 2 words for you: Masked Autoencoders. Check out the following for more details!
Paper: arxiv.org/pdf/2312.00863.pdf
Project details and demo: yformer.github.io/efficient-…
with: @balakrishnan_vr @klightlm @mukosame @Fanyi_Xiao @dilin_wang @fiandola @raghuraman @vikasc, et al
Vikas Chandra retweeted
🚨 New paper!🚨
A few months ago, Meta released Segment Anything Model (SAM). It's already in photography apps, medicine, and video-generation. Now we’ve invented SAM’s little brother, EfficientSAM. It is small but mighty!
With 20x fewer parameters and 20x faster runtime, EfficientSAM is within 2 points (44.4 AP vs 46.5 AP) of the original SAM model.
How'd we do it? I have 2 words for you: Masked Autoencoders. Check out the following for more details!
Paper: arxiv.org/pdf/2312.00863.pdf
Project details and demo: yformer.github.io/efficient-…
with: @balakrishnan_vr @klightlm @mukosame @Fanyi_Xiao @dilin_wang @fiandola @raghuraman @vikasc, et al
Vikas Chandra retweeted
🚀Excited to share our EfficientSAM (SAM meets MAE) for interactive segmentation.
📷Demo (@Gradio) & Project: yformer.github.io/efficient-…
@balakrishnan_vr @klightlm @mukosame @Fanyi_Xiao Chenchen Xiaoliang @dilin_wang Fei @fiandola @raghuraman @vikasc
youtu.be/KeMhbF7j62I?si=wu1C…

































