Alan Melling (@alanmelling) profile picture

junction of Computer Vision and Graphics, Principal R&D Engineer at Carvana, Co-Creator at Nature Time

ATL
Joined December 2010
Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)Photo by Alan Melling (@alanmelling)
Alan Melling retweeted
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks: Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better: Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better: Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better: Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work! In summary: - As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding. - Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
1,711
6,772
1,587
56,891
8,247,990
Alan Melling retweeted
For 3D learning, the “Bitter Lesson” doesn’t properly respect Kolmogorov complexity. The colloquial version of the bitter lesson is that, in a nutshell, just scaling up data and compute always beats problem-specific modeling (note that this is not actually what Sutton said; more on that later). But look at the current situation with LLMs and agents generating 3D content: the really impressive examples depend on configuring an API, language, or software package (Three.js, OpenSCAD, Blender, etc.). Each representation impacts the description size for a given object—and for the classes of object that can be faithfully represented via one of these procedural descriptions, the encoding is vastly more efficient than an explicit description (points, splats, polygons, etc.) or even an implicit description (SDF, NeRF, etc.). Moreover, the smaller you can make the procedural description, the better the LLM works: it has to predict fewer tokens to describe the same object, because each token does a better job of compressing geometric information. E.g., the words “unit sphere” are far more compact than a list of points or a collection of voxels. So, in the procedural setting, building models for predicting shape becomes an exercise in bounding Kolmogorov complexity: what’s the smallest “program” I can use to describe a given object? (Or more precisely: which language minimizes description length *and* the entropy of the description, for a model trained on that language.) That’s why current tools lean on carefully crafted abstractions (again, Blender, OpenSCAD, Three.js, etc.), for which many examples are available. Astra doesn’t run off and build you a giant neural network, hoping that the “right representation” will somehow be learned deep within the network weights. None of this, by the way, contradicts what Rich Sutton actually said: the input/output encoding matters, even if the rest is just scaling up search and learning. And the reason we see examples in Three.js and not a more esoteric language is that there is a large corpus of examples to learn from. But the idea that the network will just “learn the best representation” by adjusting its weights is a misguided fiction. Instead, it really seems like we are entering an era of “code as geometry” counterposed to WorldLabs’ “geometry as code”, where design of domain-specific languages is a key task. (And of course, search and learning will inevitably play a role in this design…) Computer Graphics also plays a central role, because it’s the mechanism for decompressing the procedural description into an explicit representation (building a cylinder, tessellating a curve, etc.). That’s the tool calling part of the agentic process. And the better these graphics algorithms get, the more effective the language becomes. People are worried about the death of Computer Vision—but it seems Graphics is coming back from the grave. 🧟‍♂️ (And, might I add, that differential geometry is a very expressive language that few are taking advantage of in this setting. 😉)
24
51
6
427
43,551
Alan Melling retweeted
Had the surreal experience of telling a room full of computer vision researchers at the ICCV25 AC workshop why “computer vision researcher” won’t be a thing in 5 years 🌶️ Of course, this was an extreme stance to keep things lively in a fun debate setting but it echoed some of my own internal monologue over the past few years as someone who has identified as a computer vision researcher for the last decade. The argument went as follows: ⚡️a research community needs a set of core problems and methods that are specific to that community ⚡️the vision community used to have these 10-15yrs ago but today’s general purpose multimodal architectures assume very little about the input/output modality and are likely to subsume more tasks and modalities over time ⚡️time and again we have had to swallow the bitter pill — methods that bake human intuition into learning algorithm might show gains in the short term but are eventually surpassed by more general methods that utilize more data and compute - llms, vlms, sora, genie etc ⚡️gains in vision systems over the last many years have come from things that have nothing to do with vision or images but general advances in deep learning - optimizers, normalization layers, attention, residual connections, quantization, parallelization methods, larger models etc. Computer vision ends at tokenization and then deep learning and distributed systems engineering take over. ⚡️so not only would “vision researcher” be obsolete, we must actively fight the urge to play a “computer vision researcher” to avoid our biases from creeping into our AI systems ⚡️in short, there is nothing uniquely vision in today’s computer vision research and there is too much overlap with other specialized communities like robotics, graphics etc Thanks again @ICCVConference PCs for hosting the debate and @anikembhavi for inviting me to participate! It was incredibly awesome for everyone at the AC workshop to take this discussion in a fun spirit 🙌 Arguing for the motion with me were @sarameghanbeery and @RoozbehMottaghi In our opposition were @HildeKuehne @aagrawalAA and @bluevincent If you are a vision researcher, share your thoughts whether you agree or not!
10
21
4
209
29,078
VLAs have become the fastest-growing subfield in robot learning. So where are we now? After reviewing ICLR 2026 submissions and conversations at CoRL, I wrote an overview of the current state of VLA research with some personal takes: is.gd/1pqw9w
11
101
8
529
53,747
Is the terror reign of redundant scene representations ending? Where VGGT, CUT3R, and other recent models relied on godless redundant outputs (depth+points+pose) without guaranteeing internal prediction consistency, MapAnything and DepthAnything 3 are now heroically pushing back.
4
11
2
70
21,327
Alan Melling retweeted
𝗗𝗟𝗥 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵𝗲𝗿𝘀 𝗴𝗮𝘃𝗲 𝗮 𝗿𝗼𝗯𝗼𝘁𝗶𝗰 𝗮𝗿𝗺 𝗳𝘂𝗹𝗹-𝗯𝗼𝗱𝘆 𝘁𝗼𝘂𝗰𝗵 𝘀𝗲𝗻𝘀𝗶𝘁𝗶𝘃𝗶𝘁𝘆 𝘄𝗶𝘁𝗵 𝗻𝗼 𝗮𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝘀𝗸𝗶𝗻 𝗻𝗲𝗲𝗱𝗲𝗱. They used internal force-torque sensors at 8 kHz + deep learning. The robot can feel where you touch it, recognize letters drawn on its surface, and respond to virtual buttons placed anywhere on its body. What's interesting is the infrastructure behind it. To train these models, you need high-frequency sensor streams, manifold learning to unfold trajectories, and the ability to iterate fast. They collected 2,300 samples from 20 people and hit 95.5% accuracy on digit recognition. This is what's possible when you have the right data infrastructure. 📄 lnkd.in/exgWfeXf Video credit: @DLR_en
54
333
75
2,289
173,953
Can mathematical models of history help us escape the worst outcomes? @Peter_Turchin, the pioneering complexity scientist, and Interintellect founder @TheAnnaGat explore his groundbreaking work on cliodynamics, recurring cycles of societal collapse, and America's current position on the brink of upheaval. Watch the full, thought-provoking conversation: youtu.be/52hM9vbi-ck
2
5
1
14
2,969
LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA. thinkingmachines.ai/blog/lor…
83
553
177
3,467
1,460,378
Alan Melling retweeted
🎉CameraBench has been accepted as a Spotlight (3%) @ NeurIPS 2025. Huge congrats to all collaborators at CMU, MIT-IBM, UMass, Harvard, and Adobe. CameraBench is a large-scale effort that pushes video-language models to reason about the language of camera motion just like professional cinematographers. 🌍 Our open-source dataset, models, and code are also gaining strong interest and adoption from frontier labs such as DeepMind and Kling to advance video generation research. 📄Paper: arxiv.org/abs/2504.15376 🌐 Website: linzhiqiu.github.io/papers/c…
📷 Can AI understand camera motion like a cinematographer? Meet CameraBench: a large-scale, expert-annotated dataset for understanding camera motion geometry (e.g., trajectories) and semantics (e.g., scene contexts) in any video – films, games, drone shots, vlogs, etc. Links below! We contribute a taxonomy of motion primitives, co-designed over months with professional cinematographers, and apply rigorous quality control to label and caption all aspects of camera motion. CameraBench shows that even the best SfMs and VLMs struggle with real-world, dynamic videos. Yet, a generative VLM post-trained on our high-quality data matches SOTA SfM (MegaSAM) in geometric understanding and outperforms SOTA VLMs (Gemini-2.5 / GPT-4o) in semantic understanding, e.g., describing how the camera moves. 📄 Paper: huggingface.co/papers/2504.1… 🌐 Website: linzhiqiu.github.io/papers/c… Work led by CMU, MIT-IBM, UMass, Adobe, Harvard, Emerson with @censiyuan1, @chancharikm, @JayKarhade, @du_yilun, @gan_chuang, and @RamananDeva.
7
25
2
160
26,244
Alan Melling retweeted
how crazy they have put 15TB worth of physics simulation datasets on internet
17
298
12
3,232
125,743
I was lucky to work in both China and the US LLM labs, and I've been thinking this for a while. The current values of pretraining are indeed different: US labs be like: - lots of GPUs and much larger flops run - Treating stabilities more seriously, and could not tolerate spikes in large flops run, thus invented so many stability-related tricks, including all kinds of soft-cap, MuP, and spectral norm control tricks - Treats predictabilities more seriously. Check GPT 4 report for reference, even trying to predict the eval task performances - Because of the stability and predictability ask, treats hyper-params and optimization more seriously - Generally believe more in data, optimization than arch China labs be like: - has very limited GPUs, e.g. k2 in 4k GPU and v3 in 2k GPU - as a result, pushing for the limit of pretrain modeling-infra co-design, see so many tricks in V3, and K2 has some cool stuff too (the offload trick helps remove the stupid MoE gating constrain and only uses EP 16) - cares model arch/token efficiency over optimization, stability - cares more about data quality than data quantity - taking inference into consideration day 0, even before the training starts In general, China labs are trying to use <4e+24 flops models to catch up with >1e+25 flops models. It is hard or impossible, but they are making good progress. I am actually very happy to see Qwen's new try on model archs, they used to be focusing more on data side rather than on model arch side. They developed linear attn, not just for people to think they are innovating, it is actually considering pushing the limit for test time scaling. Llama4 failed for many reasons, but qwen-next is different. They just used very limited flops and it is a brave try for good reasons.
I bet OpenAI/xAI is laughing so hard, this result is obvious tbh, they took a permanent architectural debuff in order to save on compute costs.
58
323
36
3,127
526,780
Alan Melling retweeted
You say this but a lot of early to mid 20th century physics papers were like "notes about what kind of things I'm thinking about the stuff we are all thinking about" 😛
1
10
830
Towards the Next Generation of 3D Reconstruction @Parskatt PhD Thesis. tl;dr: would be useful in teaching image matching - nice explanations. (too) Fancy and stylish notation. Cool Ack section and cover image. liu.diva-portal.org/smash/re…
1
17
125
6,466
Introducing: Hyperscape Capture 📷 Last year we showed the world's highest quality Gaussian Splatting, and the first time GS was viewable in VR. Now, capture your own Hyperscapes, directly from your Quest headset in only 5 minutes of walking around. meta.com/experiences/8798130…
Hyperscape: The future of VR and the Metaverse Excited that Zuckerberg @finkd announced what I have been working on at Connect. Hyperscape enables people to create high fidelity replicas of physical spaces, and embody them in VR. Check out the demo app: meta.com/experiences/7972066…
42
264
40
1,954
341,086
Alan Melling retweeted
Goodbye SOGS, hello SOG! 👋 PlayCanvas open sources Spatially Ordered Gaussians - a new super-compressed format for 3D Gaussian Splatting. blog.playcanvas.com/playcanv…
8
72
7
424
48,504
How AI Taught Itself to See Self-supervised learning is fascinating! How can AI learn from images only without labels? In this video, we’ll build the method from first principles and uncover the key ideas behind CLIP, MAE, SimCLR, and DINO (v1–v3). Video 👇
12
85
6
569
36,201
beach 💔 dolly in
102
965
35
10,284
296,823
Alan Melling retweeted
AI is great at hitting explicit goals, but often at the cost of the hidden ones. Terence Tao just wrote about this. He points out: AI is the ultimate executor of Goodhart’s law, i.e. when a measure becomes the target, it stops measuring what we care about. Take a call center. Management sets a KPI: “shorten average call time.” Sounds reasonable: shorter calls should mean faster resolutions, happier customers. At first, it works. Agents become more efficient. But soon, people start gaming it: nudging customers to hang up when the problem is tricky, or just dropping the call themselves. The numbers look amazing. Call times plummet. But customer satisfaction? Straight into the ground. Now replace “call time” with “prove theorem X.” If human mathematicians did it, they’d refine definitions, polish lemmas, contribute back to Mathlib, train juniors, deepen the understanding of math structures, and strengthen the community. The AI, by contrast, optimizes only for the explicit goal. It might generate a 10,000-line proof in hours. Perfectly correct, but unreadable, unusable, and useless for human learning. The summit is reached but the forest along the way is gone. We need to start making our implicit goals explicit and design systems that protect the values we actually care about, not just the numbers we can measure.
79
114
23
954
90,074