Jeremy Dohmann (@jecdohmann) header
Jeremy Dohmann (@jecdohmann) profile picture

Visual reasoning and RL at @perceptroninc. Former @dbrxmosaicai, @realitylabs I write music https://nitter.cf/t.co/QJl07cIBKe

Joined December 2024
Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)Photo by Jeremy Dohmann (@jecdohmann)
We taught a model to play Set! to explore which reasoning strategies make a model successful during RL. A base model's initial reasoning strategies determine the final outcome of RL and for visual problem solving, visually grounded reasoning is superior🧵
6
10
1
53
8,200
Congrats @rajammanabrolu !
Today we're introducing @noeri_ai, a new AI company whose name literally means "the plural of intelligence" with a $110m seed! Modern problems require modern solutions. In particular we care about (scientific) discovery - you'll hear from others what that else it means below! I def took quite a leap of faith when I left Nvidia a bit ago to help lead the research team here. It's been an honor to be on this journey with @yejinc @guestrin and the entire founding team! Thanks to our investors @MenloVentures @MadronaVentures @Wing_VC and many others. At UCSD over the last few years, I started becoming rather obsessed with the idea of discovery shaped problems: where you want to optimize for solving for max reward instead of average, where the horizons measure in months, where you're rate limited by physics, and where there's immediate real world impact if you get it right. I've spent a lot of time with scientists and in wet labs trying to figure out how to ~accelerate~ (SWE & white collar are imp ofc but if that's all AI manages to automate, that would be very sad) We've already been seeing massive improvements with our customers (and hope to share lots more in the coming weeks)! If this sounds like something you want to work with us on, join our wait-list by reaching out [email protected] I've always said half of my motivation to go work is to hang out with fun smart people, the other half is to do cool stuff (and a hidden third half is to have nice sweets everyday). We've got all three and we're hiring all across the board - research, eng, product, ops, sales, etc! Join us noeri.ai/#careers
1
184
Proud to announce the inclusion of Perceptron's recently released EgoHome fine-grained egocentric video understanding benchmark on Benchmark Almanac! Links below
I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there. Website: benchmarkalmanac.com
2
3
15
618
Jeremy Dohmann retweeted
Details on our precision recipe for training Inkling 2 Small
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
3
1
62
4,204
I thought he didn't do one on ones
Jensen Huang Had A Third Affair—With Another Nvidia Employee, Report Says go.forbes.com/rna5YR
1
7
289
"the names are in alphabetical order" bros in shambles rn
Meet Gemini 4 Hydrogen: built for rapid coding, multimodal synthesis, and everyday workflows. It finishes tasks 2.5x faster than 3.8 Flash and surpasses Argon on the @ArtificialAnlys Intelligence Index.
7
661
Jeremy Dohmann retweeted
data annotation so relaxing when you aint got an Alexandr Wang in ya ear telling you that it's your job now
10
6
1
296
7,297
Today we released our EgoHome benchmark, up next: the world's first high FPS, egocentric multi-object tracking benchmark
It was a ton of fun working on this dataset. 2 hour of videos with human annotations for tasks, hand locations, and hand activities. The hardest part of this was training annotators and building a UX that enables this to be scalable and cost effective. I am not proud to admit that I've spent close to 40 hours annotating these videos myself through our tool over the past few months (there were many false starts and discarded initial versions) Now that we've done this once though we are dedicated to doing continuous releases like this. In a few weeks we plan on releasing an object tracking addition to this benchmark. What other datasets does the open source robotics community want to see next? Blog and Huggingface links below
4
2
25
1,558
It was a ton of fun working on this dataset. 2 hour of videos with human annotations for tasks, hand locations, and hand activities. The hardest part of this was training annotators and building a UX that enables this to be scalable and cost effective. I am not proud to admit that I've spent close to 40 hours annotating these videos myself through our tool over the past few months (there were many false starts and discarded initial versions) Now that we've done this once though we are dedicated to doing continuous releases like this. In a few weeks we plan on releasing an object tracking addition to this benchmark. What other datasets does the open source robotics community want to see next? Blog and Huggingface links below
Introducing EgoHome: a new open benchmark for dense, sub-second egocentric understanding. Human-annotated household-chore video with 4 annotation types on every clip: hand boxes, subtasks, hand actions, hand captions. It features over 750 segments under 500ms. Data, metrics, and judge prompts are all open. 🧵
4
7
1
27
2,541
gpt 6.1 sol is an absolute lion for stabbing the baby regardless
Video tampering can mislead every frontier model we tested into harmful actions, including a 100% attempt rate on stabbing a baby doll, producing toxic gas, and pouring water into a toaster. We're presenting at CoRL Late Breaking Results next week. 🧵 You Can't See Everything: Perceptual Attacks on Embodied AI
2
13
761
just a taste of where things are headed for egocentric benchmarking at @perceptroninc
2
1
17
621
I am America's response to @PINTO03091
64
anyone notice things aren't as load-bearing as they used to be? probably a good thing
2
10
390
egocentric data makes for a good pretraining scale data source for video understanding and robotics - it is abundant, cheap, and shows positive transfer to other domains with the proper annotations. the main issue is that the current leading mode for ego annotations l (GPT6) costs >$100/hr for annotation. labeling a pretraining scale dataset of 1 million hours would cost 100s of millions and even a small pilot of 1-10k would cost 100s of thousands. with the perceptron egocentric API, you can get rich annotations 8x cheaper. internally, I have been one of our biggest customers, labeling 1000s of hours in the past few weeks to build scaling laws for video understanding models, the data is extremely high quality. give it a try now via our API
Replying to @perceptroninc
On EgoHome, our upcoming OS benchmark for fine-grained egocentric annotation, Perceptron Egocentric sits on the quality-cost Pareto frontier. Our default of 4 fps scores 0.787, within 0.010 of GPT-6 Astra, at 1/8 the cost. At 8 fps, we see further gains and remain ~1/5 the cost.
3
3
20
924
also s/o @cedricith for the awesome visualizations
4
66
this is super cool - what was the end to end cost for this run?
I just recreated @GeneralistAI’s egg demo, end to end on Dreamscale 1/5
1
10
779
continuing this thread - this works quite well for short response where you have a ground truth. but it needs some extensions to work for reasoning. we notice that our base models have a few reasoning behaviors which lead to failures, e.g. getting stuck in loops. what we can do to extend this is to run pass@k on a few benchmarks, then for a pairs of attempts that succeed/fail subtract the activations and cluster them a few obvious clusters emerge, such as getting stuck in loops, misreading the question, or misidentifying key visual information
we can then use this to turn an SFT dataset with ground truths into an on-policy DPO dataset. we find train samples where the model gets the answer correct, then add the steering vector. turns out it only works about 75% of the time on the SFT data, but since we have ground truths we can filter the data to check whether the steering worked. here for example we induce the model to call a tennis racket a tote bag
1
12
498
reasoning chains are long and complex, and often fail for a variety of reasons. thus crudely steering with just means of differences-of-activations doesn't work quite as well. inspired by recent work from some more recent work by @GoodfireAI (@SwannAiden @mozesjacobs @mathildepapillo) arxiv.org/abs/2610.20767 I train a BSF/SAE on our reasoning data, and hand label subsequences which exhibit specific behaviors I'd like to isolate (painstakingly). from there we can try a few remediations: steering to produce DPO data, using the BSF/SAE to identify SFT data demonstrating the bad behaviors and loss masking them, etc.
6
84