“Surprisingly, our bubbleforce equilibrium is secretly a theory of momentum invoking the geometry of polynomial roots on the complex plane.” 🤯🫧
we discovered the edge of stability for muon
(and also every other non-adaptive optimizer)
we call it bubbleforce.
also we shaved off 100 iterations from the nanoGPT muon record (-3.2%)
11
663
Cade Daniel 🇺🇸 retweeted
we discovered the edge of stability for muon
(and also every other non-adaptive optimizer)
we call it bubbleforce.
also we shaved off 100 iterations from the nanoGPT muon record (-3.2%)
Cade Daniel 🇺🇸 retweeted
In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes.
It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.
Cade Daniel 🇺🇸 retweeted
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
Cade Daniel 🇺🇸 retweeted
Yes, it's true. After years working as a researcher in the field of AI, my jumpshot is finally on camera. And @oneill_c beat me in a friendly game of HORSE. or did he?....
(also thanks to BaseTen for letting me keep the ball, along with many bags of Cheez-Its)
AI researchers' idea of a good time: playing HORSE and debating continual learning between shots.
@oneill_c (co-head of training at Baseten) and @jxmnop (co-founder and head of research at @EngramLab) discuss:
- when to put knowledge in weights vs. the KV cache,
- why forgetting matters less for specialized models,
- on-policy self-distillation vs. RL, and
- what research publishing should look like today.
Watch the full game (and discussion) here: youtu.be/q_-_t5AmKz0
Work from @EngramLab on study compute reducing inference tokens while simultaneously increasing their quality👇
Replying to @EngramLab
Our models study the workspace through thousands of rollouts before deployment.
When solving new tasks, they produce better responses and get to the right answer faster.
With study, we're able to outperform Opus 4.8 X-high while using 3.3x fewer tokens.
Cade Daniel 🇺🇸 retweeted
Today we're publishing our first research blog, Understanding a Law Firm through Study.
We're sharing a glimpse of a future where agents are trained with native memory:
Cade Daniel 🇺🇸 retweeted
In collaboration with @harvey, we’re excited to build a new kind of agent environment to reflect realistic knowledge work: an entire synthetic law firm, Calderwood & Harkness, with over 100M (!) tokens of documents and 250 client cases. 💼
Today’s AI models know a lot about the law, but don’t understand how the law is practiced, because this information remains proprietary within firms. Unlike how agents are benchmarked today – starting each task from scratch with a new set of context — lawyers accumulate knowledge over time, building on years of experience.
Calderwood & Harkness makes it possible for agents to do the same. Legal agents do many tasks in the same environment, making it possible to leverage memory and experience to do better work over time.
We’ve had a great time co-developing this benchmark with Harvey’s research team, partnering @ItsJulioPereyra @nikogrupen @gabepereyra. We’ll share more results on this soon.
RT @harvey: We're open sourcing a 100M+ token synthetic law firm we built with @EngramLab.
The firm contains work product from 250+ synthe…
This quoted post is unavailable.
Cade Daniel 🇺🇸 retweeted
the longevity of the DSV3 architecture should be an update to anyone whose world model hinges on unbounded algorithmic progress via recursive self-improvement
Cade Daniel 🇺🇸 retweeted
Thrilled for my friend @dan_biderman to launch @EngramLab
Dan is a visionary who has consistently let seen around corners in AI
Cade Daniel 🇺🇸 retweeted
I'm really excited to be working with Engram to bring their models into Custom Agents in Notion. There's something magical that happens when the agent stops needing to search your workspace, and just knows it.
Cade Daniel 🇺🇸 retweeted
this is insanely cool. engram is pretty much working on a model that can improve itself through every interaction across *all* users
which will likely be the single biggest capabilities jump since o1-style chain of thought reasoning
Cade Daniel 🇺🇸 retweeted
Now that Tacit Labs and Engram are out of stealth, a casual plug for interning on the research team at @databricks:
* @ninklefitz → Founder @tacitlabsco
* @lindensli → Founder @appliedcompute
* @dan_biderman → Founder @EngramLab
* @ChrisRinard → Founder @Standard_Kernel
RT @simran_s_arora: So thrilled for @EngramLab, working on such an important problem!!! Had the pleasure of working closely with @EyubogluS…
This quoted post is unavailable.
Cade Daniel 🇺🇸 retweeted
Have been excited about this team for a long time! Memory is one of the next big unlocks in AI. Thrilled to partner with this exceptional group @dan_biderman, @EyubogluSabri, @realJessyLin, @jxmnop, @scott_linderman, and Chris.
There will be many new internet-scale datasets, created by agents
Agents will exponentiate the amount of data that exists in the world. Most are focused on building agents, @dan_biderman and the @EngramLab team are one of the few groups thinking about what happens to knowledge work when it is agent-led.
Congrats to them on the launch and pumped to continue working together on scaling compute for knowledge work.
Cade Daniel 🇺🇸 retweeted
Engram is the bitter lesson applied to context.
Whenever people ask me about the coolest under the radar companies I tell them about Dan. No longer under the radar, very excited for @dan_biderman and co to be sharing Engram with the world!

























