Excited to share one of the essential infrastructure in Kaon! Session level AB is how we ground all our ideas into concrete measurable treatment effects. This vastly speeds up our iteration, and allows for true online research
Sharing a bit on how we're evaluating hundreds of models per week internally. Having millions of users means we can test how changes to our models affect actual conversations. We've been working on making that feedback easier for our researchers to use.
We used to rely on Elo comparisons and user-level A/B tests. Elo tells us which response someone prefers, but misses how a model’s replies shape the rest of the conversation. User-level A/B tests capture that, but require enough traffic and time that testing every checkpoint becomes impractical, limiting evaluation throughput.
We built Session AB to test a model across multiple exchanges and measure what happens. Researchers can submit experiments through their own agents, with controls for traffic, sample size, and latency.
kaonlabs.com/session-ab
Straight up cartoonish alien technology. Absolutely incredible work
Today in @Nature, we report a new approach to studying human brain development and modeling disease in vivo.
Human neural #assembloids and #organoids have opened new ways to study development in vitro, but they also have important limitations. Transplantation can provide a more physiological environment, yet human cortical cells then develop within a rodent brain that matures much faster, constraining their growth and integration.
To address some of these limitations and obtain more advanced functional readouts relevant to disease biology, we generated #apallial mice, in which ~98% of the cortex and hippocampus fail to form, creating space for transplanted human cortical organoids to grow and integrate extensively.
We call these #xenocortical mice (XCX).
Nearly 7 years in the making. Congratulations to an extraordinary team and wonderful collaborators!
Link to the article below 👇
Alex retweeted
nobody wants to hear this but the classical NASA systems engineering is the perfect model for developing code with LLMs. people try to approximate this with planning modes, but if you’re explicit in your docs it’s never been easier to build, test, and verify complex codebases.
Super excited to share this with the community; always been wanting to do something like this and Kaon's unique position makes this sort of exploration possible
Current auto research benchmarks rely heavily on offline, static evaluations. We have shifted this paradigm by connecting an auto research loop directly to live online traffic—allowing agents to propose hypotheses, execute SFT experiments, and evaluate results against real-time user preferences.
Over 7 experiments across 2 days, our online preference win rate increased from 56.0% to 60.1%. Crucially, the improvement came from discovering better training data distributions rather than hyperparameter tuning. Heterogeneous, real-world data distributions—closer to actual product traffic—outperformed pristine, expert-generated data.
The entire pipeline—data filtering, fixed-budget SFT, candidate deployment, and online ELO evaluation—runs on a single 8×H100 node. The agent learned to pipeline training, deployment, and evaluation across GPU, network, and CPU lanes to maintain maximum hardware utilization.
We have published our full technical write-up detailing the harness design, data discoveries, and our next steps regarding real-time evaluation and on-policy data flywheels:
kaonlabs.com/blog/online-aut…
Now we see economists entering the game :)
One line of code is all it takes to prevent LLM agent delusions, instead of post-training patches like RL.
love4all.ai/blog/why-it-is-i…
❤️ 4 ∀
github.com/nandodef/love4all…
Something I've been working on: a chrome extension that can scrape Gemini or ChatGPT sessions and automatically sorts your chats into piles / shelves for bookkeeping. I find myself frequently using this as notetaker + daily journal (via live chat) + idea dump and evolver so I decided to opensource it just in case people have usecases.
The original incentive for doing this is just that I find myself chatting with live models a lot but the live models don't really have the capabilities I need to do in-depth idea exploration. Also there's a lot of ad hoc things that I speak during these sessions that are just not captured and it only exists in the chat history.
I want something to easily port these into more powerful models to make them more streamlined for bookkeeping and then use it for some personal knowledge graph building so that I have a pile of things that I can use agentic search on top of or unleash my openclaw on top of. It should be a self-evolving pile and I think a lot of the information just comes from my daily chat with these AI providers, especially via live mode, sometimes via voice input into thinking modes.
A core design decision for the factual versus idea split is that language models already have an internal representation of facts and building knowledge graphs on existing facts is just not productive or useful in most scenarios involving agents. Ideas are more dynamic. There are supporting and opposing arguments; the ideas evolve. There are contradicting ideas. You need to attribute the idea to different people or sources. It's more dynamic and it will be more interesting to visualize the growth of the ideas. I think that merits more of a care approach so I separate them into these piles and then each pile will have its separate processing pipelines. This should be just like an all-in-one plug-in that you can self-host that logs and book keeps all the interactions with these main AI providers in your daily life.
Currently supporting Gemini, ChatGPT, and Grok.
Still actively polishing, any suggestions welcome.
Here's the repo btw, will publish to chrome extension store after thorough testing. github.com/nociza/SaveMyCont…
Why isn't there an opensource browser extension that scrapes all the AI chat histories from major chat providers?
I'm not sure if people actually working in the field consider this goated or irrelevant. But the Lex's podcast with @drmichaellevin is one of the most intriguing onslaught of ideas I've had the pleasure of hearing in a very very long time.
