Kaon Labs

Sharing a bit on how we're evaluating hundreds of models per week internally. Having millions of users means we can test how changes to our models affect actual conversations. We've been working on making that feedback easier for our researchers to use. We used to rely on Elo comparisons and user-level A/B tests. Elo tells us which response someone prefers, but misses how a model’s replies shape the rest of the conversation. User-level A/B tests capture that, but require enough traffic and time that testing every checkpoint becomes impractical, limiting evaluation throughput. We built Session AB to test a model across multiple exchanges and measure what happens. Researchers can submit experiments through their own agents, with controls for traffic, sample size, and latency. kaonlabs.com/session-ab

Post-training on user feedback is often hard to interpret. A/B testing tells us whether users prefer the new model, but gives us little explanation of what changed. We used Anthropic’s Jacobian lens to compare our post-trained model with its instruction-tuned starting checkpoint on real conversation states, looking at their internal readouts before either generated a reply. Some differences lined up with our training goals. Others were less expected. That raises a question we want to spend more time on: when users prefer a model, what else might be changing along the way? Comparing the models before generation gives us another way to investigate this, beyond preference scores and selected outputs. We’re interested in whether these patterns can help us understand what we’re reinforcing through user feedback, and where to take the next round of training. kaonlabs.com/writing/j-space…

Current auto research benchmarks rely heavily on offline, static evaluations. We have shifted this paradigm by connecting an auto research loop directly to live online traffic—allowing agents to propose hypotheses, execute SFT experiments, and evaluate results against real-time user preferences. Over 7 experiments across 2 days, our online preference win rate increased from 56.0% to 60.1%. Crucially, the improvement came from discovering better training data distributions rather than hyperparameter tuning. Heterogeneous, real-world data distributions—closer to actual product traffic—outperformed pristine, expert-generated data. The entire pipeline—data filtering, fixed-budget SFT, candidate deployment, and online ELO evaluation—runs on a single 8×H100 node. The agent learned to pipeline training, deployment, and evaluation across GPU, network, and CPU lanes to maintain maximum hardware utilization. We have published our full technical write-up detailing the harness design, data discoveries, and our next steps regarding real-time evaluation and on-policy data flywheels: kaonlabs.com/blog/online-aut…