DataLab Research

This is a great post on building benchmarks! Though I do disagree on the point that every task must be outcome-verified (as we are grading outcomes, not the process to get there) Pure outcome grading in agent benchmarks can be too narrow. An agent can pass the final tests by luck, reward-hacking, or a brittle path while never properly updating its beliefs or decisions as new information arrives in a multi-turn environment It's important to also measure whether the agent actually updates its predictions and actions in the right direction when the world changes or as new information is introduced, not just whether the final container state looks correct. A good rubric can take into account positive/negative rewards along the trajectory

Excited to share that my paper, "Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents", is now available on @arxiv (link in the comments). The paper was accepted to the inaugural RLEval Workshop at @CAISconf- a workshop focused on methods and reinforcement learning environments for evaluating AI agents - and was selected for an invited talk based on reviewer ratings. LLM evaluation is difficult. Models that look equally capable on traditional benchmarks can behave very differently when the underlying facts change. Most current benchmarks focus on whether a model's output looks correct. But in real-world settings, especially in healthcare, what often also matters is whether the model appropriately updates its recommendations when the underlying facts change. In this work, I introduce the Causal Sensitivity Score (CSS), a pre-registered counterfactual evaluation framework designed to measure exactly that: "When clinically important patient facts change, does the model appropriately change its recommendations?" Across six frontier models and several hundred oncology tumor board cases, I found that models with very similar performance on standard coverage-based metrics often behave dramatically differently under counterfactual interventions. In fact, model rankings were nearly reversed depending on whether you measured coverage or responsiveness. The paper also shows that these findings transfer to tool-using agents, revealing failure modes that remain hidden under conventional evaluation approaches. The broader takeaway is that producing the right answer and responding appropriately to new information are distinct capabilities - and future evaluation frameworks should measure both. This work is also closely aligned with the mission of our DataLab at @withprotegeai: building rigorous datasets, benchmarks, and evaluation frameworks that better reflect how AI systems perform in the real world. As AI moves into increasingly complex, high-stakes domains, measuring what models know is important - but we also need to measure how they respond when reality changes. I'm grateful to my colleagues and collaborators, especially @engyziedan and Wes Hopkins, as well as the medical professionals who helped validate the results.