Leon Qi retweeted
We spent over 500 dollars running Claude Fable 5 on χ-Bench.
But it didn't win.
Across 75 healthcare workflow tasks, Fable 5 achieved 24.0% pass@1. Claude Opus 4.6 scored 28.0% — and cost less. Fable spent $29.20 per passed task which has weakest return in this comparison. A bigger model bill doesn't necessarily buy a better agent.
We will keep the χ-Bench leaderboard up to date with latest frontier models.
See more results tomorrow.
#ChiBench #AIAgents #HealthcareAI #AIEvaluation #HealthTech #ClaudeFable5
Leon Qi retweeted
𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗖𝘂𝗿𝗮: 𝟭𝗧 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿 𝗺𝗼𝗱𝗲𝗹 𝗳𝗼𝗿 𝗮𝗴𝗲𝗻𝘁𝗶𝗰 𝗵𝗲𝗮𝗹𝘁𝗵𝗰𝗮𝗿𝗲 🏥 by actAVA AI.
🤗: huggingface.co/papers/2607.1…
Within hours of launch, we received 100+ healthcare professionals sign-ups🚀Across clinical reasoning, medical evaluation and diagnosis, and long-horizon healthcare workflows, 𝗖𝘂𝗿𝗮 𝘀𝘂𝗿𝗽𝗮𝘀𝘀𝗲𝘀 𝗚𝗣𝗧-𝟱.𝟲, 𝗖𝗹𝗮𝘂𝗱𝗲 𝗢𝗽𝘂𝘀, 𝗮𝗻𝗱 𝗚𝗲𝗺𝗶𝗻𝗶 on HealthBench, MedXpertQA, AgentClinic, MedAgentBench.
Enterprises should not have to rent all of their intelligence from frontier labs. We help customers turn open models into 𝘀𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗲𝗱, 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿-𝗼𝘄𝗻𝗲𝗱 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 𝗯𝘂𝗶𝗹𝘁 𝗮𝗿𝗼𝘂𝗻𝗱 𝘁𝗵𝗲𝗶𝗿 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀, 𝗽𝗼𝗹𝗶𝗰𝗶𝗲𝘀, 𝘁𝗼𝗼𝗹𝘀, 𝗮𝗻𝗱 𝗶𝗻𝘀𝘁𝗶𝘁𝘂𝘁𝗶𝗼𝗻𝗮𝗹 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲. Cura is the first demonstration of that vision at scale.
We believe the next generation of enterprise AI will not be one model rented by everyone. It will be thousands of specialized models owned and continuously improved by individual enterprises.
We are beginning conversations with investors who share that view. My DMs are open.
Proud of this one. A 1T-parameter model trained on the actual work healthcare organizations do — not scraped internet text.
Beats Opus, matches Fable on six of the hardest healthcare evals, at 20–100x lower inference cost. Running in the customer's own VPC.
Ship day. 🚀
Today we launched Cura, a 1-trillion-parameter model trained for clinical and administrative healthcare work. With it, actAVA becomes the first U.S. startup to help healthcare organizations customize and own a model at this scale.
Earlier this year we ran χ-BENCH, a study of real healthcare administrative tasks. The best frontier agent we tested, Claude Opus 4.8 with Claude Code, completed 33% of the work. Researchers at Stanford, Johns Hopkins, Yale, and Salesforce AI Research reviewed the study. 33% is not a prompting problem. You don't fix it with better retrieval or a longer system prompt.
So we trained our own model, on the actual work these organizations do. Across six of the hardest healthcare evaluations, CURA beats Claude Opus and matches Claude Fable, the leading frontier models, at 20 to 100 times lower inference cost. It runs standalone or on KORA, our agent platform, and deploys inside the customer's own VPC. Patient data never leaves their control.
The part we keep coming back to is ownership. Right now, most healthcare organizations rent their intelligence: per-seat tools charge a markup on every model call, and the customer's operational knowledge quietly improves someone else's product. Owning the model flips that. Your data, your standards, your infrastructure, and the model gets better on your work instead of a vendor's.
Healthcare has the data, the domain depth, and the economic pressure to be the first industry that runs on models it controls. That's the bet.
More at actava.ai/cura
Leon Qi retweeted
The strongest healthcare LLM, custom-built for your enterprise, owned by you🌸
Meet @actAVAai Cura: 1T agentic model trained by recursive self-improvement for long clinical + health admin workflows
Try: actava.ai/cura
Share your use case: $20 credits + early access👇
Leon Qi retweeted
Huge congrats to @Humana's Erius agent taking the #1 spot on CHI-Bench for Prior Auth and 6th for all domains. It outperforms every frontier lab on one of healthcare's hardest workflows.
Opus 4.8 is in the lead @claudeai @ClaudeDevs
CHI-Bench leaderboard just gets updated with the newest and highest score from @claudeai Opus 4.8.
CHI-Bench is world's first long-horizon benchmark for healthcare AI agents.
Leaderboard: actava.ai/benchmarks
Leon Qi retweeted
CHI-Bench is the world's 1st long-horizon healthcare benchmark for AI agents.
If you're building or buying AI for healthcare, this is the test that actually matters — real clinical workflows, not toy demos.
U.S. healthcare needs this. 🏥🔬
The best AI agent (Claude Code + Claude Opus 4.6) passes only 28% of real healthcare workflow tasks. CHI-Bench by @actAVAai @iscreamnearby @HaolinChen11, built with Johns Hopkins, Yale, Stanford, CMU, Oxford and 20+ institutions, was designed to find out exactly how far we are. 🏥 Try it yourself 👉 modelscope.ai/datasets/actav…
Three long-horizon domains tested:
🏥 Prior Authorization: provider intake and PA preparation for new referrals
📋 Utilization Management: full payer review cycle from intake to peer-to-peer
👥 Care Management: chronic disease follow-up, outreach, assessment, care planning
75 tasks + 3 marathon tasks + 23 end-to-end dual-agent scenarios. 20 medical apps via MCP, 1,279-document handbook.
💻 Git: github.com/actava-ai/chi-ben…
🔗 Leaderboard: actava.ai/benchmarks
Leon Qi retweeted
actAVA AI integrates CHI-Bench with @huggingface and @harborframework today.
Users can run the CHI-Bench evaluation and RL training from both platforms.
Introducing CHI-Bench on @huggingface: the world’s first long-horizon healthcare benchmark for AI agents.
75 real healthcare workflows + 20 apps + 200+ MCP tools + 1,290 skills + process / outcome rewards
huggingface.co/datasets/acta…
Any questions, lmk!
Check out our dataset on Hugging Face.
Great news. Today we ranked #6 most popular dataset on Hugging Face! Wow! 😊 huggingface.co/datasets
Awesome!
🚨 Historic moment for @actAVAai ! 📷Just one day after launch, our benchmark dataset is already #10 most popular on Hugging Face — out of 1 million+ datasets! Huge thanks to @iscreamnearby , @HaolinChen11 , Deon Metelski, Leon Qi, Tao Xia, Joon Lee, Steve Brown, Kevin Riley, T. Y. Alvin Liu, M.D., Zhiwei Liu, Qingsong Wen, @CaimingXiong , Sanmi Koyejo, Eric Xing & all our collaborators. 📷📷
Great work! @HaolinChen11 @iscreamnearby
(1/n) After a few months of work with multiple hospitals, universities and research facilities, today we're open-sourcing CHI-Bench: the first long-horizon benchmark for healthcare AI agents on real clinical and healthcare workflows.
Best frontier agent overall: 28% pass@1.
End-to-end prior authorization: 0%.
A thread on what we found 🧵
1/ Introducing CHI-Bench 🧵
Can AI agents automate U.S. healthcare workflows end to end — given only clinician & insurer apps, operations, and a medical policy library?
75 long-horizon workflows × 30 frontier agents. Best agent solves just 28%.
#AIinHealthcare 👇
7/ …and university & industry AI research labs 🔬
@StanfordAILab @zeyu1tang, @sanmikoyejo
CMU & @mbzuai: @XiangchenSong, @LingjingKong, @kunkzhang, @ericxing
UCSD: @ffeng01, @huang_biwei
@SFResearch: @JimZhiwei
UW: @zixianma02
Northeastern: @hjian42
8/ …and more 🔬
Brown: @FangliGeng
Boston College: @YuanYuan_MIT
Stony Brook: @Charlesyooo1
Oxford: @qingsongedu
ASU: @realhuawei, Yanjie Fu
USC: Yue Zhao
Emory: @yangji9181
@Recursive_SI: @CaimingXiong
UIC: Philip S. Yu