Leading AI Research at Actava AI @actAVAAI, RL, Agents, Ex-Salesforce AI Research @SFResearch, Ph.D. in Applied Mathematics @ucdavis
Palo Alto, CA
Joined May 2022
- Tweets89
- Following128
- Followers154
- Likes92
Pinned Tweet
(1/n) After a few months of work with multiple hospitals, universities and research facilities, today we're open-sourcing CHI-Bench: the first long-horizon benchmark for healthcare AI agents on real clinical and healthcare workflows.
Best frontier agent overall: 28% pass@1.
End-to-end prior authorization: 0%.
A thread on what we found ๐งต
Haolin Chen retweeted
Our paper ฯ-Bench was accepted as a poster at NeurIPS 2026 ๐ We built a ๐๐ถ๐บ๐๐น๐ฎ๐๐ฒ๐ฑ ๐ต๐ผ๐๐ฝ๐ถ๐๐ฎ๐น ๐ฟ๐๐ป๐ป๐ถ๐ป๐ด ๐ฎ๐ป ๐๐ฝ๐ถ๐ฐ-๐๐๐๐น๐ฒ ๐๐๐ฅ ๐ฎ๐ป๐ฑ ๐ฐ๐ผ๐ป๐ป๐ฒ๐ฐ๐๐ฒ๐ฑ ๐ถ๐ ๐๐ผ ๐ฎ ๐๐ถ๐บ๐๐น๐ฎ๐๐ฒ๐ฑ ๐ต๐ฒ๐ฎ๐น๐๐ต ๐ฝ๐น๐ฎ๐ป ๐ฟ๐๐ป๐ป๐ถ๐ป๐ด ๐ฎ ๐จ๐ป๐ถ๐๐ฒ๐ฑ๐๐ฒ๐ฎ๐น๐๐ต๐ฐ๐ฎ๐ฟ๐ฒ-๐๐๐๐น๐ฒ ๐ฝ๐ฎ๐๐ฒ๐ฟ ๐๐๐ฎ๐ฐ๐ธ.
It has 20 apps, 87 MCP tools, and a 1,290+ document operations handbook covering medical, insurance and operational policy.
Three things make this hard, and most benchmarks don't test them. Decisions depend on a large library of rules. One task means playing several roles and handing work between them. And many steps are multi-turn conversations, like peer-to-peer reviews and patient outreach, where mistakes can't be undone.
We think similar gaps will show up in other enterprise domains with dense policy, multiple roles and irreversible steps. That's why we built ฯ-Bench as an RL environment and not just a leaderboard.
Thank you to our 33 co-authors, and to the clinicians and operations experts who helped make the simulation realistic.
We did it! ฯ-Bench was accepted as a NeurIPS 2026 Evaluations & Datasets Track poster! ๐
We tested AI agents on 75 end-to-end healthcare workflows across prior auth, payer UM, and care management. Paper: arxiv.org/abs/2605.16679
Huge thanks to my coauthors: @DeonActava , @dmon2048 , Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, @FWang9959 , T. Y. Alvin Liu, Hank Capps, @zeyu1tang , @XiangchenSong , @LingjingKong , @ffeng01 , Tianyi Zeng, @JYJimLiu , @zixianma02 , Hang Jiang.
And @FangliGeng , Yuan Yuan, @Charlesyooo1 , @qingsongedu , Hua Wei, Yanjie Fu, Yue Zhao, Carl Yang, @huang_biwei , @kunkzhang , @CaimingXiong , @sanmikoyejo , @ericxing , Philip S. Yu, and @iscreamnearby . Grateful for all the work that went into this!
๐ค Made with AI
Just submitted the rebuttals for the NeurIPS. And now we are updating the leaderboard again!
The ฯ-Bench frontier moved again.
Claude Code + Claude Opus 5 produced the best result weโve measured so far: 37.3% PASS@1 (28/75) at $359.25 for one complete run.
Compared with:
Claude Opus 4.6: +9.3 points and $126.00 less per run
Claude Fable 5: +13.3 points and $166.34 less per run
We observe that Opus 5 barely make workflow errors: only 5 of 75 trials ended in workflow-completion errors. The remaining 42 failures were classified as clinical-reasoning failures. No not tool-use, policy-compliance, hallucination, or stuck failures.
This is meaningful progress, but also a reminder of what remains: even the strongest result completes fewer than four in ten long-horizon healthcare workflows end to end.
#HealthcareAI #AIAgents #LLMEvaluation #Benchmarking
Today, weโre adding Inkling to the ฯ-Bench leaderboard. Inkling passed 6 of 75 tasks, while 58 of 69 failures were workflow or tool-use errors.
โข 8.0% PASS@1
โข $69.26 per full run
โข $11.54 per passed task
Using the ฯ-Bench failure taxonomy, the 69 unsuccessful trials broke down as:
โข 36 workflow-completion failures
โข 22 tool-use errors
โข 8 clinical-reasoning failures
โข 3 stuck or abstained
Take a deeper look: 20 out of 22 tool-use errors are caused by model calling a wrong tool alias that is not provided to it, given it has sufficient tools and could use them to finish the task.
We are also evaluating Opus 5 which is released in this morning, stay tuned!
#ChiBench #AIAgents #LLMEvaluation #HealthcareAI #AIEngineering
Intelligence is safe only if you OWN it.
For my first post, Iโm sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-Wโฆ
Yesterday, we published our Claude Fable 5 result on ฯ-Bench: 24.0% PASS@1 at $525.59 per run. Today, GPT-5.6 Sol came in at 25.3% and $154.52 per runโ1.3 points above Fable 5 for $371.07 less.
Within the GPT-5.6 family:
Luna โ 13.3% PASS@1 | $32.65/run | $3.27/passed task
Terra โ 13.3% PASS@1 | $62.63/run | $6.26/passed task
Sol โ 25.3% PASS@1 | $154.52/run | $8.13/passed task
Luna and Terra tied on PASS@1, but Terra cost nearly twice as much. Sol gained 12 points over Luna at 4.7ร the run cost.
Across all three models, every unsuccessful trial fell into one of two categories: workflow completion or clinical reasoning.
We are still evaluating other models including recent open-weights competitors to these proprietary models. The next result comes tomorrow.
#ChiBench #OpenAI #AIAgents #LLMEvaluation #HealthcareAI
Training intelligence will, and is consistently getting cheaper and democratized. What's extremely valuable is your data, workflows, SOPs, and caveats and experience in you vertical. Our belief and vision is that in next a few months or a year, every organization would be able to train their own model instead of renting a proprietary one from frontier labs. This is essential to those whose data are sensitive & carrying all building blocks for industry moats. Releasing Cura is our first step towards that future. actAVA will provide the capabilities for healthcare organizations to own their intelligence.
#HealthcareAI #SovereignAI #DataOwnership #OpenSourceAI #VerticalAI #HealthTech #Cura #actAVA
Newsweek Asked If Healthcare AI Hit a Reliability Wall. The Answer Is an Architecture Choice. actava.ai/news/newsweek-askeโฆ
Haolin Chen retweeted
Newsweek just featured our interview for Cura, actAVAโs one-trillion-parameter model for agentic healthcare๐
The article examines a shift we believe is beginning across healthcare: moving beyond renting intelligence from frontier labs, into transforming institutional knowledge, including workflows, policies, tools and operational expertise, into model assets they fully own.
Thank you to Jake Cole and Newsweek for taking a closer look at what we are building.
We spent over 500 dollars running Claude Fable 5 on ฯ-Bench.
But it didn't win.
Across 75 healthcare workflow tasks, Fable 5 achieved 24.0% pass@1. Claude Opus 4.6 scored 28.0% โ and cost less. Fable spent $29.20 per passed task which has weakest return in this comparison. A bigger model bill doesn't necessarily buy a better agent.
We will keep the ฯ-Bench leaderboard up to date with latest frontier models.
See more results tomorrow.
#ChiBench #AIAgents #HealthcareAI #AIEvaluation #HealthTech #ClaudeFable5
Checkout our leaderboard: actava.ai/benchmarks/leaderbโฆ
Haolin Chen retweeted
Cura 1T
Leads frontier models on 5 of 6 hardest healthcare benchmarks through a human-gated recursive self-improvement loop โ each round plans, trains, evaluates, and refines on failure modes.
I really can't believe we two people build this. But this comes true today.
The strongest healthcare LLM, custom-built for your enterprise, owned by you๐ธ
Meet @actAVAai Cura: 1T agentic model trained by recursive self-improvement for long clinical + health admin workflows
Try: actava.ai/cura
Share your use case: $20 credits + early access๐
How many researchers do you need to build a SOTA LLM? We @actAVAai spent 1 month, with 2 researchers to build Cura, the strongest healthcare LLM, custom-built for your enterprise.
Meet Cura: 1T agentic model trained by recursive self-improvement for long clinical + health admin workflows Try: actava.ai/cura
Share your use case: $20 credits + early access๐ท
We are able to do this thanks to the recursive self-evolution loops we develop. Check out our technical report at actava.ai/cura/cura-technicaโฆ
Our CHI-Bench leaderboard actava.ai/benchmarks/leaderbโฆ has another update! We tested latest models including GLM-5.2 and Sonnet 5 but they didn't show significant improvement. Besides, we received several submissions from the community including Feynman Xu from MedArise, Michael Johnson from Humana, and also MedGuard. We thank for all contributors for your work. Community submissions really pushes the boundary of this benchmark, surpassing existing frontier models.
To further encourage people to try out CHI-Bench, I am thrilled to announce that actAVA.ai is hosting a CHI-Bench competition as part of the BigData Cup in IEEE International Conference on Big Data 2026! Researchers, hackers, or anyone interested in this challenge are welcomed. We will see you in Phoenix, AZ this December!
After all it is for the greater good and safety of human-beings. Don't people agree๐คฃ?
