Bobby Samuels retweeted
Congratulations @_jasonwei and team. What I am most excited about here is the distance between the doctor who sees you and the researcher who studies you will go to zero. One mind. Reducing the marginal cost of research to minutes means you can see patients at the bed side and test hypotheses in data much faster. So its an accelerator to both research and clinical care. Onward!
In our health workstream at Meta we did a project where we wanted to see what research papers medical experts could write using a Muse Wonder coding scaffold that could access a database of 5M electronic health records we licensed from a vendor. The goal is that with 5M patients in this database, AI could do research to answer arbitrary questions like “did people who take statin have higher rates of diabetes later in life”? The thing we were betting on here is on what happens when you put human experts, a strong coding model, and access to data all together
We worked with a small group of medical experts to help them use this AI tool and the punchline is that they all found that this setup enabled a much faster pace of health research. They generated ten manuscripts in around two weeks, with one researcher (Paddy Ssentongo) generating three. Throughout the process, they collaborated with each other to review manuscripts and write skill files to help our coding scaffold improve at doing research. The medical experts liked how quickly they could test different hypotheses and choose the best ones to continue working on
Health research is pretty unique in that it is super important to be able to interpret noisy data and have taste to know which variables are important, which was still a significant value add from human researchers. Given that medicine is one of the oldest fields in human history and it is ultimately about humans, keeping a substantial human component in this type of research is probably important. In the near future I see AI primarily as a tool to facilitate human-led research. With the right scaffold I believe health researchers can be 10x or even 100x more productive using AI tools that have appropriate access to the wealth of data that the world has collected about human health
One way to see this work is through the broader lens of AI for science. You can think of science as two parts: (1) collecting raw data about the world, and (2) synthesizing that raw data into understanding. In this project we already had existing data about the world and we aimed to accelerate part (2) using AI. Eventually, I imagine a world where we have such great AI tools that the time to synthesize raw data into understanding is driven towards zero. In that world, science will be substantially faster and most of the work will be focused on collecting the right raw data via physical experiments
We did this in a pretty short amount of time (which was the point) and I think the future of scientific research will be more faster and more exciting than ever
Bobby Samuels retweeted
AI data factories like Handshake, Scale, Surge, and Mercor have a big role to play in the advancement of AI at the frontier and in the enterprise. This is my full thesis.
At @withprotegeai, we’re seeing what many others in healthcare are seeing: There’s a huge clinical readiness gap - the gap between how health AI performs in test settings and how it performs in clinical workflows.
Leading academic institutions and health systems alike all face a similar issue: How do we best measure AI for healthcare applications when AI is already being used by doctors and patients alike?
On some healthcare tasks, AI already performs as well as the best clinicians in the world. On others, the model struggles to recall a simple lab result from a simple report.
Protege Co-Founder and Chief Scientific Officer @engyziedan wrote about this for The AI Collective in their most recent newsletter — in it, Engy explains how she and the rest of the @datalabresearch team approach this challenge from a few different perspective:
#1 - Test Design: Does the test measure the “right” answer?
#2 - Data: Does the data represent real-world scenarios?
#3 - The Referee: Who decides which model is best?
#4 - Gamification: What happens when passing the test becomes the goal?
As the industry moves beyond static test questions and toward realistic, multi-turn scenarios built on real-world healthcare data, we increasingly need an independent evaluator of how AI models are performing.
Check out Engy’s full write-up with The AI Collective here: newsletter.aicollective.com/…
Bobby Samuels retweeted
NEW: AI is on track to write nearly 1 in 3 medical SOAP notes.
SOAP notes are the standard way clinicians document patient visits, covering history, findings, assessment, and plan.
Clinical documentation remains one of healthcare AI’s fastest-growing use cases.
Bobby Samuels retweeted
How do we know a model is actually good at medicine, rather than just good at the test?
Protege/a16z argue this is one of the biggest bottlenecks in healthcare AI (we tend to agree).
A few things stood out:
1. Medicine often has no clean “right" answer.
> Across 15 surgeries with multiple treatment options, physician identity explained anywhere from 7% to 77% of the variation in which option was chosen.
2. Benchmarks see less context than a doctor.
> The median real patient record in Protege’s data contains ~8,500 tokens. Five of six public healthcare AI benchmarks provide less context than that, and several use fewer than 200 tokens.
3. Small choices can change the winner.
> Reordering the same 19 diagnostic answer choices caused models to change answers. Changing one sentence in a prompt erased performance gaps and even flipped which model performed best.
4. Better benchmark scores =/ better health.
> The measures that ultimately matter are things like mortality, QALYs, clinician burnout, medical debt, denials, and whether AI actually improves outcomes at the same or lower cost. That distinction is getting more important as AI use grows across both patients and clinicians.
Spirit Airlines made headlines this week after it sold its company data to Google for $10 million. What makes this data valuable is the value of the people’s real output from actual work environments.
What we shouldn’t miss - and what I shared with @JamesJNani of @BLaw in their analysis of the data purchase - is the need for trust and maintaining privacy in what the data represents.
The market for company data is heating up. The core tenets of data privacy, such as de-identification, consumer privacy, and employee protections, all need to be carefully preserved here.
This is something we’re thinking about, especially since many of our backgrounds come from sensitive industries like healthcare where privacy is so important. Here too, the spirit of the law matters just as much as the letter. That’s the standard we’re building toward at @withprotegeai: real-world data, handled with care, and used to make AI effective and people more productive.
Check out the full article here: news.bloomberglaw.com/bankru…
Thanks to James and Bloomberg for covering this and speaking with me.
Bobby Samuels retweeted
But who grades the AI your doctor uses? How do we judge which AI is the most reliable in the doctor’s office?
That's what Protege Co-Founder and Chief Scientific Officer @engyziedan asks in her latest write-up.
Healthcare AI is being deployed - now. It's not slowing down anytime soon. Yet outdated healthcare benchmarks are what's still being used to determine how healthcare AI is being deployed in hospitals and healthcare systems today.
And unlike drug development or journal reviews, AI evolves on timescales that our existing institutions were never designed to handle.
Engy puts it this way:
"We need an independent arbiter of truth for evaluating AI models for healthcare that has deep expertise in all angles of healthcare AI deployment."
"The AI model training and settings. The health data exposure and statistical methodology applied to healthcare. The actual in-office medical practice and nuances of patient care."
Read her full piece here: linkedin.com/feed/update/urn…
@datalabresearch @bobbysamuels
Bobby Samuels retweeted
Great to see the MedScribe benchmark we designed with @ValsAI and Harvard used to judge current SOTA models.
Today, Protege holds over 2,000,000 visits worth of medical audio, constituting over 200,000 total audio hours, along with 1,000,000,000's of medical notes that correspond to hundreds of millions of SOAP notes.
For this benchmark and from this corpus, we curated medical records and SOAP notes and test the model's ability to generate accurate SOAP notes from a medical transcript.
Most importantly.... none of the benchmark cases have ever appeared in any foundation model training!
Getting this right is huge for any medical AI application. A SOAP (Subjective, Objective, Assessment, Plan) note is a key endpoint in care delivery frameworks, where a doctor writes the care plan and assessment for an office visit.
We'll continue to see models improve on real-world tasks like these so long as they have access to the real-world data that accurately reflects real-world conditions.
full article: forbes.com/sites/cortneyhard…
@withprotegeai was just featured in @Forbes in a piece called “The Future of AI Training Data Is Human. The Question is How” by @cortneyharding. I wanted to take a minute to highlight how this reflects our core company values: that trust and quality are even more important for this next era of AI development.
We have a strong stance on what quality data looks like for AI training, evaluation, and application. As the piece notes, “AI is only as good as the metadata that powers it.” We’ve seen how the best data is data that reflects the real world - its quirks, nuances, and human behavior alike.
Every leading AI lab has come to this realization, and that’s how we’ve grown so quickly across multiple domains. At Protege, we value data that is truly representative of ground truth human behavior, whether that’s in video, healthcare, audio, or 3D and world models.
Our partnership with @vlge__ reflects this philosophy. VLGE’s data captures how people actually move, explore, hesitate, and decide while pursuing their own goals inside spatial worlds. As Grant Murphy-Herndon, our GM of Spatial and Physical Intelligence, says in the article: most datasets are "only as good as however you ask people to do it... versus a data set like VLGE where people are kind of going and pursuing their own goals."
That makes the data more informative of real behavior, and inherently less biased. And the data has been collected in a compliant way. This is a key part of how Protege operates: we believe that ethics and data quality point in the same direction.
Ultimately, that's the part I care about most. The next phase of AI will be built on real, lived human experience. Whether that data gets sourced and governed responsibly — this cannot get lost in the shuffle. This is true whether you’re a media rights holder, healthcare system, or spatial data provider.
Thanks to Forbes and Cortney Harding for the thoughtful piece. Worth a read (link in thread)
Bobby Samuels retweeted
Protege co-founder and CEO Bobby Samuels sat down with Pablo Srugo on The Product Market Fit Show to talk about building Protege from scratch — $1M GMV in year one to $30M in year two, raising a $10M seed with no revenue, and what founder-led enterprise sales actually looks like in the AI era.
Three of our favorite moments:
🤝 Trust is Irreplaceable [9:10]: AI and automation can make the world feel fast and the real constraint is time. Showing up on a personal level and delivering with integrity is how you earn the trust and keep it.
💬 The "On Texting Terms" Litmus Test: Bobby's personal test for whether a relationship is real and could go somewhere.
🌊 From a Trickle to a Flood [3:01]: how Product-Market-Fit actually felt - not one big deal, but multiple deals of different sizes that showed up where the thesis was resonating.
Listen to the whole show:
pmf.show/he-raised-a-10m-see…
Today, I’m excited to announce our newest vertical: Spatial & Physical Intelligence.
We have been investing in an entirely new category focused on supporting world models and robotics labs. From working closely with the labs in both domains we've noticed a consistent pattern: they end up needing the same underlying training data.
Our thesis is anchored to four fundamental data types that are important for this development stage:
1) Ego- and Exo-centric Video: First and third-person footage of humans performing real-world tasks and vehicle-based captures of dynamic environments. Depth data, LiDAR, hand tracking, descriptive annotations, overlapping camera views, and time-synced data all increase spatial understanding.
2) Motion Capture: Mapping the "physics of the mundane," moving beyond entertainment and gaming to capture tactile object manipulation, locomotion, and human to human interactions.
3) Video Gameplay Data: Studio-grade simulated environments paired with precise player telemetry.
4) 3D Assets: 3D scans of objects & scenes including raw input files before construction.
The Core Challenge: data is siloed.
High quality training data is trapped in fragmented datasets instead of accessible in a unified data layer. Robotics and world models are developing more quickly than ever – and the data layer should move just as fast. We’re here to help build high-quality, content-rich datasets that create the data supply chain for Spatial & Physical Intelligence data for AI.
This is a key area where we're scaling at Protege – we’re actively looking for builders and feedback.
Come build with us and tell us what we’re missing!
(see 🧵 below for more details)
