Professor @northeastern | Chief Economist @micro1_ai | Social Scientist @BKCHarvard
Boston, Geneva, Dubai
Joined August 2012
- Tweets67.8K
- Following11.7K
- Followers44.5K
- Likes138K
Mark Esposito, PhD retweeted
We mourn the passing of Patrick Stewart, who inspired countless individuals at NASA (and beyond) to boldly go where no one has gone before.
ALT Image description: A packed, sparkling starfield captured by our Hubble telescope — with strange new worlds potentially awaiting. Credit: ESA/Hubble and NASA, R. Cohen
Mark Esposito, PhD retweeted
Canada and the EU are working towards the deepest possible partnership — short of full membership — with Canada becoming the European Union’s first Associate Member.
Today in Montréal, we took a major step forward with more than twenty new measures that will create greater affordability, opportunity, and sovereignty on both sides of the Atlantic.
pm.gc.ca/en/news/news-releas…
Mark Esposito, PhD retweeted
We gave 10 frontier AI models control of a surgical robot in a simulated operating room.
Across 800 trials, only two models completed the task at all: Gemini 4 Argon nine times, and GPT-6 Astra twice.
Models claimed success 213 times. Every claim was wrong. In 94 of them, the robot never touched the object.
Introducing Dual-Arm Sim, our new robotics benchmark evaluating how well frontier models can control embodiments in simulated environments. More info in thread around what we tested and found.
At our recent @micro1_ai Forum, I sat down with Shaudi Mahdavi (micro1) and @stribwal (Head of Product at Rogo) to discuss the launch of the Cortex retrieval benchmark.
So what did I learn that continues to tickle my thinking …
Most evaluation of financial AI asks whether the model got the answer right. Far fewer ask whether it was reasoning from the right document in the first place.
The benchmark makes the gap visible. Authoritative sources appear in the top ten results 92 to 99 percent of the time. They rank first only 42 to 69 percent of the time. The system finds the right document. It doesn't know it's the right one.
In finance, that matters enormously. Last year's annual report and this year's look nearly identical to a search engine. To an analyst, they are not interchangeable at all. Provenance isn't a formality; it's part of the claim. And when trust breaks, professionals stop bringing the hard problems to the tool.
The conversation also pushed me on a question I care about deeply: if AI absorbs the tedious work through which junior analysts learned what good looks like, how do we deliberately rebuild that learning?
My full essay is on the micro1 blog:
micro1.ai/blog/retrieval-gap…
Mission report shows how models will continue to extend their criticality to governments functions and decisions and the reliability of a data training company is not only in the accuracy of the answer but the contextual understanding of the reality in which the answer will be deployed. Another epic piece of work
Explore the full results and methodology below: micro1.ai/benchmark/mission-…
Mark Esposito, PhD retweeted
Government reporting demands more than factual accuracy.
Introducing MissionReport, our latest benchmark testing whether frontier models can turn working files into reports commanders and senior leaders can rely on.
AI models can get the facts right and still miss what matters for sensitive U.S. Government use cases. We tested nine frontier models on government mission reporting: only 3 of 540 reports met every criteria.
Introducing MissionReport, a benchmark built by @micro1_ai with current and former U.S. government and military professionals to test whether AI can identify, prioritize, and communicate what commanders and senior leaders need to know for mission critical matters.
More info in thread 🧵
Mark Esposito, PhD retweeted
Great work by our gov research team. We analyzed frontier models' capabilities to streamline how commanders and senior leaders act on mission critical matters. There's promising room for improvement 🇺🇸
AI models can get the facts right and still miss what matters for sensitive U.S. Government use cases. We tested nine frontier models on government mission reporting: only 3 of 540 reports met every criteria.
Introducing MissionReport, a benchmark built by @micro1_ai with current and former U.S. government and military professionals to test whether AI can identify, prioritize, and communicate what commanders and senior leaders need to know for mission critical matters.
More info in thread 🧵
Mark Esposito, PhD retweeted
super important work 🦅
AI models can get the facts right and still miss what matters for sensitive U.S. Government use cases. We tested nine frontier models on government mission reporting: only 3 of 540 reports met every criteria.
Introducing MissionReport, a benchmark built by @micro1_ai with current and former U.S. government and military professionals to test whether AI can identify, prioritize, and communicate what commanders and senior leaders need to know for mission critical matters.
More info in thread 🧵
Mark Esposito, PhD retweeted
AI models can get the facts right and still miss what matters for sensitive U.S. Government use cases. We tested nine frontier models on government mission reporting: only 3 of 540 reports met every criteria.
Introducing MissionReport, a benchmark built by @micro1_ai with current and former U.S. government and military professionals to test whether AI can identify, prioritize, and communicate what commanders and senior leaders need to know for mission critical matters.
More info in thread 🧵
The social responsibility of a firm like @micro1_ai is in detecting where models fail and how benchmarks help mitigate that failure by embedding a 100% human judgement in the training of the benchmark itself. This is not just innovative and frontier but ethically mandated by the responsibility we have to train models used by millions. Another epic moment.
Today we’re announcing the micro1 Medical Index: a suite of medical benchmarks built and graded by working doctors, biostatisticians, regulatory experts, and psychiatrists.
It brings all of our medical research into one place, with a combined score across models and a living leaderboard that grows as we publish new benchmarks. We’re starting with three, each built around a fundamental question:
- RealmPathology Benchmark: can a model reason over a real pathology report, staying exact on what it says, within its diagnostic limits, and without escalating beyond what the evidence supports?
- PharmaBench: can a model work through the regulatory and clinical documentation behind an FDA drug program with the precision a clinical development team would expect?
- WellnessBench: when a young person brings a sensitive health question, can a model stay safe without shutting the conversation down, and hold that line under pushback?
Together, these three benchmarks cover a meaningful range of medical work, from diagnostic reasoning to pharmaceutical research and patient safety. While they don't yet capture the full scope of the entire medical industry, they establish a strong starting point that will become more comprehensive as we expand the index.
Huge shoutout to every expert who built and graded these tasks. Full reports linked in the comments below!
Mark Esposito, PhD retweeted
Today we’re announcing the micro1 Medical Index: a suite of medical benchmarks built and graded by working doctors, biostatisticians, regulatory experts, and psychiatrists.
It brings all of our medical research into one place, with a combined score across models and a living leaderboard that grows as we publish new benchmarks. We’re starting with three, each built around a fundamental question:
- RealmPathology Benchmark: can a model reason over a real pathology report, staying exact on what it says, within its diagnostic limits, and without escalating beyond what the evidence supports?
- PharmaBench: can a model work through the regulatory and clinical documentation behind an FDA drug program with the precision a clinical development team would expect?
- WellnessBench: when a young person brings a sensitive health question, can a model stay safe without shutting the conversation down, and hold that line under pushback?
Together, these three benchmarks cover a meaningful range of medical work, from diagnostic reasoning to pharmaceutical research and patient safety. While they don't yet capture the full scope of the entire medical industry, they establish a strong starting point that will become more comprehensive as we expand the index.
Huge shoutout to every expert who built and graded these tasks. Full reports linked in the comments below!
Mark Esposito, PhD retweeted
very excited for the launch of LegalReasonBench, from the braintrust of our legal data lab at micro1.
we set out to fill an industry gap: a multi-step, long-horizon legal reasoning benchmark. every task is entirely attorney-authored with evolving matters & requests, as opposed to one-shot evaluation.
this benchmark is the closest representation of how legal professionals collaborate with AI agents to complete work; iterating on documents, debating concepts, and researching caselaw. some very interesting failure modes were revealed and I am looking forward to seeing future models overcome them!!
AI models can be as convincing as a lawyer, but can they complete the complex multi-step reasoning a real legal matter requires?
Today we're excited to introduce LegalReasonBench, a new benchmark to explore this question and test models on realistic, long-horizon legal matters authored by practicing attorneys.
More info in thread 🧵
Mark Esposito, PhD retweeted
nothing to add onto this… our approach to robotics data perfectly laid out here
Why is robotics still so far behind AI? And what will it actually take to get to general-purpose robots?
Ali Ansari (@aliansarinik), founder of @micro1_ai, which just passed $1B ARR selling data to AI labs, goes deep on the problem and explains what it actually takes to collect and build datasets for robotics. Worth the 10 minutes:
• Robotics lags language models because there is no internet-scale pre-training dataset for the physical world. Ali argues it is almost entirely a data problem.
• The real world has a much larger, three-dimensional action space, so the data investment is orders of magnitude greater.
• Self-driving works because driving is a relatively structured task, with physical lines and guidelines, and we’ve spent a decade collecting data for it.
• The main approach today: egocentric data, where humans do diverse everyday tasks on camera, then annotate and structure it.
• Overtime, Robots could potentially collect and generate their own product data, but models have no innate sense of consequence, so humans will always be in the loop and supply the reward signal.
• Are edge cases really more valuable? To automate the economy, models need the full distribution of work, and almost all of it is ordinary tasks, not elite expertise.
Mark Esposito, PhD retweeted
AI models can be as convincing as a lawyer, but can they complete the complex multi-step reasoning a real legal matter requires?
Today we're excited to introduce LegalReasonBench, a new benchmark to explore this question and test models on realistic, long-horizon legal matters authored by practicing attorneys.
More info in thread 🧵
Mark Esposito, PhD retweeted
As agents evolve, evaluations need to keep up. Knowing whether an agent is improving requires consistent measurement and evidence that progress carries through to real-world performance.
Our Economic Research Unit explores this in its first paper. Learn more:
Most companies can’t determine whether their AI agents are genuinely improving and our findings reveal why. The Economics Research Unit (ERU) at @micro1_ai , which I am privileged to lead, has released its inaugural paper, co-authored with @liuzhang11 : “Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program.”
Key Insights from 2,611 Real Production Interviews
By tracking a live AI interviewing agent in production, we uncovered two major dynamics:
1) Your evaluation metrics may reflect reviewer bias rather than AI performance. The identity of the reviewer was a stronger predictor of scores than the timing of the interview. In fact, a single overly strict evaluator masked a genuinely improving agent, making it appear stagnant. Once adjusted, reliability metrics increased by half a standard deviation.
2) AI evaluations go stale after roughly five weeks. Performance drifted even without code updates or new deployments. Without ongoing testing, after five weeks you lose the ability to detect a change as significant as your largest release. A one-time evaluation tells you about the agent you had, not the agent you are running.
Agents drift, so evaluation cannot be a launch-day event. It has to run continuously for as long as the agent is live. That is the principle behind Cortex, micro1's reliability layer for agents in production, and this paper shows why it matters.
Measuring AI agents is not a mere compliance checkbox; it serves as core economic infrastructure. This research marks just the beginning for the ERU, with further studies on the horizon.
micro1.ai/research/are-ai-ag…
