Pinned Tweet
This is the most valuable free resource we've created on AI Evals 🎉 (not exaggerating!)
I organized our public materials into this guide. It allows you to find answers to your eval problems w/o searching aimlessly.
Humans: pick the row that sounds like you.
Agents: point it at the post
The material draw on 60+ hours of office hours from our Evals course, where @sh_reya and I have taught 5k engineers and PMs Evals.
We add new material often, with 15 FAQs added in the last two weeks. Recent additions are marked with a "New" badge. You can find all of these and more here:
hamel.dev/blog/posts/evals-f…
Hamel Husain retweeted
incredible time to be working in data too - _someone_ needs to build systems that turn trace data into something actually valuable to the R&D team
guess who's been looking at product data for many years at this point? if you squint your eyes, traces are really not that different from traditional data work
I believe Traces are the essential new primitive of software development and agentic evolution.
Git changed the world by helping us collaborate around the delta unit of code.
Traces are even more powerful than that. They capture the upstream intent, prompts, reasoning, skills and tools use. They are your intelligence.
You should, of course, own and make full use of your intelligence. You start by securely capturing your traces. Then, set up the recursively self-improving systems based on the learnings they contain: friction, latency, outcomes.
Excited to build this future with Tarun and team!
Evals evals evals
Tobi Lütke (@tobi), Shopify CEO, at Vercel Ship SF: "we are now all verification engineers. We are building harnesses, we are building acceptance tests, and we are building… verifiables."
When agents write the code, the work moves to the checks:
→ harnesses that make agent output repeatable
→ acceptance tests that define "done"
→ anything a machine can verify without a dev squinting at a diff
Full fireside with @rauchg in the reply 👇
Hamel Husain retweeted
Myself and @HamelHusain have been saying for the past couple years that DS is not dead, and that in fact it's an incredible time to be a data scientist. This slope over 24/25 doesn't surprise me.
I expect that even the tail here is a bit noisy due to many AI Engineers mostly being Data Scientists.
#ds4ai
Hamel Husain retweeted
Quail now supports models with hybrid architectures! Hybrid architectures are the future. With fewer attention layers, KV sizes are smaller, which improves reuse for future Quail queries.
Arnav wrote a blog post on how we adapted our cost models, extended Quail’s caching to save the recurrent state, and got a >3x speedup over vLLM: fsdatalab.github.io/blog/hyb…
Hamel Husain retweeted
BREAKING: Jensen Huang reportedly discussing a return as Nvidia CEO with allies just days after the board pushed him out.
OCR folks are having a heated debate about particular benchmarks in this thread. Much of the frustration stems from the unreasonable sway benchmarks have on vendor selection: nobody looks behind the top level score!
This invites moral hazard when publishing a benchmark, and understandably shines a light on motivations and incentives at play, even if there are good intentions.
As a consumer of models, if you are relying on benchmarks the least you should do is look at the traces from 3-5 failing tasks and try to make an informed decision yourself. Vik has a checklist as well below.
Don't make up your mind based on twitter arguments either! Look at the data. Also sometimes "benchmarks of benchmarks" are slop. You really need to look at the data yourself if you are making large purchasing decisions based on the numbers.
There are so many flawed OCR benchmarks, and more come out every week.
I gave a talk on how to evaluate them in @HamelHusain's course. Here are common issues, how to spot them, and a guide on evaluation 🧵
I am not for against any of these benchmarks or vendors btw. What I'm advocating is that people look at the data themselves
5
8
1,962
Oh and more more clarification, I'm not saying you should rely on benchmarks for decision making alone.
You need to use your own product specific evals to know if something really works rather than an abstract benchmark.
However, I know people still gonna use benchmarks for decision making so at least look at the data 🤷♂️
1
5
1,502
Hamel Husain retweeted
There are so many flawed OCR benchmarks, and more come out every week.
I gave a talk on how to evaluate them in @HamelHusain's course. Here are common issues, how to spot them, and a guide on evaluation 🧵
Hamel Husain retweeted
Here's the full guide: how to check any benchmark in 5 minutes, and how to run an eval on your own documents - datalab.to/blog/how-to-evalu… .
Hamel Husain retweeted
“Look at your data.”
The most useful advice in AI right now.
Got my signed copy of @sh_reya and @HamelHusain’s Evals for AI Engineers at @danolsen’s product meetup tonight.
Evals ftw!
Hamel Husain retweeted
Got a signed copy @HamelHusain @sh_reya 🙌
Ever wondered how the number one context layer company is the world does evals??
Well there’s a YouTube video about it now
“Basically the goat of enterprise retrieval” and “the best public example of someone doing evals” - @HamelHusain 🐐
youtu.be/hzWv-apa4k4
Hamel Husain retweeted
How it feels to be in @HamelHusain 's circle of friends:
Hamel Husain retweeted
turning a big dial taht says "index weight" on it and constantly looking back at the audience for approval like a contestant on the price is right
Love to see error bars on benchmarks
It’s very much a chart crime not to show them
Terminal-Bench 4.1 is here with updated verifiers for all tasks.
In this release, we focused on:
- resisting reward hacking
- reducing false negatives
- improving reproducibility
Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.
Leaderboard scores remain within previously reported confidence intervals.



































