Hamel Husain (@HamelHusain) header
Hamel Husain (@HamelHusain) profile picture

Evals Evals Evals @OpenAI About Me: https://nitter.cf/t.co/P6WyeKk13C

Looking at the data
Joined September 2012
Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)Photo by Hamel Husain (@HamelHusain)
This is the most valuable free resource we've created on AI Evals 🎉 (not exaggerating!) I organized our public materials into this guide. It allows you to find answers to your eval problems w/o searching aimlessly. Humans: pick the row that sounds like you. Agents: point it at the post The material draw on 60+ hours of office hours from our Evals course, where @sh_reya and I have taught 5k engineers and PMs Evals. We add new material often, with 15 FAQs added in the last two weeks. Recent additions are marked with a "New" badge. You can find all of these and more here: hamel.dev/blog/posts/evals-f…
75
133
17
1,246
140,666
Hamel Husain retweeted
1
2
8
1,238
Hamel Husain retweeted
The next generation of models needs a data stack built for AI. We just raised a $57.5M Series B to help researchers turn petabytes of video, images, and audio into better models, faster. Read the announcement: lancedb.com/blog/lancedb-ser…
26
31
27
181
38,931
Hamel Husain retweeted
incredible time to be working in data too - _someone_ needs to build systems that turn trace data into something actually valuable to the R&D team guess who's been looking at product data for many years at this point? if you squint your eyes, traces are really not that different from traditional data work
I believe Traces are the essential new primitive of software development and agentic evolution. Git changed the world by helping us collaborate around the delta unit of code. Traces are even more powerful than that. They capture the upstream intent, prompts, reasoning, skills and tools use. They are your intelligence. You should, of course, own and make full use of your intelligence. You start by securely capturing your traces. Then, set up the recursively self-improving systems based on the learnings they contain: friction, latency, outcomes. Excited to build this future with Tarun and team!
6
3
33
3,232
Evals evals evals
Tobi Lütke (@tobi), Shopify CEO, at Vercel Ship SF: "we are now all verification engineers. We are building harnesses, we are building acceptance tests, and we are building… verifiables." When agents write the code, the work moves to the checks: → harnesses that make agent output repeatable → acceptance tests that define "done" → anything a machine can verify without a dev squinting at a diff Full fireside with @rauchg in the reply 👇
38
29
2
497
26,753
Myself and @HamelHusain have been saying for the past couple years that DS is not dead, and that in fact it's an incredible time to be a data scientist. This slope over 24/25 doesn't surprise me. I expect that even the tail here is a bit noisy due to many AI Engineers mostly being Data Scientists. #ds4ai
Data science vs. UXR over the last 4 years Data science openings more than doubled off the 2023 lows, while user research kept falling and is now down 85% from 2022, a record low.
6
1
11
1,976
Hamel Husain retweeted
Quail now supports models with hybrid architectures! Hybrid architectures are the future. With fewer attention layers, KV sizes are smaller, which improves reuse for future Quail queries. Arnav wrote a blog post on how we adapted our cost models, extended Quail’s caching to save the recurrent state, and got a >3x speedup over vLLM: fsdatalab.github.io/blog/hyb…
12
9
1
34
4,306
Hamel Husain retweeted
BREAKING: Jensen Huang reportedly discussing a return as Nvidia CEO with allies just days after the board pushed him out.
575
1,280
398
19,459
1,343,171
OCR folks are having a heated debate about particular benchmarks in this thread. Much of the frustration stems from the unreasonable sway benchmarks have on vendor selection: nobody looks behind the top level score! This invites moral hazard when publishing a benchmark, and understandably shines a light on motivations and incentives at play, even if there are good intentions. As a consumer of models, if you are relying on benchmarks the least you should do is look at the traces from 3-5 failing tasks and try to make an informed decision yourself. Vik has a checklist as well below. Don't make up your mind based on twitter arguments either! Look at the data. Also sometimes "benchmarks of benchmarks" are slop. You really need to look at the data yourself if you are making large purchasing decisions based on the numbers.
There are so many flawed OCR benchmarks, and more come out every week. I gave a talk on how to evaluate them in @HamelHusain's course. Here are common issues, how to spot them, and a guide on evaluation 🧵
66
2
3
74
6,588
I am not for against any of these benchmarks or vendors btw. What I'm advocating is that people look at the data themselves
5
8
1,962
Oh and more more clarification, I'm not saying you should rely on benchmarks for decision making alone. You need to use your own product specific evals to know if something really works rather than an abstract benchmark. However, I know people still gonna use benchmarks for decision making so at least look at the data 🤷‍♂️
1
5
1,502
Hamel Husain retweeted
There are so many flawed OCR benchmarks, and more come out every week. I gave a talk on how to evaluate them in @HamelHusain's course. Here are common issues, how to spot them, and a guide on evaluation 🧵
12
13
5
112
15,199
Hamel Husain retweeted
Here's the full guide: how to check any benchmark in 5 minutes, and how to run an eval on your own documents - datalab.to/blog/how-to-evalu… .
2
1
1
18
2,508
Hamel Husain retweeted
“Look at your data.” The most useful advice in AI right now. Got my signed copy of @sh_reya and @HamelHusain’s Evals for AI Engineers at @danolsen’s product meetup tonight. Evals ftw!
40
40
3
600
24,739
Got a signed copy @HamelHusain @sh_reya 🙌
6
3
1
31
3,999
Hamel Husain retweeted
Ever wondered how the number one context layer company is the world does evals?? Well there’s a YouTube video about it now “Basically the goat of enterprise retrieval” and “the best public example of someone doing evals” - @HamelHusain 🐐 youtu.be/hzWv-apa4k4
3
4
1
89
8,207
evals
c̵o̵n̵t̵e̵x̵t̵ ̵e̵n̵g̵i̵n̵e̵e̵r̵i̵n̵g̵ ̵h̵a̵r̵n̵e̵s̵s̵ ̵e̵n̵g̵i̵n̵e̵e̵r̵i̵n̵g̵ goal engineering
12
12
8
191
11,729
How it feels to be in @HamelHusain 's circle of friends:
4
1
34
2,330
Hamel Husain retweeted
turning a big dial taht says "index weight" on it and constantly looking back at the audience for approval like a contestant on the price is right
The Artificial Analysis Intelligence Index is currently made up of 9 benchmarks with arbitrary weights. I wanted to know how these weights varied over time. They vary quite a bit!
10
1
84
7,823
I took a stab at visualizing the transitions of weights this year in different categories:
The Artificial Analysis Intelligence Index is currently made up of 9 benchmarks with arbitrary weights. I wanted to know how these weights varied over time. They vary quite a bit!
2
5
17
2,492
Love to see error bars on benchmarks It’s very much a chart crime not to show them
Terminal-Bench 4.1 is here with updated verifiers for all tasks. In this release, we focused on: - resisting reward hacking - reducing false negatives - improving reproducibility Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping. Leaderboard scores remain within previously reported confidence intervals.
68
8
3
144
8,915