
Co-founder @Calibre_Labs | Applied AI research & consulting | Agents, AI Evals | prev EVP @amplitude_HQ | VC @khoslaventures @sequoia | @stanford @iitbombay
Joined May 2009
- Tweets6.8K
- Following382
- Followers9.5K
- Likes7.9K
Loved the recent article on evals from @HamelHusain
and @sh_reya in the @lennysan newsletter. One question we get everytime with PMs is what evals to write for long horizon agents that do a broad swath of work (what H/S call error discovery, similar to product discovery).
Below is a great framework for how you can organize these discoveries as a product team:
1️⃣ Outcome - were the results great
2️⃣ Trajectory - was the process followed correct
3️⃣ Experience - did it feel right
4️⃣ Governance - did it follow critical rules
You start with your product pov, but the only way to hone focus is to look at traces (trace analysis). Here's an example of an annotated trace for a local shopping assistant called Corner and how often the trace reveals flaws before they become obvious to the customer.
claude.ai/artifact/PBvMFLueC…
easily #1 issue amongst teams we work with - no one is reading.. incredible amounts of alpha in being detail oriented rn!
I've spent the last 1.5 years working with the best AI startups globally. This distills my learnings on prompts. Engineers & builders, I hope it's useful 🤝
2
3
861
Increasingly hearing (and feeling) the word “addiction” being used by ai-pilled developers when describing our daily work.. mistaking output for progress and impact the way you mistake euphoria for happiness.
I want my moments of friction to function as thinking breaks - forcing me to question whether something should exist rather than shipping. Building /= Shipping. The latter is a means to the former end.
H/t @mitsuhiko
272











