Founder & CEO @ Plumloom | Building LLM eval & reliability infrastructure | https://nitter.cf/t.co/fOF7ThgdPq | OSS Autoeval
San Francisco
Joined December 2008
- Tweets3.6K
- Following160
- Followers171
- Likes8
Pinned Tweet
“But our evals passed” is a painful explanation when an AI failure reaches customers.
A passing score can hide three different problems: a judge that doesn't reflect your team's standards, inconsistent judge assessments, or application behavior that varies across trials.
Plumloom Autoeval addresses these separately, helping teams build more reliable evaluation evidence and make better release decisions.
We're offering free Autoeval pilots for teams building AI applications and agents.
Bring a workflow you're already testing. We'll help you set up evaluations, review the results, and configure release gates. Hands-on onboarding, evaluation support, and AI model usage are included.
Interested? Sign up using this link:
plumloom.ai/free-pilot/
Google’s Gemini 4 Carbon ranks #1 on Artificial Analysis, yet most developers can’t access it.
Google reports 88.2% vulnerability discovery and 54.8% exploit success. Vetted cyber defenders get first access.
The model’s cybersecurity strength is also the reason for restricting it.
What do an AI Native Engineer, a Head of AI Native Operations, and an Agent Eval Engineer have in common?
One essential skill: AI evals.
Why? Because before shipping an AI product, you need to know whether it's reliable. That means completing tasks correctly, following your rules, and delivering consistent results across runs.
This requires realistic test cases, clear quality standards, calibrated AI judges, multiple trials, and release gates to catch regressions before they reach users.
This is what we've been building at Plumloom Autoeval. Over 100 companies have joined the conversation on AI reliability.
Here's a quick visual guide to reliable evals and release gates, and how to know when your AI is ready to ship:
canva.link/3wqu1duz6ff3fz6
Explore our open-source project:
github.com/AlchemyInCode/plu…
Supabase's AI Platform Engineer JD says: “The thing that grades has to be graded too.”
Plumloom Autoeval offers golden suites, calibrated judges and reliability evidence. Run evals in Claude Code or Codex via MCP; gate releases in GitHub Actions.
github.com/AlchemyInCode/plu…
Morgan Stanley analyzed 40M employee records across 750 tech companies.
Firms where revenue per engineer grew faster than engineer pay delivered average 12-month stock returns 6.2 percentage points above the median.
The data cannot isolate AI's contribution.
morganstanley.com/im/en-us/i…
A customer asked an AI agent to reserve a desk lamp. The reservation tool returned OUT_OF_STOCK, but the agent told the customer it was reserved.
Plumloom Autoeval evaluates recorded agent actions and responses against expected behavior.
Example: github.com/AlchemyInCode/plu…
Years of fuzz testing missed bugs Claude found.
Anthropic and Theorem used formal verification to prove 239 entry points across 123 open-source libraries cannot crash on inputs up to 64 MB, under defined conditions.
Mathematical proofs alongside test results could change how we ship software.
Your AI agent could be losing 16% in benchmark performance because of its search provider.
Exa 3.0 shows how swapping search providers changes results, even with the same model and harness.
Before upgrading your AI model, I’d check what the agent is actually retrieving.
Anthropic is taking on OpenAI Dots with Claude Duet. The pitch hits home: what did I forget today?
Duet monitors work apps and flags things you missed. One tester says it caught a property showing scheduled for the wrong day.
The real test? Knowing when to interrupt you.
We've just launched OpenAI Evals import in Plumloom.
If you've already run evals on OpenAI, you can now import your evals, runs, and results using your own API key.
We also analyze pass-rate trends and potentially flaky graders.
Try it at app.plumloom.ai
1/3 Anthropic tripled its workforce in a year, yet junior roles fell from 27% to 9% of technical hires.
As Claude takes on more coding and debugging, hiring managers are placing greater emphasis on autonomy and judgment.
2/3 Another interesting finding: as agents get faster, human attention becomes a constraint.
Employees spend more time supervising agents, reviewing outputs, and building tools to manage their work.
3/3 Engineers develop judgment through years of coding, debugging, and learning from mistakes. Much of that early work is moving to agents.
How do we develop experienced engineers if fewer people get the opportunity to start as junior engineers?
anthropic.com/research/emplo…
MiniMax took an unusual approach with M3.1 Flash.
An early version ran anonymously as Space Bunny, processing over 200 trillion tokens before its identity was revealed.
The official release now claims nearly 3x faster inference and 70% lower costs.
minimax.io/blog/minimax-m31f
Your coding agent can keep improving your AI app. But how do you know those improvements are reliable?
Claude Code, Codex, and Cursor can drive the iteration. We built Autoeval to measure reliability across trials and judges, and gate releases based on that evidence.
Open source. Apache-2.0.
github.com/AlchemyInCode/plu…
We've added Claude Opus 5.5 and Sonnet 5.5 to Plumloom, both supporting a 1M-token context window.
You can now evaluate these models with Autoeval, compare results across multiple trials, and run release gates directly from your terminal or agent harness.
Give it a try:
github.com/AlchemyInCode/plu…
A little creative experiment at Plumloom.
While designing our free pilot page, we explored retro pixel art inspired by early computing, with chunky pixels, limited colors, and Bayer dithering.
We liked it enough to make it part of Autoeval’s visual identity.
Sharing a few examples. Curious what you think!
SF Tech Week has more than 1,600 events, and LA Tech Week has around 800. Finding a handful that match your interests and schedule can take hours.
We built Tech Week Picks in Lovable, partly to explore what we could build with the platform. The app asks five questions about your role, goals, and interests, then recommends a personalized lineup without scheduling conflicts.
We chose TypeSafe's Jev because we needed fast, structured judgments. Jev classifies events by topic, audience, and format, then helps rerank the strongest matches. The scheduling and ranking rules stay in code.
The next challenge was tuning the recommendations for quality.
We installed Plumloom Autoeval directly in the Lovable development environment with one command:
npm i -g @plumloom/cli
From there, the Lovable coding agent could run evaluations against the actual app outputs, across multiple visitor profiles and trials, and examine relevance and factuality scores.
Autoeval uncovered unsupported explanations, weak recommendations toward the bottom of lineups, and noise in the event data that was producing false matches. The coding agent used those findings to refine the recommendation logic, and we reran the evaluations to check the changes.
This evaluation and tuning took a couple of hours. Manually testing and refining recommendations across those visitor profiles would have taken several days.
The early response has been encouraging. For SF Tech Week, 79% of visitors who started the quiz completed it.
What stood out from this experiment was how quickly we could move from building the app to evaluating and improving its recommendations, using Autoeval directly in the Lovable workflow.
Try Autoeval (open source):
github.com/AlchemyInCode/plu…
Link to the SF tech week: techweek.plumloom.ai/sf
And LA tech week:
techweek.plumloom.ai/ls
Here's the link to LA tech week:
techweek.plumloom.ai/la
And SF Tech week:
techweek.plumloom.ai/sf