Staff Engineer @ Braintrust. All posts are my own. Formerly at Zapier (Sr. Engr), omcode (founder), Code RGV (founder), Google, UCLA (M.S. EE), UTSA (B.S. EE)
USA
Joined June 2008
- Tweets2.3K
- Following673
- Followers550
- Likes811
In Zed, I've disable LSPs (Language Servers). I wanted more memory back and less CPU pressure. Bye bye background esbuild, pylint etc..
I still think there's more we need to do with tracing back the cause of a security prompt like Okta, 1password, and Apple security prompts.
As a reviewer, I need to know who, when, and where something tripped to cause an interruption.
Olmo Maldonado retweeted
Bubble is an AI app builder that creates full-stack web and mobile apps. Their Agent turns a user's prompt into elements that bring this intent to life.
The Bubble AI team defines all of its metrics as scorers against eval datasets in Braintrust, then scores the visual quality of the outputs, while also tracking latency and inference costs.
Read more → braintrustdata.link/customer…
Reviewing code (and slop) by working from the API (or UI) down to the unit of work is better than going file by file. If we can't agree on the interface then why complain about anything else?
AI agents shouldn't have access to every tool and MCP for every task.
That means extra context, overhead, and security exposure.
I built codex-wrapper to load only the Codex profiles a task needs.
Less noise. More control.
github.com/ibolmo/codex-wrap…
~1990s wait for java/c++ compilation to finish
~2000s wait for php/python to respond
~2010s wait for react/angular to render/sync up
~2020s wait for rust compilation to finish
full circle moment
~2025s-present wait for agents to finish thinking
~2030+ ?
Random: English majors may need to inject grammatical errors to avoid being accused of writing slop.
Related: I bet English majors are highly sought after human reviewers by foundational models. Getting the "taste" just right is the next frontier.
There's still time!
Come by to The Midway to attend:
Builder Odyssey: AI Infra, Multimodal & FDE | Founder Demos & AI Film Showcase · Luma luma.com/seamate-tey7
I'll be sharing insights and lessons learned by building products for enterprise ai builders.
Olmo Maldonado retweeted
Your tokens have plans this Sunday 🤖
We’re bringing Builder Odyssey to SF on Oct 18—run, build, deploy, repeat.
@H0wie_Xu @jfischoff @ibolmo @jove @zylberstein
Grab your spot on @partiful 👇
partiful.com/e/0PAdTfFcp3hgB…
PSA: If a random security/permission prompts you to accept/deny, and you've got agents working. You *must* count to 10, and then check on the rascals.
Days of auto accept (because you're the one tripping the detection) are long gone.
Olmo Maldonado retweeted
GPT-6.1 Sol promises similar performance to last week's GPT-6 release at one-fifth the token cost of Astra. Does it deliver?
We re-ran our eval on how well models can guide users through multi-step problems. GPT-6.1 Sol reached 78.6% problem-solving accuracy at token prices far cheaper than Astra, Opus 5.5, or Fable 5.1.
The most notable improvement was how the new model guided users through teaching interactions. Our "teaching quality metric" for GPT-6.1 Sol rose from 64.1% to 77.6%, compared to Astra's 82.2%.
The results indicate GPT-6.1 Sol is able to interact with users in the ways they specify, leading to better pedagogy performance.
Read more → braintrustdata.link/gpt-6-1-…
New tactic:
One agent that only adds test (red team) to prove something is working or not working. Not allowed to edit non test code.
Separate agent that only changes code (blue team) but is not allowed to edit tests.
You're part of the red team.
Friendly reminder:
Send Braintrust all of your data (text or binary) with zero/zip/nada of setup (no schemas).
Brainstore will take it from there.
You (or your agents) can retrieve with (un)structured calls virtually immediately. Now even faster.
We built Brainstore for the amount of data agents generate. Now agents are helping teams search through traces and run far more queries than a person would.
Nitro is Brainstore's new asynchronous query execution engine. It separates waiting for storage from processing the data it returns, bypassing the limits of traditional object storage.
We ran Nitro on more than 1 million real queries and found that full-text searches were over 2x faster. It's already on for Braintrust SaaS and BYOC customers.
Read more → braintrustdata.link/nitro-bl…
Olmo Maldonado retweeted
We built Brainstore for the amount of data agents generate. Now agents are helping teams search through traces and run far more queries than a person would.
Nitro is Brainstore's new asynchronous query execution engine. It separates waiting for storage from processing the data it returns, bypassing the limits of traditional object storage.
We ran Nitro on more than 1 million real queries and found that full-text searches were over 2x faster. It's already on for Braintrust SaaS and BYOC customers.
Read more → braintrustdata.link/nitro-bl…
Olmo Maldonado retweeted
We tested where Jev holds up as a judge.
Jev was fast, cheap, and highly competitive for judging groundedness. But it lagged behind models with reasoning for math and code domains.
Use it for certain eval tasks, but don't throw out your LLM-as-a-judge just yet.
Read more → braintrustdata.link/jev-vs-g…
Jev from @typesafeai can replace your LLM-as-a-judge for scoring agent responses.
It returns a choice or numeric result with information about uncertainty, so you don't need to spend time and resources prompting a general-purpose model into an LLM judge.
Use Jev as a judge scorer in Braintrust and review its selected answer, confidence, and probabilities alongside the score. Trace Jev calls from your own application with the JavaScript or Python SDK.
Read more → braintrustdata.link/jev-eval
Olmo Maldonado retweeted
Jev from @typesafeai is now available as an evaluation model in Braintrust.
If you have an existing scorer, just change the model in the dropdown to Jev and reduce your scoring cost by 400x.
We're very excited about this and are doing more of our own evals in the coming days.
Jev from @typesafeai can replace your LLM-as-a-judge for scoring agent responses.
It returns a choice or numeric result with information about uncertainty, so you don't need to spend time and resources prompting a general-purpose model into an LLM judge.
Use Jev as a judge scorer in Braintrust and review its selected answer, confidence, and probabilities alongside the score. Trace Jev calls from your own application with the JavaScript or Python SDK.
Read more → braintrustdata.link/jev-eval
I've heard agents aren't good at product design. Try this:
Assign a few agents to create criteria for X.
Have a panel of agents score against said criteria.
I'm expecting a score of 40%. Adjust criteria until the panel agrees with my score.
Improve X until the score is 80%+.