stripe's coding agent eval averaged 78.1%. only 29.4% of runs passed every required check.
same runs, same rubric. the gap comes from how you add up the score.
stripe just published the method: 69 integration scenarios, 14 model and harness pairs, a second agent role-playing the developer, and a rubric split into MUST and SHOULD criteria.
what i'd copy from it if you build evals or rl tasks:
> grade each criterion on its own
> stripe does this so one criterion doesn't bias the judge on the others. one big "rate this 1-10" prompt hides which part failed
> separate must-haves from nice-to-haves
> a checkout that looks clean but skips a required step is a failed integration, not an 80%
> report the strict pass rate next to the average
> partial credit says the agent got close. a customer only cares whether the code works
> play the user as a role
> real developers start with a goal, not a spec. stripe's roleplayer steers toward the product outcome without teaching the agent how to pass
> don't train on the average
> reward partial credit and the model learns to hit the cheap criteria. the expensive failures sit in the ones it skips
honest caveat: a must list is only as good as whoever wrote it.
an 80% agent that misses a required step 7 times out of 10 is a 30% agent.
8
9
330
merged changes are the easy half.
the review rounds before the merge hold the real eval signal: the diff a reviewer rejected, the comment that says why, the second try that passed.
a task built only from the final commit tests whether the agent can reach the answer. it doesn't test whether it avoids the mistakes your reviewers already catch.
Interesting paper from Meta showing how to build coding evals from your own codebase.
(bookmark it)
Good benchmarks are hard to create, but using this method with agents looks promising.
They propose Benchmark Factory, an agent loop that turns merged internal code changes into benchmarks. It writes the query that selects tasks, runs models of known relative strength, and repairs the benchmark until it passes a set of quality checks. These checks need no per-task human labels.
To make sure the checks catch bad benchmarks, the authors fed them bad benchmarks on purpose. A benchmark that was far too easy got rejected. So did benchmarks with five different kinds of broken grading. When they scrambled which model was which, a bad benchmark slipped through at most 1.4% of the time.
On configurations never used to build them, the benchmarks ranked Claude Opus 4.7 at high effort above low effort (60.6% vs 54.4%) and GPT-5.5 above GPT-5.4 (65.8% vs 60.5%). 9 of 10 per-team comparisons agreed.
arxiv.org/abs/2611.18890
Chat with Paper: academy.dair.ai/papers/bench…
1
7
376
most rl training data goes stale faster than teams expect.
you record an agent using a tool, build a task around it, and six months later the api changed, a package updated, or the ui moved. the trajectory still looks clean on disk. the environment no longer runs.
what keeps rl data usable a year later:
> pin the exact repo commit, container, and dependency tree
> mock or snapshot external calls instead of hitting live endpoints
> separate the task state from the grader so you can re-grade when the rubric tightens
>> if you can't replay the starting state from scratch, you don't have an environment. you have a log.
a lot of expensive agent data is going to rot before the next model run even starts.
anthropic paid its data and eval vendors $757M more than a year earlier.
that's from the ipo prospectus: the first nine months of 2026 against the same months of 2025. the stock starts trading today.
the same line rose $165M in 2024 and another $165M in 2025.
what the filing doesn't say:
> the total bill. these are increases only
> which vendors got paid
> how much went to data work and how much to eval services
so nobody can tell who won from this. the direction is clear, though. i read it as a frontier lab paying more every year for outside people who check and grade model work.
the filing also says anthropic trains on "datasets licensed or commissioned from third parties." commissioned means built to order.
9
28
2,429
a coding agent's reject log is the raw material for an rl environment. on its own it's missing most of what makes one.
what the log usually holds:
> the prompt and the files the agent could see
> the diff it proposed
> what the developer did with it: accepted, edited, or threw it out
> sometimes the edit they made instead
what an environment needs on top:
> the repo frozen at the commit before the agent touched it, with dependencies pinned so it still builds next year
> an instruction that includes what the developer asked for later in the session, since the first prompt rarely says it all
> a test that fails on the rejected diff and passes on the developer's fix
> a reference solution kept outside the image the agent runs in
>> if no test can tell the two diffs apart, the trace stays a preference pair. i'd guess most rejections end up there
the test is where the work goes. developers reject diffs over naming, over style, or because something broke three files away. only a change in behavior gives you a test that can carry the reward.
so a big log yields a small number of real tasks. that's fine. each one is a spot where a model failed on code a developer actually needed.
4
6
519
the first page of a company data deal should be about use. size can wait.
> train, evaluate, or both
> one lab, or licensed on to others
> what happens to copies if the deal ends
> who reviews each export before it leaves
terabytes go in an appendix.
5
341
a final answer shows a model what good looks like. a correction shows it what it got wrong, and why.
the open web is full of the first kind. the second kind mostly sits inside products where experts work every day.
that's what i'm building toward.
Labs have trained on most of the public internet. The next signal they need is the moment an expert rejects or fixes a model's work.
AI products in finance, legal and support see those moments every day.
If you're a business owner who is curious, or hesitant, about licensing your data, we would love to hear from you.
More in the replies.
4
686
a $30M seed just went to turning agent traces into training environments.
remanence takes a company's agent runs, builds tasks and verifiers from them, then post-trains an open-weight model the company owns. eqt ventures and firstmark co-led.
i think the bet is right. every company that trains its own model becomes a data buyer.
the part i'd watch is the verifier. a trace shows what the agent did. it rarely shows what a senior would have changed, and that edit is the reward signal.
agree, and the grader decides which primitive gets learned.
if the reward only checks the final answer, a model that guesses a balance and gets lucky scores the same as one that opened the ledger.
check the read step, not just the answer, and that primitive has a chance to transfer.
if you break down different types of tasks into basic primitives, it makes a lot of sense that RL generalizes.
i.e. if we split operations into the normal CRUD operations + some domain specific tools, understanding when and how to read records vs making inferences, a model will likely pick up the underlying patterns. we are already seeing generalized intelligence improve some model performances (reflection's beam had an ~8% increase in browser use tasks after training on code). although it's not a perfect 1:1 replication, given a really good dataset that has good underlying principles (and the dataset was made mindfully, instead of a bunch of isolated tasks), a lot more generalization would likely happen than expected.
11
754
the gap is hard to train because the work between people rarely gets graded.
a contract draft goes through several rounds before anyone signs off. most evals score only the final version.
the rounds show what working with others means: what the reviewer pushed back on, and how the drafter fixed it.
One of the biggest remaining gaps in AI is the ability to work with other people.
AI will handle 50% of knowledge work tasks in the digital economy within 3 to 5 years. But 100% is much further out, because people still need to work together.
I expect this to become a major focus of frontier model development over the next few years.
@andyfang turned the questions to me the latest episode of The Frontier ⬇️
1
5
762
anyone sold expert corrections to mercor, micro1, afterquery, handshake, protege or turing?
lawyer or analyst fixes to a model's output, with grades. not raw dumps.
who on their side bought it, and was it for training or evals?
2
14
1,895
same models underneath, and the harness picks the winner.
most evals score the model with the harness held fixed. for agents that run in the background, i'd bet the harness decides more of the result than the model does.
Persistent Agents are the next unlock of AI usage
Our spend on AI is now 45% persistent agents
Unfortunately for Anthropic, Claude Tags sucks
OpenAI's Slack agent is even worse
We primarily use Perplexity Computer, which uses a lot of different models
Perplexity's Harness mogs
2
4
281
the terminal-bench team updated the verifiers, and about 1% of the rewards flipped.
the leaderboard barely moved. that's the quiet version of what echen posted today.
a weak check doesn't just mis-score a run. it pays the model for the wrong behavior.
the 4.1 release tries to resist that. it also cuts false negatives, which are right answers the old check rejected.
if 1% of rewards flipped on a coding bench with unit tests, a legal or finance rubric has more room to drift. pin the grader version. re-score a held-out set when you change it.
Terminal-Bench 4.1 is here with updated verifiers for all tasks.
In this release, we focused on:
- resisting reward hacking
- reducing false negatives
- improving reproducibility
Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.
Leaderboard scores remain within previously reported confidence intervals.
2
8
712
if someone offers six figures for your company's data, ask who trains on it.
a real buyer can name what the model gets wrong and why your data fixes it.
a middleman talks price first and the model last.
what i'd ask before any number:
> which team uses it, and is it for training or evals?
> what gets checked before it's accepted, and who checks it?
> what happens to customer names, emails and client details before anything leaves your building?
>> if the answer is "we handle that", ask how
> do you need raw files, or the graded output of your experts' work?
consent and redaction come before price. a deal that skips them can come back as a lawsuit.
5
605
agree, and the hard part is upstream of the purchase. you can't sell failures you haven't found.
in professional work the failures mostly don't show up as wrong answers. they show up as a senior quietly rewriting the draft. the edit is the failure label.
volume got cheap. finding the miss before the lab does is the whole product.
Catching up with labs and data vendors lately, the same thing keeps coming up: labs now pay most for data on what their models get wrong.
Volume got cheap. Models generate their own code, sim scenes and tasks now. The deals go to data aimed at a weak spot, or data nobody else has.
1. Meta, Nvidia and AWS are buying sim data from Physicl, a Paris startup that just raised a $10M seed. It's built only around scenes their robot models get wrong, like odd lighting or objects they've never seen.
2. 4 of the top 5 US closed labs pay Halluminate, a 9-person team, for finance RL environments like PE due diligence. On its test the best of 7 frontier models averaged 51%.
3. Google paid $10M for a bankrupt airline's 100M emails and 500M Teams messages, and outbid Mercor to get it. Heard exclusive deals like that run 4-5x non-exclusive ones.
The data vendors I'd watch find failures before the lab does, or own something nobody else can sell.
If you sell data to labs: are you getting paid for volume or for failures?
4
468
if every enterprise ends up training its own model, data gets cheaper, not more expensive.
the expensive part is picking the 2% worth training on.
most company data is a record of what happened. a much smaller slice is a record of someone senior saying "no, this way." that slice is the training signal.
more buyers won't make raw dumps worth more. they'll make the people who can find the correction layer in a pile of email and tickets very hard to hire.
a rubric written in plain english is a second model you never evaluated.
"identifies the key risks" reads fine until two graders disagree on what counts as key. then the score is measuring the grader.
it gets missed because rubrics look like documentation, not code. nobody diffs them, versions them or tests them.
where natural-language rubrics break on professional work:
> vague thresholds. "adequately explains" has no failure case
>> write the failing example next to the line, not just the passing one
> hidden order. a credit memo that flags the covenant breach on page 6 instead of page 1 is a different memo
>> most rubrics never say "first," so they don't check it
> double counting. one missed assumption fails three rubric lines and looks like three errors
>> group lines by root cause before you sum
> assumed context. "uses the correct policy" means the policy in the author's head
>> attach the policy, or nobody else can grade the item
> judge drift. swap the judge model and the same rubric gives a different curve
>> pin the judge version and keep a small human-scored set to recheck against
before trusting one: have two seniors score the same 50 outputs blind and read every disagreement. the disagreements are the rubric bugs.
honest caveat: some of this doesn't get fixed with more words. past a point a rubric needs examples, tests or a reference answer, not better english.
2
3
303
most graders for professional work only check the last draft. the more useful signal is how many rounds it took to get there.
an agent that gets a credit memo signed off on round one and an agent that gets there on round four both "pass." the first one saved a senior an afternoon. the second one cost them one. final-answer scoring calls them equal.
what to record per task if you want rounds to be trainable:
(1) every draft the agent submitted, not just the accepted one
(1a) the reviewer's edit or comment between drafts, tied to the exact span it touched
(2) why the draft was rejected, in a small fixed set: wrong fact, missed policy, wrong format, judgment call
(2a) keep the free-text reason too. the fixed set is for counting, the free text is for the grader
(3) who reviewed it and their seniority. a junior's "looks fine" and a partner's "looks fine" aren't the same label
(4) rounds to sign-off as a number, plus the time the reviewer spent
(5) whether the same reviewer flagged the same issue on a previous task. a repeated fix is a policy the agent never learned
then the grader can score two things: did the final pass, and how much human work the path consumed.
>> caveat: rounds are noisy. some reviewers send everything back once on principle, and some tasks get rescoped mid-review. normalize per reviewer before you trust the number.
final-answer graders are easier to build, which is why most datasets stop there. but firms want an agent for the senior's time, and that only shows up in the rounds.
1
5
405












