
founding FDE @sandstonehq // prev: applied AI @blackstone i write about AI engineering, evals, and coding agents
New York, NY
Joined November 2014
- Tweets415
- Following109
- Followers571
- Likes1.2K
Pinned Tweet
SlopCodeBench is the most relevant coding agent benchmark left--the top score is 16% vs. 50-75% on other coding benchmarks like Terminal-Bench, FrontierCode, and DeepSWE
summary of how it works: an agent receives a spec for a small program, like a backup scheduler or a mock HTTP server. the agent implements it, then the agent receives a new spec and has to extend the program. this repeats 3-8 times, and with each new spec there are hidden tests that check the new feature and all the old features.
there are a few reasons why I really like it:
1. it comes far closer to simulating real, long-horizon work with new and updating requirements, which is what it’s actually like to work on a codebase vs. solving a single isolated task
2. code coherence, maintainability, and extendability naturally matter more
3. it’s significantly less saturated than the other coding agent benchmarks
full results in the image (credit to @dexhorthy for the benchmark run + charts), the “core” number measures whether the main tests for the new feature pass; the “strict” number is the headline and measures whether every test passes, including regression tests for all earlier features
astra and sol are at the top (interestingly 6.1 is slightly worse than 6 although likely just noise), but given how low the scores are overall + how small the spread is, i don’t think it’s super meaningful yet in terms of determining which model you should use--opus 5.5 is still my daily driver, and i’m kind of surprised it’s not higher
the big caveat imo is that these runs all used the base harness with no customization. i personally think having a couple of core skills + agents.md instructions (planning, verification, etc) is necessary and improves performance, and that improvement is much more likely to show on SlopCodeBench than the other benchmarks
my prediction is that custom context + skills would improve performance by 5 percentage points or more and (hopefully) result in a ranking that’s a better representation of how good different models are to actually use for engineering work
do you think the improvement from proper skills/verification would be more or less than 5%?
the openai models are RL’d so hard that they refuse to listen to the user when the agents.md says something different
i used to think this was nice around gpt 5.5 when opus 4.7/4.8 would completely ignore system instructions (and were generally pretty bad)
but now it’s kind of ridiculous especially as new model releases often make prior instructions obsolete
i tested each model 50 times, using pi as the harness for both
4
10
856
Ryan Brown retweeted
Your repository shouldn’t need a data-entry project.
In Sandstone:
50+ key fields fill in automatically from every document
Calculated fields compute themselves, so there’s no side spreadsheet
Messy file names are renamed to match your format
Add a custom field in plain English, like “Renewal Notice,” and it fills down every row
Sandstone also looks past the single page in front of it. It dedupes repeat uploads, keeps versions in order across sources, and links each document to its family: the MSA, amendments, order forms and SOWs. When an amendment changes the term, the repository shows the current deal.
It all runs on context Sandstone extracts as documents land, so the repository stays complete and accurate as it grows.
underrated technique i’ve been using to build evals for laywers at @sandstonehq that i haven’t see anyone talking about: decomposing answers into key facts
if you give an LLM judge a raw reference answer to compare to, a binary 0/1 is usually too simple and misses nuance, but an LLM-generated 0-1 score is hard to calibrate and interpret--what parts of the answer are actually necessary vs. optional fluff? what does a 0.3 vs. 0.5 mean?
instead, decompose the reference answer into a set of “key facts”: minimal statements or pieces of info that a good answer needs to contain in order to be correct
each fact gets scored separately with a binary 0/1, and now for each reference answer in the eval you get a 0-1 score that’s meaningful, consistent, and allows you to diagnose where the agent is falling short
any other interesting eval techniques you've been using?
95%+ of code doesn't need to be reviewed line by line anymore if you have specific + granular specs and adversarial cross-model review. for the other 5% (permissions, data model, etc.) slop is highly detrimental
worrying about neater types and variable names obviously doesn’t matter, and code duplication can actually be beneficial if it results in coherent colocation
permissions are different: if you let an agent slop up a permissions model and the rules end up duplicated in many places, there’s a significantly higher chance of something slipping through, which in the worst case (i.e. high-stakes environment like legal tech) could literally result in a lawsuit
same with the data model: migrations are still painful; if you initially set things up poorly, your data model becomes a mess and you end up with backfills, multiple versions/schemas, and a system that’s rigid + hard to extend
from what i’ve seen working at an incredibly fast-paced and AI-native startup @sandstonehq, the more common risk is behavioral slop: not being specific enough in what you ask for, the agent making reasonable-but-not-correct choices at a bunch of small decision points, and finding out days or weeks later when customers report a bug
Im sorry im not buying this ‘not reviewing code’ thing. Maybe later, but certainly not now, even with frontier models with max thinking
Every single day I’m catching nonsense. If you’re merging this shit without looking, the systems your working on are either
- not important and/or
- have zero consequences for being wrong
5
13
560
sharing the single most important skill if you want to succeed as an FDE
the definition of FDE varies widely between companies, but a large portion of the roles include some combination of “talk to the customer” and “decide how to solve their problem”. if you don’t do these things exceptionally well it doesn’t matter how strong of an engineer you are
i think of this part of the job as translating from ask to build. customers usually ask for a specific feature or process that they think will solve their problem; if you build exactly what they ask for, it often doesn’t scale or isn’t the best approach
an FDE’s job is to dive deeper on the customer’s problem and ideal outcome, then use their product and technical expertise to choose and build the best solution (even if it’s not exactly what they asked for)
change management can be hard, but many existing processes are inefficient/broken and should be redesigned rather than sped up as-is (@vasuman is a strong proponent of this idea)
there’s a delicate balance between listening to the customer and knowing when to push back. it’s a skill that’s mostly built through experience; i learned it during my three years at blackstone building AI systems for private equity investors and non-technical portfolio company execs
i think this is going to become an increasingly valuable skill as AI continues to accelerate how fast things can be built and customers start asking for more
do you think there’s another more important skill for FDEs?
Ryan Brown retweeted
took a while to get these all collated together, but excited to share latest SlopCodeBench (scbench dot ai) findings. Shouts out @gorlanski for helping put this together - findings should make it into the main results leaderboard soon. View article
LLMs are great at software engineering but still quite bad at AI engineering, here's what i'd put in a skill to fix that based on my experience building agents for private equity investors and lawyers:
1. no target leakage. LLMs are laughably bad at this by default; if you ask it to hill climb an agent’s system prompt it’ll put in the exact examples you’re testing against. i’ve found adversarial review works well here
2. show, don’t tell. detailed instructions telling an agent how to do something are less clear / hard to follow and you’ll bitter lesson yourself as the agent finds ways to complete its task that you didn’t think of. when you see undesirable behavior, try to add an (actually generalizable) example that clearly displays what it shouldn’t do
3. keep prompts modular, MECE, and free of contradictions. @wulfie_bain_ has a really good article on this (pinned to his profile) but basically agents just continue adding to prompts and you end up with prompt spaghetti with repetition and contradictions throughout
4. know when to raise. we’ve largely moved past reading the code but i don’t think we’re there yet for prompts; sometimes a quick human read-through of the prompt can save 5-10+ hours worth of autonomous hill-climbing (given the much slower cycle time for agent engineering vs. software engineering). i experienced this firsthand when i built Retrodict, a frontier agent harness for ARC-AGI-3
5. evolve the eval. hill-climbing against an eval that’s subtly incorrect or just measuring the wrong thing can waste a lot of time and make it seem impossible to improve the agent. this isn’t always necessary but evals (especially domain-specific ones) should usually be human-curated and that means they can have mistakes and gaps. you even see this at the highest level with frontier LLM benchmarks changing and improving over time
anything i missed?
for the last ~2 months i’ve been using an orchestrator within @get_bb_app as a single entrypoint to manage my agents
with the launch of cursor projects and @nicbstme’s viral post about AI feeds, figured it was time to have opus vibe up a hype video for my workflow
at this point there are a lot of ways to have one agent orchestrate a bunch of other agents to do work, but there are a few things an ideal experience needs to solve for:
observability: you should easily be able to talk directly to any of the spawned agents; especially for me as an fde there’s still a decent amount of in-the-loop work where going through the manager would be an unnecessary layer of translation
attention: there’s only so much i can focus on at once. the split in bb feed makes it easy: agents still working are in a separate section, so the ones that actually need responses are clearly emphasized. if you have too many that need you, you can “pause” some to set them aside (bottom section) for later
interaction: support various levels of interaction with the agents - let the manager handle it, talk to the manager directly and have it talk to the agents, reply to the agent directly without leaving the feed (either based on the summary line or expanding the card to see the full most recent message), or fully click into the agent session
there might be other things out there that have all three of these but i’m a tinkerer at heart and constantly customizing so bb’s plugin system makes this seamless, shoutout @sawyerhood for building a great product
legal engineers are one of our biggest assets at @sandstonehq.
they combine legal expertise with engineering capability and can build complex legal features (like redlining and contract drafting) end-to-end, without the usual friction/slowdown that comes with engineer <> lawyer coordination
Looks like everyone is hiring Legal Engineers this week:
Legal Engineer @Microsoft ($278.9K)
Legal Engineer @heyivoai ($320K + equity)
Legal Engineer @Holland_Knight ($245K)
Legal Engineer @EpiqGlobal ($245K)
Legal Engineer @goodwinlaw ($236.5K)
Legal Engineer @harvey ($320K)
Legal Engineer @LexisNexis ($209.7K)
Legal Engineer @WeAreLegora ($325K)
it’s been ~2 months since this banger tweet, I think it’s still super true and it influences how I think about feature development at @sandstonehq
TLDR of the tweet: long-horizon agents are a poor fit for truly novel product work because complete requirements do not exist upfront and present models cannot reliably judge what humans find intuitive, enjoyable, or valuable. they are better at reproducing known forms—like a Call of Duty clone whose requirements have already been absorbed into model weights—than inventing profitable new products alone; novel work needs tight human-agent loops that prototype, test with people, question the approach, and iteratively discover requirements.
especially as an FDE a big part of my job is to talk to the customer and figure out what’s actually needed to solve their pain point; balancing their specific ask with something that’s cohesive with the rest of our product
but for a decent portion of features, once that’s figured out, it is possible to create a complete definition of the requirements upfront
with those requirements, an agent can autonomously plan, build, self-review, and end-to-end verify without my input, only raising things to me when implementation reveals that we should deviate from the requirements for some reason
because of the way AI agents write though, creating that definition in a way that’s easy and reasonable for me to review + iterate on is nontrivial
i’ve been playing with the idea of a “behavior spec”; a Linear document with a set of scenarios (grouped under various headings) that have a descriptive title and three bullets: one for the setup (”before”), one for the action taken, and one describing the state “after” (including a comparison to the current behavior without the feature)
they have to be quite detailed/granular in order to be complete; for a feature i’m currently working on, there are 62 scenarios split across two behavior spec documents. it seems long and daunting, but because the scenarios are written in such a simple way (instructed with a skill) they’re actually quite quick to review, quicker (at least for me) than agent-written documents that are shorter but require more mental leaps / context held in working memory
i leave comments on Linear and iterate with my agent until i agree with the entire document, at which point autonomous implementation begins
note that this on its own isn’t enough for features that have:
- a large amount of net new UI
- complex or ambiguous architecture decisions (things that aren’t clear from following existing codebase conventions + docs)
for the former, aligning first on a figma is super helpful; for the latter a separate document to align on key architecture decisions is usually necessary
opus 5 is a sign that the obsession with “long-horizon agents” in model training is finally backfiring
i don’t like long-horizon agents, and i’ll explain why they fundamentally don’t work
some people will immediately jump out and say “skill issue”. well, show me one profitable business you built with a long-horizon agent working all by itself - i’d love to learn
so far, the only thing they were able to build that’s even interesting enough for people to talk about are those 3d games that are a partial clone of something that already existed
the reason an agent was able to build a working prototype of complex games like call of duty was that a team of humans already figured out all the requirements years ago for how such games should work, what kind of controls are intuitive, what mechanics are fun etc
all those requirements were already absorbed into the model weights, so when you say “build me call of duty” the model already knows the details. its long horizon execution capability can get all the requirements implemented, which i must say is indeed impressive
but now you can see - the value of long horizon execution has a prerequisite of a massive amount of high quality requirements clearly defined upfront. it took a big team of very talented humans months of effort and many iterations to define that for call of duty
now imagine games like call of duty don’t exist yet, how would we use agents to build it for the first time? we can’t say “build me call of duty” any more. and there’s no way we can define months-worth of game design details upfront
we’ll have to build a tiny prototype of the most basic mechanics, play with it, see if it’s fun, then iterate and expand the complexity. even with the smartest humans, that’s how we work towards something great
we don’t need agents to go dark for a long time, spend tens of thousands of dollars worth of tokens, and come back with a product the agent randomly decided to build - try build something truly novel with this and you’ll see it can’t come up with anything that’s actually profitable (i’ll show you why in a bit)
we need a tight feedback loop where we can collaborate with the agent, plan with it, understand what it’s done, question its approach, apply our judgement, give it real world feedback and iteratively arrive at a good outcome
and that’s exactly what opus 5 absolutely suck at. why? i explained it in more depth with my previous post on how RLVR works - RLVR trains the model to generate code that can pass predefined tests in an isolated environment, which is fundamentally incompatible with the idea of having human in the loop. the more we train the models with RLVR to be “long-horizon”, the less they care about talking to humans
ok now - why do they have to talk to humans? why can’t the models iterate and apply judgement by itself?
maybe one day they could, but not today, due to many limitations. two examples -
1. LLMs today can’t “watch a video” yet. they can look through a lot of screenshots, which is extremely inefficient at observing a high fps animated signal. so anything that requires continuous visual attention is something LLMs can’t do very well
2. LLMs don’t truly understand what’s “intuitive” or “pleasant” for humans. they know what’s already proven to be intuitive and pleasant in the past, but if you present a truly novel concept, it can’t predict whether humans will like it accurately
because of those limitations, human judgment is still needed for almost anything valuable. without humans in the loop, agents will only be able to repeat something that already existed, or go in random directions without true understanding of whether it’s building something useful
in summary, long horizon agents assume requirements all exist upfront. they are fundamentally against human in the loop. and they don’t have true judgement for what humans like
that, my friend, is why they don’t work
going to start providing commentary on some of the opinions from my opinions agent (see pinned tweet for more detail) that I feel most strongly about
first installment: AI can make an individual faster while making the organization slower when unedited output transfers verification, compression, and understanding to many downstream readers. generated work is not finished until its author has checked it, cut it to what matters, and can defend every claim; otherwise local productivity is merely displaced cognitive cost.
source: @matheusml terriblesoftware.org/2026/06…
why I agree with this
- I think it’s rude and borderline disrespectful to send someone slop that you haven’t read/reviewed yourself and it takes longer for them to consume than for you to generate
- it’s easy as the producer to think “oh well there are certain parts that are clearly slop / not important / skippable”, but a reader with doesn’t know what you actually agree with and has less context so it’ll take even longer for them than it would have for you to review it
- I agree with @kunchenguid that anything that’s one-shotted is by definition slop, so unless your agent had some really unique / hard-to-replicate context before you one-shotted it, honestly more useful to send someone the prompt you used
size of impact
- this is mostly likely to have a negative organization-wide impact on productivity/output at large, interconnected organizations that move fast but are still ramping up on AI and don’t have a strong culture around it. the size also makes a strong culture harder to create and spread
- for more isolated teams (<3-6 people working on a given project/product area and can work relatively independently, even if part of a larger group) that don’t move as fast (decisions on what work gets done goes through layers of management/bureaucracy), the volume isn’t high enough for it to be more than a personal inconvenience. speaking from experience based on my last job
- for highly AI-native small to medium teams (e.g. startup) it can be a problem if unaddressed but much more likely to be addressed culturally; at @sandstonehq we’ve explicitly discussed this for the eng team and slop is minimized + transparently communicated
one note is that none of this applies to content intended to be consumed by an agent rather than a human; i’m starting to see more of “attached a markdown doc with more detail for your agent to read” or similar, which is an interesting multiplayer use case
Ryan Brown retweeted
shoutout @c_amataa for building the best slack and salesforce experience on the planet
Replying to @sandstonehq
@Sandstonehq brings your legal team's workflows into Slack. intake, triage, drafting, redlining, all in real time. @mention it and it asks the right questions, pulling from your business context.
Tag: @Sandstonehq👇 sforce.co/3TG1DHV
Replying to @sandstonehq
@Sandstonehq brings your legal team's workflows into Slack. intake, triage, drafting, redlining, all in real time. @mention it and it asks the right questions, pulling from your business context.
Tag: @Sandstonehq👇 sforce.co/3TG1DHV
inspired by @spakhm for the format + emphasizing the lower friction of tweet vs. blog and @kunchenguid for the inspiration for OPINIONS.md, sharing my process for generating opinions and a selected few of my favorites:
process
- kun chen’s opinions.md agent takes in his tweets / blog posts as input; i don’t tweet or blog nearly enough and half of the time they’re about side projects anyways, so that wouldn’t be a valid source of opinions
- instead i leveraged Readwise, which is what i use to centralize my reading + storing of content from substack, twitter, web blogs, etc.
- i highlight interesting or important snippets on some articles when it feels worth saving for later; the agent runs on Railway on a weekly cron, receives the highlights, my existing opinions, and can read more into each article if needed
- since these aren’t things i wrote myself i may not agree with them, so the agent sends me telegram messages with opinion proposals that i can accept, reject, or reply with feedback. it can revise existing opinions or attach new evidence that supports existing opinions
- this ended up being a super interesting project on agent and eval engineering + evolution; what defines a good opinion? what kinds of mistakes (false positive vs. negatives) do i care about more? how can i make the agent consistent on something inherently qualitative?
opinions
- people remain accountable for AI-generated artifacts they ship and should understand them well enough to defend them under questioning; review culture should give harsh, specific pushback when output crosses into slop.
- AI makes volition more important, not less. the people who benefit most will actively wrestle with it to strengthen their own capabilities and accomplish more, rather than using it merely to avoid effort. education should therefore prioritize the desire to work hard, learn, and put abundant machine knowledge to creative use.
- a vertical AI startup can defend itself against foundation-model labs by obsessing over a narrow set of real customer workflows. continuously turning those workflows into benchmarks and product improvements creates a learning loop the labs are unlikely to match because the vertical is the startup’s whole business, not the lab’s main quest.
- as AI makes products easier to imitate and more superficially similar, specialized product taste becomes more defensible. taste comes from deep understanding of a particular user’s work and compounds across many small choices in the UX, model, defaults, and interaction; individual pieces can be copied more easily than the coherent whole.
these last two are (part of) why i’m so bullish on @sandstonehq
and these four are a small subset of the 84 total opinions as of today; the list keeps growing every week, the full thing is on my website/github
ARC-AGI-3 from @arcprize has gotten a lot of attention recently with many near-perfect scores
I consolidated the results from the last six weeks; they reveal an interesting spectrum in cost and how much ARC-specific strategy each harness encodes
2
3
258
currently, all harnesses (including the simpler pro-long and potentially prime agent) require specific instructions on how to approach ARC. a human doesn't.
humans can quickly + easily figure out the rules and solve the games (which is why action efficiency relative to humans is what the benchmark is scored on)
so obviously ARC doesn't quite measure AGI, but I think there are two interesting frontiers left to be broken on the arc-agi-3 games:
1 - a harness as or more general than pro-long achieving 99%+ for a reasonable cost
2 - a harness with no specialization and no instruction beyond "you're being given a 64x64 game and your goal is to figure out the rules + solve it" achieving 95%+
1
33
chart methodology: I selected the best performing run for each harness; for cost I either used the reported cost or calculated it myself (e.g. codex subscription used, chart-only)
full detail: github.com/ryanbbrown/Retrod…
33
the level of specialization differs across harnesses:
highly specialized harnesses (baseline1, Tycho, Schema) saturate the benchmark (99%+) but are expensive ($2500+)
Retrodict is the exception to the cost rule, achieving 99.9% for only $654
pro-long from @alexisfox and team is more general, with a small (~400 word) system prompt and ARC-specific logic coming in through what gets appended to the text log; it comes close on performance but is a small step below, with 94.6% pass@1 and 97.4% best@2
1
60
prime agent from @PrimeIntellect says the task prompt was its only ARC-specific change, but the exact prompt and setup are unreleased
if the prompt is similarly minimal as pro-long's, then the 95.5% for ~$944 likely represents a new point on the cost/performance/specialization tradeoff
however, if the prompt is basically just text encoding/instruction of the more specialized 99%+ harnesses, it is still a new and unique datapoint but isn't quite as interesting
1
30


















