
founding FDE @sandstonehq // prev: applied AI @blackstone i write about AI engineering, evals, and coding agents
New York, NY
Joined November 2014
- Tweets415
- Following109
- Followers571
- Likes1.2K
the openai models are RL’d so hard that they refuse to listen to the user when the agents.md says something different
i used to think this was nice around gpt 5.5 when opus 4.7/4.8 would completely ignore system instructions (and were generally pretty bad)
but now it’s kind of ridiculous especially as new model releases often make prior instructions obsolete
i tested each model 50 times, using pi as the harness for both
4
10
856
sharing the single most important skill if you want to succeed as an FDE
the definition of FDE varies widely between companies, but a large portion of the roles include some combination of “talk to the customer” and “decide how to solve their problem”. if you don’t do these things exceptionally well it doesn’t matter how strong of an engineer you are
i think of this part of the job as translating from ask to build. customers usually ask for a specific feature or process that they think will solve their problem; if you build exactly what they ask for, it often doesn’t scale or isn’t the best approach
an FDE’s job is to dive deeper on the customer’s problem and ideal outcome, then use their product and technical expertise to choose and build the best solution (even if it’s not exactly what they asked for)
change management can be hard, but many existing processes are inefficient/broken and should be redesigned rather than sped up as-is (@vasuman is a strong proponent of this idea)
there’s a delicate balance between listening to the customer and knowing when to push back. it’s a skill that’s mostly built through experience; i learned it during my three years at blackstone building AI systems for private equity investors and non-technical portfolio company execs
i think this is going to become an increasingly valuable skill as AI continues to accelerate how fast things can be built and customers start asking for more
do you think there’s another more important skill for FDEs?
SlopCodeBench is the most relevant coding agent benchmark left--the top score is 16% vs. 50-75% on other coding benchmarks like Terminal-Bench, FrontierCode, and DeepSWE
summary of how it works: an agent receives a spec for a small program, like a backup scheduler or a mock HTTP server. the agent implements it, then the agent receives a new spec and has to extend the program. this repeats 3-8 times, and with each new spec there are hidden tests that check the new feature and all the old features.
there are a few reasons why I really like it:
1. it comes far closer to simulating real, long-horizon work with new and updating requirements, which is what it’s actually like to work on a codebase vs. solving a single isolated task
2. code coherence, maintainability, and extendability naturally matter more
3. it’s significantly less saturated than the other coding agent benchmarks
full results in the image (credit to @dexhorthy for the benchmark run + charts), the “core” number measures whether the main tests for the new feature pass; the “strict” number is the headline and measures whether every test passes, including regression tests for all earlier features
astra and sol are at the top (interestingly 6.1 is slightly worse than 6 although likely just noise), but given how low the scores are overall + how small the spread is, i don’t think it’s super meaningful yet in terms of determining which model you should use--opus 5.5 is still my daily driver, and i’m kind of surprised it’s not higher
the big caveat imo is that these runs all used the base harness with no customization. i personally think having a couple of core skills + agents.md instructions (planning, verification, etc) is necessary and improves performance, and that improvement is much more likely to show on SlopCodeBench than the other benchmarks
my prediction is that custom context + skills would improve performance by 5 percentage points or more and (hopefully) result in a ranking that’s a better representation of how good different models are to actually use for engineering work
do you think the improvement from proper skills/verification would be more or less than 5%?
for the last ~2 months i’ve been using an orchestrator within @get_bb_app as a single entrypoint to manage my agents
with the launch of cursor projects and @nicbstme’s viral post about AI feeds, figured it was time to have opus vibe up a hype video for my workflow
at this point there are a lot of ways to have one agent orchestrate a bunch of other agents to do work, but there are a few things an ideal experience needs to solve for:
observability: you should easily be able to talk directly to any of the spawned agents; especially for me as an fde there’s still a decent amount of in-the-loop work where going through the manager would be an unnecessary layer of translation
attention: there’s only so much i can focus on at once. the split in bb feed makes it easy: agents still working are in a separate section, so the ones that actually need responses are clearly emphasized. if you have too many that need you, you can “pause” some to set them aside (bottom section) for later
interaction: support various levels of interaction with the agents - let the manager handle it, talk to the manager directly and have it talk to the agents, reply to the agent directly without leaving the feed (either based on the summary line or expanding the card to see the full most recent message), or fully click into the agent session
there might be other things out there that have all three of these but i’m a tinkerer at heart and constantly customizing so bb’s plugin system makes this seamless, shoutout @sawyerhood for building a great product
ARC-AGI-3 from @arcprize has gotten a lot of attention recently with many near-perfect scores
I consolidated the results from the last six weeks; they reveal an interesting spectrum in cost and how much ARC-specific strategy each harness encodes
2
3
258




