The Whiteboard Guy and Factory Foreman @warpdotdev
Ordering more markers
Joined September 2018
- Tweets15.6K
- Following1K
- Followers49.1K
- Likes37K
Pinned Tweet
We made a skill for assessing the risk of merging a PR: /merge-risk
This addresses the code review bottleneck by helping devs trust the agent for lower risk changes.
It grades risk across 5 categories: alignment, impact, verification, rollback cost, and error detection.
- Alignment: does code follow the PR description and related tickets?
- Impact: what is the blast radius, and does it touch sensitive code paths?
- Verification: does the change include end-to-end verification? This means computer use, screen recordings, before/after screenshots... going beyond unit test coverage.
- Rollback cost: once it's shipped, how trivial is it to revert? @mattpocockuk calls this a "two-way door" vs. a "one-way door."
- Error detection: are errors reported, logged, or simply swallowed?
Try it out:
> npx skills add warpdotdev/common-skills --skill merge-risk
Ben Holmes retweeted
I’ve been working on an iOS factory to help build out our mobile app, and wanted to share a cool self-improvement PR it produced.
Across at least 10 unique runs sent to our mobile-factory, our scorer agents kept flagging failures.
It turned out the Simulator skill was trying to launch an app with a stale bundle ID. The agents then followed the bad instruction, failed to launch the app, and couldn't verify the UI behavior with our iOS simulator.
Once enough of those failures accumulated, the factory autonomously triggered a self-improvement PR: instead of hard-coding the Simulator launch identifier, the agent now reads CFBundleIdentifier from the built app’s Info.plist and passes it on.
After I merged this in, the factory started sending me computer use-verified videos again.
Autonomous self-improvement in these systems is so cool!
If you're going to build a software factory, you should try these two loops: self-improvement and benchmarking.
Self-improvement: have agents 1. score conversations as an LLM-as-a-judge, and 2. review poor performers for improvements.
Benchmarking: replay past conversations against different model selections, and use LLM-as-a-judge to grade outputs to find a winner.
Self-improvement drives up factory quality, and benchmarking drives down factory cost. Combined, you get a better factory!
ALT A dashed orange circle labeled "Self-improvement loop" surrounds a core factory loop with four steps: review agent conversations, score quality (rubrics: efficiency and code quality pass, test coverage fails), isolate failures ("5 conversations wrote redundant tests"), and propose improvements every 24 hours (a test-writing/SKILL.md edit adding a coverage pyramid). A dashed green circle labeled "Benchmark loop" also has four steps: sample past conversations, benchmark Claude, Grok, GPT, and GLM on task scores, pick the winner on a cost vs. performance chart, and deploy model changes (PR review agent swaps a frontier model for an open-weight one).
For self-improvement, we made a skill you can add to your harness of choice to try this out. For benchmarking, well, it rhymes with Shmarp Shmactories :)
warp.dev/skill-doctor
Gemini 4 Argon is the best copywriter I've ever seen, and it's not even close.
It matches tone exactly and has the best default voice
A lot more goes into building a software factory than you might think. There are 7 must-haves we've landed on:
- Configured as code
- Central conversation history
- Benchmarking tools
- Factory chat
- Artifacts
- Computer use
- Agent debugger
In detail:
Ben Holmes retweeted
From working with teams building software factories, we know that this is a big behavior change and needs a lot of thought in how to approach it.
The important principle is meeting engineers where they already work. Introducing a new product surface or workflow for inputting factory work is going to be higher friction than allowing engineers to tag agents in the tools they are already using (GitHub, Slack, JIRA, etc). We do not recommend building a new “factory web portal” as the primary place to work with a factory.
Stop relying on public benchmarks. Everyone has the ingredients to run their own benchmarks now.
You probably have thousands of agent sessions that you can replay to decide what model and reasoning level are best!
3d game demos aren't the best way to judge a model's coding ability according to @theo. neither is SWE-bench, CursorBench, or TermBench.
instead, you should benchmark models against your own traces to find the best one for your tasks.
here's how the Warp team benchmarks various harness/model configs, which helped reduce cost / PR from $80 to $20 in our software factory (alongside uncovering a number of unintuitive findings!)
Demo:
Got to speak at the AWS Loft in San Francisco on self-improving agents alongside @sethkarten (Prime Intellect) @sydneyrunkle (Langchain) and Andrey Risukhin (@Factory).
Our talks circled the same idea: we need to give agents tools to edit and improve their own setup.
That means:
- Let agents edit their own skills by learning from past mistakes.
- Let agents improve their own workflows, and even their own harnesses, while they work. @sethkarten's research on self-improving harnesses is especially interesting.
- Let agents improve a project's code architecture for better outcomes. That can mean better code patterns (see @mattpocock's "deep modules" suggestion) or hillclimbing on SOLs like compilation times or startup times.
It's never been a better time to be a systems thinker. Thanks to @Shashikant86 for organizing this event!
ALT Ben presents at AWS Loft in San Francisco, discussing self-improving agents to an engaged audience.
ALT Diagram outlines a process for improving agent performance through regular reviews, proposing improvements, and isolating failures.
ALT An empty auditorium features chairs arranged neatly, with presentation screens displaying event details on engineering and software.
There's a balance flip going on with software factories:
Before: I ping an agent, we work together
After: Agent finds what to do, then it pings me with questions
It's the right direction for managing human attention as we scale up the volume of tasks
You should focus on adjusting code architecture and skills to make agents perform better, yes.
But what's more important is having metrics to track that those adjustments are working!
These could be quantitative (we use cost-per-PR), or qualitative (LLM-as-a-judge scoring).
At Warp, we are having fun tweaking our model mix and factory workflow to drive our cost per PR down. It’s gone from $80 → $20 using a factory approach, with a lower revert rate and 40% higher PR throughput than in the last month.
Folks on the team are suggesting experiments to improve further, like reducing subagents and building model routers. As a startup founder who cares deeply about efficiency and burn, seeing us reduce costs while increasing velocity and quality is incredible.
ALT Line graph shows cost per PR trending downward from mid-September to mid-October. It shows a peak at $80 in aggregate across compute, platform, and inference costs, and a lower total of $20 at the far right of the graph.
Ben Holmes retweeted
At Warp, we are having fun tweaking our model mix and factory workflow to drive our cost per PR down. It’s gone from $80 → $20 using a factory approach, with a lower revert rate and 40% higher PR throughput than in the last month.
Folks on the team are suggesting experiments to improve further, like reducing subagents and building model routers. As a startup founder who cares deeply about efficiency and burn, seeing us reduce costs while increasing velocity and quality is incredible.
ALT Line graph shows cost per PR trending downward from mid-September to mid-October. It shows a peak at $80 in aggregate across compute, platform, and inference costs, and a lower total of $20 at the far right of the graph.
This is a really great article. To start, we should be adding some version of this to every PR description: "Testing", "Risk," and "How to revert."
"Testing" should include static checks and end-to-end verification from the agent. Screen recordings with computer use are the holy grail.
"Risk" should encompass both the depth of the change (i.e. is it a copy change or a deeper system change?) and the sensitivity of the change (i.e. does it touch mission-critical backend or PII?). Anthropic uses a low - medium - high ranking.
"How to revert" should be a playbook for incident response. If a revert wouldn't be easy, consider how it could be!
All of these serve to make code review faster, and to make bouncing back from incidents easier.
Ben Holmes retweeted
💻 Code Engineering: From Coding Agents to Software Factories
Excited to have our next event in San Francisco 🌁 on coding engineering tomorrow (27th October). Don't miss out the chance to learn from the innovative agent labs and learn more about the coding agents and software factories..
🎙️ Amazing Speakers
@sethkarten - @PrimeIntellect
@BHolmesDev - @warpdotdev
@sydneyrunkle - @LangChain
@AndreyRisuka - @FactoryAI
We are about to hit the limit but still some spaces available
👉 RSVP Now: luma.com/cico2dxy
Ben Holmes retweeted
On Friday, we became aware of what we believe to be an active and narrow-banded exploitation of a vendor in Warp's infrastructure. We detected the operation within hours and were able to remediate and mitigate the impact. We have been in touch with the vendor about the vulnerability.
Everyone who uses Warp places a lot of trust in us and that’s not something we take lightly. I’m very sorry that this incident has occurred.
The attack appears to have come from an agent swarm using a very sophisticated penetration and escalation technique, likely starting with a hole in a vendor dependency, and ultimately accessing a limited amount of agent trace and telemetry data.
We have been working over the weekend and we have notified anyone who we have positively identified as impacted. As of today, we believe an upper bound of roughly 1% of Warp users may have been impacted.
We are working closely with the team at Crowdstrike to do a more extensive and comprehensive investigation and have engaged law enforcement.
Anyone who has been impacted will receive instructions from us on action they need to take. We will continue to post updates as we learn more.
We need better debugging tools for agents. So, we've been building them:
- A timeline to show each subagent handoff, with visibility into each subagent trace.
- Cost breakdown per subagent (both token and compute cost)
- Models used at each handoff. Especially useful with automatic routers
- [testing internally] A tool-by-tool flame graph to identify wasted work
These are built into Warp Factories for you to try!
Ben Holmes retweeted
final edit of our software factories masterclass with @vaibcode and @zachlloydtweets is live! We chatted about:
- software factory building blocks
- a good software factory starts with measurement, then self improvement against those metrics
- turning successful agent runs into evals for testing future models / harnesses / etc
- DORA, why counting lines of code is bad metric, and why counting PRs might be good enough
enjoy!
Ben Holmes retweeted
Want better quality output from your agent? Give it a way to verify its own work.
Love what Warp is doing here: every agent gets a computer and brings back a video of what it did. I do something similar in my workflow - on top of E2E tests, Claude gives me recorded videos + screenshots as part of the evidence work is "done".
My bet is verification becomes one of the most important problems to work on this year.
verification is the biggest bottleneck for coding agents right now according to @addyosmani.
we @warpdotdev have put a lot of thought into building a good computer-use tool for Warp Factories. every agents gets a computer to navigate the apps they built, take screenshots and recordings, and post them back for human review.
3-4mins of activity get edited down to 15 clips stitched together with cursor clicks, mouse animations, and key presses.
we've been able to dramatically cut our time to merge and unlocked faster iteration + review cycles, especially for projects that are hard to mock out locally!
Engineering is now 20% prompting, 80% tuning the agents that execute on the prompts
Position software factories not as a “product,” but as an infra platform upon which everyone is expected to do “factory engineering.”
Factory engineering is work to make internal factories better. That means improving the skills, models, and context that drive engineering throughput. It means encouraging engineers to build automations for their chore work. It means giving them the tools to score and benchmark factory workflows, and empowering them to use agents to build self-improvement loops. It means that everyone owns lowering factory costs and improving factory ROI.
If you use a platform like Warp Factories, all of this tooling comes built-in, but beyond the tooling “factory engineering” is really a mindset.
Stop asking agents for text summaries of a PR. They're capable of so much more:
- Architecture diagrams to visually explain the change. Could be ascii, mermaid, HTML, or animated walkthroughs. /show-me is a great starting point.
- Screen recordings verifying the change end-to-end. Most harnesses can do this now with Chrome DevTools or computer use. Warp Factories will auto-edit video and highlight cursor clicks as well.
- Planning documents when humans were in the loop, checked in as code. This is optional, but a product spec explaining the purpose and expected user flow will a) guide the agent to better screen recording verification, and b) help the reviewer when looking at another engineer's work.
Warp Factories shows all artifacts connected to an agent session to make these easy to find.
I met a team of devs regularly zipping up Claude and Codex sessions from their computers and shipping them to a shared S3 bucket.
Why? Because collecting data on agent convos is that valuable. I think shared cloud sandboxes are the way though to avoid all the zipping 😄
We should stop keeping all of our agent conversations local. They should be stored in a shared place.
This lets you do some powerful things:
- Schedule self-improvement agents to review runs and pull out failure modes worth fixing
- Replay past conversations against different models for benchmarking
I met a team of devs zipping up Claude and Codex sessions from their computers and shipping them to a shared S3 bucket. That's... a start 😄 But really, all agent sessions should happen on a single source (like a cloud runner) so that conversation traces are stored together by default.
It's why Warp Factories uses cloud sandboxes and makes sessions available via API. We also added an inbox view to see which conversations need your attention, which is quite convenient: