Ambassador @FactoryAI Building Annular, 300+ PRs a month with Droid and Devin Honest tests of AI coding tools and models
Joined December 2021
- Tweets845
- Following250
- Followers111
- Likes345
Pinned Tweet
Droid saves me 15–20 hours a week on acceptance testing. Here's the issue workflow behind Annular, the app I'm building right now. More infos soon :)
1. Every finding becomes a GitHub issue: from a desk run, a code review or a test. Each one gets labels and a decision on the spot: does it block the current goal, or does it wait?
2. Simple ones get resolved by automation.
3. Anything that touches windows, input, displays or live sync does NOT close on green CI. Those PRs say "Refs", not "Closes". Learned that the hard way: two fixes closed on green CI and failed at the desk two days later.
4. For those, I sit down with Droid and explain the case and the pass rule up front, e.g. "fullscreen window on another Space must show up in the list, 10 out of 10 tries". Droid writes a script for exactly that case.
5. Droid runs on my PC and drives the Mac over SSH, so cross-device cases get measured on both machines in the same run. Tests that need a quiet desk wait until I'm away and abort on my first keystroke.
6. Every run leaves a record in the repo: build commit on each device, exact commands, numbers with n, verdict, and what didn't match. Raw evidence gets zipped to a backup with checksums.
7. I read the results. The issue only closes once that record is on main.
Same setup covers acceptance of every new feature, not just bugfixes: the pass cases are written before the code.
Thinking about turning this into an open source skill where you plug in your own devices and test cases. Would you use it?
@FactoryAI @droid
We’re so lucky to have @droid and @cognition. I’m enjoying my holidays on the beach while shipping just as many PRs as I would at home, all thanks to these two tools. Being able to work from anywhere is amazing.
He is right we all need to chill. Opus 5.5 is more than enough, and it’s getting improved over time.
I no longer believe Anthropic is secretly routing Fable 5.5 through Haiku, Sonnet, and Opus in Claude Code.
I think they're just rolling out newer checkpoints with more recent knowledge cutoffs, running continuous A/B tests, and improving their models every day.
Asking a model what Token Gremlin posted last week and getting an answer doesn't magically mean you're talking to Fable.
At this point, we're all going crazy waiting for a model that's taking so long to launch it's practically becoming a legend.
reading a lot of posts about kimi, i really hope they get close to opus. open source needs to win and moonshot is my favorite open source lab
That is why I use Fusion at the moment.
It works great, it is faster and cheaper than just using the model alone.
Give Fusion a shot you won’t regret it.
Devin Fusion with Claude Opus 5.5 (max) and a GPT-6 Luna (high) sidekick scores 68 on the Coding Agent Index, 2 points above Claude Code with Opus 5.5 (max) at 31% lower cost per task
@cognition's Devin Fusion runs a frontier lead model alongside a lower-cost sidekick model. The two work in parallel with their own context and tools, with the lead keeping planning, delegation and final review. We benchmarked Devin Fusion CLI with Claude Opus 5.5 (max) + GPT-6 Luna (high).
It scores 68.1 on the Artificial Analysis Coding Agent Index, closely matched at the top with Claude Code with Claude Sonnet 5.5 (max) at 68.4 and Muse Code with Muse Wonder Full (max) at 68.0. Against Claude Code with Claude Opus 5.5 (max) at 66.0, it costs $9.01 vs. $13.04 per task and averages 55 vs. 65 minutes per task.
Across the three benchmarks, the Fusion configuration scores higher on DeepSWE v1.1 (72.6% vs. 68.4%) and SWE-Atlas-QnA (69.6% vs. 66.4%) and slightly lower on Terminal-Bench 4.0 (62.1% vs. 63.1%). Its results reflect both models working together, including their combined API cost. Costs are pay-per-token API costs, not subscription prices.
Mads retweeted
Environments remove the cold start problem associated with running agents in sandboxes. Happy building!
Promised a rerun, so here it is with Opus 5.5 High.
Same prompt in @DevinAI and @droid, no skills, no subagents, no files.
Devin: 43 min, $6.42
Droid: 38 min, $7.30
Droid was faster, Devin was cheaper and the cut views came out cleaner. Honestly both are crazy
Gave Gemini 4 Argon (high) the same prompt in @DevinAI and @droid: build a 3D anatomical heart
Devin: 8:11 min
Droid: ~19 min
I like Devins result more. Droid felt like it had latency issues all day today, so not sure if the long time comes from that, its not usual for me. Wanted to add Opus 5.5 too but in Droid it aborted after exactly 2407s of reasoning, 4th time today
planning more harness comparisons like this, let me know if you want more
turned this into a skill today, open source now nitter.cf/Laz01L/status/21115815…
Spent the whole day turning my desk acceptance workflow into a skill, built with @droid.
Green CI never meant it works at my desk. With Drayl the agent runs the live check on the real machines, and every run ends as pass, fail or interrupted, with the raw evidence saved. The PR side is in there too.
Its built on months of my own workflow. Found a bug or have an idea, open an issue. I read every single one.
github.com/mxdsl/drayl-skill…
Spent the whole day turning my desk acceptance workflow into a skill, built with @droid.
Green CI never meant it works at my desk. With Drayl the agent runs the live check on the real machines, and every run ends as pass, fail or interrupted, with the raw evidence saved. The PR side is in there too.
Its built on months of my own workflow. Found a bug or have an idea, open an issue. I read every single one.
github.com/mxdsl/drayl-skill…
this is the workflow from my post yesterday nitter.cf/Laz01L/status/21111627…
Droid saves me 15–20 hours a week on acceptance testing. Here's the issue workflow behind Annular, the app I'm building right now. More infos soon :)
1. Every finding becomes a GitHub issue: from a desk run, a code review or a test. Each one gets labels and a decision on the spot: does it block the current goal, or does it wait?
2. Simple ones get resolved by automation.
3. Anything that touches windows, input, displays or live sync does NOT close on green CI. Those PRs say "Refs", not "Closes". Learned that the hard way: two fixes closed on green CI and failed at the desk two days later.
4. For those, I sit down with Droid and explain the case and the pass rule up front, e.g. "fullscreen window on another Space must show up in the list, 10 out of 10 tries". Droid writes a script for exactly that case.
5. Droid runs on my PC and drives the Mac over SSH, so cross-device cases get measured on both machines in the same run. Tests that need a quiet desk wait until I'm away and abort on my first keystroke.
6. Every run leaves a record in the repo: build commit on each device, exact commands, numbers with n, verdict, and what didn't match. Raw evidence gets zipped to a backup with checksums.
7. I read the results. The issue only closes once that record is on main.
Same setup covers acceptance of every new feature, not just bugfixes: the pass cases are written before the code.
Thinking about turning this into an open source skill where you plug in your own devices and test cases. Would you use it?
@FactoryAI @droid
Happy to announce that I joined the @FactoryAI Guild 😁 Looking forward to building with Droid, giving feedback and helping make the product better
Wow thats Crazy, when my limits are back ill test how fast fast really is 😂
Let’s go @DevinAI
Two exciting updates for SWE-2: we're extending the **free** promo for pro/max/teams users to the end of the month and we're introducing SWE-2 Lightning as a paid option to run SWE-2 even faster.
Gave Gemini 4 Argon (high) the same prompt in @DevinAI and @droid: build a 3D anatomical heart
Devin: 8:11 min
Droid: ~19 min
I like Devins result more. Droid felt like it had latency issues all day today, so not sure if the long time comes from that, its not usual for me. Wanted to add Opus 5.5 too but in Droid it aborted after exactly 2407s of reasoning, 4th time today
planning more harness comparisons like this, let me know if you want more