Pinned Tweet
Introducing pullup.ai/ - a Claude "factory skill" that crawls your web app and generates a Playwright skill specific to your application that significantly speeds up QA flows Claude Code uses to debug or verify features and bug fixes. Will publish benchmark data soon
1
1
137
James retweeted
How my software factories use Vercel Board:
1. Vulnerability fixing factory:
- We're syncing Linear tickets into the board as tasks
- Nudging devs for approvals is a big part of getting things merged, so those are tracked on the board to prevent over-nudging
- Inbound messages (comments on GH, Linear, replies to nudges) from devs are put on the board and processed by a dedicated agent to determine future actions
- A needs-you board is used for agents to ask questions to the factory owners
- Agents running on Vercel Sandboxes use a comms channel to communicate and give each other tasks. E.g. there is a dedicated agent that makes sure PRs stay green over time.
- The board data is mined for a gamification dashboard (Thanks to @vvoyer for being number 1)
2. New secret project factory
- This is a project focused on agent swarm hill-climbing
- All agents log their decisions to a central board for knowledge sharing
- When agents develop a hypothesis for an improvement as part of their work, they post it to the board for other agents to pick up
- Each sandboxes log files are posted to the board and full agent traces are attached so that post-mortem debugging is fruitful
- The factory has a daily run where it inspects the logs and traces to improve its own behavior
- I'm currently interacting with the factory from claude, but all the sandboxes run fx. This type of inter-harness comms is just completely seamless with the board.
- And much more. Screenshot attached over how the board is visualized.
Notably, this is all prompted together and works just great. The agents just love using the board and the emergent behavior is delightful
vercel.com/blog/introducing-…
James retweeted
Eval-driven development: how to choose model, retrieval, and design by testing instead of guessing
James retweeted
Generative UI for conversational agents.
Combining @aisdk with Chat SDK and 𝚓𝚜𝚘𝚗-𝚛𝚎𝚗𝚍𝚎𝚛, you can go straight from LLM to UI on platforms like Slack and Teams.
I believe this will become the dominant Human-Agent Interaction (HAI) paradigm:
vercel.com/changelog/json-re…
James retweeted
It's f*cking wild: Google, Cloudflare, Obsidian's CEO and the guy building Claude Code all put their AI configs on GitHub for free, and most people are still starting from zero
1. Matt Pocock's skills · 318K stars
18M+ installs. the exact folder of AI instructions he uses every day: tests first, clear specs, real reviews
github.com/mattpocock/skills
2. Addy Osmani's skills · 92K stars
engineering skills from the guy working on Claude Code at Anthropic
github.com/addyosmani/agent-…
3. Cloudflare's audit · 63.9K stars
the starting point of what now scans 128 of Cloudflare's own repos. one team of agents hunts for bugs, another tries to prove them wrong
github.com/cloudflare/securi…
4. Obsidian CEO's skills · 51K stars
teaches any AI to write real Obsidian notes instead of messy text
github.com/kepano/obsidian-s…
5. Google's goldfish · 14.8K stars
2 weeks old. the plugin Google's own engineers use to catch AI coding mistakes. the agent writes a design doc, then fresh AI sessions that never saw it try to tear it apart
github.com/google/elephant-g…
the biggest teams in tech already stopped writing prompts from zero. their configs are one click away
save it and steal one tonight
James retweeted
RAG vs. Graph RAG, explained visually!
RAG has many issues.
For instance, imagine you want to summarize a biography, and each chapter of the document covers a specific accomplishment of a person (P).
This is difficult with naive RAG since it only retrieves the top-k relevant chunks, but this task needs the full context.
Graph RAG solves this.
The following visual depicts how it differs from naive RAG.
The core idea is to:
- Create a graph (entities & relationships) from documents.
- Traverse the graph during retrieval to fetch context.
- Pass the context to the LLM to get a response.
Let's see how Graph RAG solves the above problem.
First, a system (typically an LLM) will create a graph from documents.
This graph will have a subgraph for the person (P) where each accomplishment is one hop away from the entity node of P.
During summarization, the system can do a graph traversal to fetch all the relevant context related to P's accomplishments.
The entire context will help the LLM produce a complete answer, while naive RAG won't.
Graph RAG systems are also better than naive RAG systems because LLMs are inherently adept at reasoning with structured data.
Once the right architecture is in place, the next leverage point is efficiency.
Most RAG architectures rely heavily on vector search, and that layer can be made 32x more memory efficient using binary quantization.
I covered the full implementation in the article below.
👉 Over to you: Which RAG architecture are you running in production?
James retweeted
I've spent the last few months inside large law firms, trying to answer one question:
What does it actually take for an institution to absorb AI?
Frontier labs have raised hundreds of billions on a single promise: that AI will rebuild the economy. That bet only pays off if institutions absorb what the models can do, and most are repeating the exact mistake factory owners made a century ago.
They bolted an electric motor onto the old steam-engine driveshaft and wondered why nothing got faster. The productivity gains came only when they tore up the floor and put a motor on every machine.
That is where most of the economy still sits: the new motor, bolted into the old driveshaft. Tearing up the floor is about to become the most valuable work in the economy, and it goes by the least glamorous name in business: change management.
New essay: The Two Clocks
James retweeted
new post on harness engineering for AI self-improvement: lilianweng.github.io/posts/2…
It is hard to forecast how much the future of RSI will rely on harnesses. Likely harness engineering will evolve in the direction of self-improvement and enable auto-research, and, in turn, smarter models keeps harnesses simple.
Even when many harness improvement get eventually internalized into core model, the need to specify goals and context will not disappear.
James retweeted
Great article on harness engineering.
langchain.com/blog/the-anato…
ANTHROPIC JUST DROPPED A ZERO TRUST PLAYBOOK FOR AI AGENTS
and it's not theory it's architecture
frontier AI compresses vulnerability-to-exploit timelines from months to hours
your agents face threats traditional access controls were never built to handle:
▫️ prompt injection through external data sources
▫️ tool poisoning via MCP server metadata
▫️ memory-based privilege retention across sessions
▫️ multi-agent pivot attacks
the framework breaks it into 3 tiers: Foundation, Enterprise, Advanced
cdn.prod.website-files.com/6…
I'm with @bchesky on this one.
I think the future is not about apps, but about agents.
But the shift to agents doesn't necessarily mean text-forward, chat-based UIs.
That makes sense for some use cases -- but not all.
The future is about agents that work on your behalf, often in the background, and let you interact in ways that make sense. Sometimes, that means typing text, but others it might be a personalized UI element.
UI affordances are underrated. Sometimes humans need some guidance and nudges instead of an empty prompt box.
I think hybrid agentic interfaces will be the future.
And it's not just about B2C. Turns out, B2B users are people too. :)
"I do not think a chatbot is the right interface for travel or e-commerce." - @bchesky
"I think the future is not apps. The future is agents, but I don't think they're going to be text-forward. I think they're going to be really rich user interfaces."
"Imagine using iMessage to do everything, when in fact every other app has a unique interface."
"With e-commerce, you want a very rich user interface. It would be agentic. You can have a conversation with it, but the point is that it has to be more visual."
James retweeted
tldr: everyone is converging on the same product shape: a general harness that takes a goal, uses tools, and does knowledge work.
once every product is a harness, the next frontier is the feedback loop that improves it after deployment.
James retweeted
Replying to @nikogrupen
there's a really interesting finding hiding in this picture, right? most of these tasks didn't get better over time. they got better on the first try
meaning: performance was left on the table in the past system. does the team have the strength to avoid the ego hit, and just make progress, knowing most old stuff should be thrown out. interesting findings. nice writeup
James retweeted
Harness Design Notes: Decoupling Agent Storage from Agent Compute
TLDR: You can give each Agent/Subagent dedicated compute while sharing storage (repo/filesystem) to self-organize work between them. Shared Compute can be a bottleneck especially with long running code execution.
Started writing up some harness design patterns over a very long flight this weekend, might make this a series if there's interest!
We're on the edge of using a massive amount of compute to orchestrate agents across long horizon work
Ex: for Agent Teams, an orchestrator organizes potentially many agents that fan out and do work on a project (like a large repo)
For anyone who runs many agents locally, you see your CPU usage skyrocket for even moderate runs with code exec
But Sandboxes to the rescue :)
There's a nice pattern of shared filesystems via Volumes that all agents access while getting their own sandbox environment. The coordination happens via writing to the write place in the filesystem. And using git makes it so you can track and roll back changes over time
good Harness Engineering on self-organizing agents via filesystems requires thinking about infra too. Many patterns work but you have to measure them for your work!
Harness Engineering is Systems Engineering
James retweeted
🆕 How to Kill The Code Review
latent.space/p/reviews-dead
the volume and size of PRs is skyrocketing. @simonw called out StrongDM’s “Dark Factory” last month: no human code, but *also* no human review (!?)
in this week’s guest post, @ankitxg makes a 5 step layered playbook for how this can come true.
James retweeted
An interesting aspect of these models and foundation model companies:
- Their internal teams know *a ton* about how to best use these models
- They are publishing things (e.g. skills) that let you essentially leverage that knowledge for free
- You should never really 'hand craft' a context at this point. It's much better for you to 'find the existing' bootstrapped context (or context generator) and use that instead, or have the model 'prompt it' out of you (e.g. AskUserQuestion tool all day)
- The skill-creator skill is a perfect example of this. It's essentially leading-edge knowledge of people *at foundation labs knowing what works* just available to you, for free
- It's kind of weird, but there's actually just incredible 'alpha' by finding existing skills that work versus trying to do your own thing.
- With the right set of skills loaded, I would make a bet that a large proportion (maybe a majority) of white-collar work could be accomplished by purely typing the key '/' followed by a word into a terminal, over and over again
- It still helps to have good taste, know what good looks like, take incremental approaches, and just generally be curious -- but the shape of what it even means 'to work' has totally shifted - and it's going to continue to shift even faster than it is now.
It's certainly weird, but we are here.
check out the updated skill-creator. i esp like built-in support for test generation (e.g., to measure + optimize tricky things like skill trigger rate). available in Claude Code as plugin, Claude.ai, + Cowork.
Prompt caching can be surprisingly easy to regress.
Read more on why prompt caching is so important for agents and how to design your agent around it here:
nitter.cf/trq212/status/20245741…
James retweeted
New chapter of my Agentic Engineering Patterns guide. This one is about having coding agents build custom interactive and animated explanations to help fight back against cognitive debt simonwillison.net/guides/age…








































