Backend/infra engineer turned founder @hatchet_dev | Databases, Go, distributed systems, durable execution, and open source
Joined May 2025
- Tweets403
- Following670
- Followers998
- Likes345
Pinned Tweet
we're open sourcing a library we’ve been working on at Hatchet called pgoutbox. github.com/hatchet-dev/pgout…
it implements a high-performance transactional outbox (specifically, for Go applications that use postgres via the pgx driver, but we'd like to make it available in other languages soon).
we’ve been running it in production for the past 3 months at Hatchet, processing tens of thousands of messages per second. benchmarks on my local laptop have it going > 200k messages / second.
what's a transactional outbox? it's a pattern for solving a lost-write problem in distributed systems. imagine a user signs up for your app and you’d like to create a new row in your users table and send them a welcome email. if your API crashes or restarts after you commit to the users table, but before sending a welcome email, the write to the email service is lost forever. so instead, you write the message to an outbox table as part of the same tx as the user write.
obviously there’s a lot of intersection with task queues built on Postgres — transactional enqueuing is one of the main benefits! we had a few requirements that make typical task queue libraries a poor fit.
first, it needs to support hundreds of thousands of different topics with exclusive consumers. this can get a bit tricky to scale due to MVCC on Postgres; if you attach a lease with a heartbeat to each topic, you’ll perform wayy too many updates and accumulate lots of dead tuples.
second, it needs to support immediate message deletion with configurable retention policies per topic. this is quite different from a durable queue, where you’d like to preserve messages for future replays and auditing.
third, it needed a solid scaling story and APIs built around batching.
here for feedback / questions!
this is basically the secret behind why @hatchet_dev aggressively uses range-based postgres partitioning. we've seen too many codebases using batched DELETEs in some kind of loop which can absolutely hammer your database and cause latency spikes and downtime.
we've even built some experimental features with very granular (hourly) partitioning to see how far we can push PG partitions. our use of partitioning this also led to us uncovering some bugs in Postgres, which one of our users reported upstream: postgresql.org/message-id/CA…
one of the challenges with heavy partitioning is ensuring that every query against partitioned tables can prune partitions during the planning phase. so a lot of our queries have a bit of a weird shape, were there's a min_inserted_at parameter that gets passed in as a bound on the time column (for example, min_inserted_at <= inserted_at).
this is less of an issue in more recent postgres versions, but we need compatibility back to postgres 15. excited for the day we're all on PG 18+.
biased but our @hatchet_dev engineering team is the strongest team of engineers I've worked with and we still do coding interviews. the difference is that we do them live and pair-program on a feature together. use claude code, cursor, etc - we don't care. we just want to see how you build and ship code.
the things we’re looking for in particular are:
- competency with tooling: how are you using/prompting your coding agent? Do you have an opinionated and productive setup?
- coding fundamentals: can’t really test this with coding output anymore, but asking people to explain or walk through their code has been a good signal.
- approach: get something working without sweating too many details right away (build first, then optimize)
I've been particularly surprised by engineers who are completely reliant on Claude Code and Cursor but absolutely horrible at using them. great engineers really care about their tools; there’s an element of specificity and clear thinking which translates directly to their use of coding agents.
for startups, the best signal by far is a paid trial period, but this likely excludes the most talented candidates who are already working impressive jobs.
The coding interview is dead, at least for my team at Anthropic. I've seen enough to know that it no longer provides signal. I removed it from loops this month.
I have a spicy, optimistic take - I think there's an opportunity for junior engineers: felixrieseberg.com/hunger-be…
most of the open-source rebuilds ("<x> in Rust") we're seeing in the software world are what I call "unproductive phantoms" - they don't really contribute anything to society except for wasting tokens and more likely than not, they'll get abandoned / out of sync with upstream.
but there are a standout few which I'd call "productive phantoms" which I think are providing actual, productive value and guidance to open-source maintainers. I think pgrust is one of these, given the number of upstream bug reports they've reported in Postgres. tsrs from @Cloudflare is another one, because it's built in close collaboration with the Typescript team. the common thread is collaboration with existing projects and maintainers rather than wholesale replacement. even if the projects fail, they'll still have contributed something to the ecosystem.
this got me curious, what would a productive phantom of @hatchet_dev look like? about a month ago, I started on trying to iterate on the Hatchet core internals with Fable using a 2ms time-to-start latency goal. it actually got there eventually, using a combination of a bunch of good ideas (some I directly prompted), a bunch of bad ideas (some of which I also directly prompted), some shortcuts, and a lot of extraneous code.
we've actually merged 8 of these ideas into the core Hatchet engine (after rewriting each into its own PR from scratch) with a few more coming! overall this was a success. not a "drop it and rebuild everything" type success, but lots of small improvements and iterations making the core engine better.
the latency metric isn't really the primary goal, the goal is efficiency and throughput, but if each tx is faster that generally helps with efficiency and throughput and it's much easier to measure locally.
after a week of security incidents in the software world, we thought it'd be a good time to release something we've been working on for a little bit: FIPS-compliant hatchet images!
these basically use Tink / Go crypto primitives under the hood but disable certain features which aren't FIPS-compliant.
(it turns out bcrypt isn't FIPS-compliant, so we had to change how passwords get hashed for open source users when running the -fips build)
anyway, this removes a bunch of friction for self-hosting hatchet in really sensitive environments, which we think is a big win.
it's been a week. to end it on a high note, we're releasing FIPS-compliant images for everyone (not gated behind some enterprise sales maze).
you're either in one of two camps: you've never heard of FIPS, or you deal with it every day. this is for the second camp.
docs: docs.hatchet.run/self-hostin…
for everyone piling on to this post after the railway incident: remember that heroku had a roughly equivalent security incident in april 2022, pre-agentic coding, where a number of user environments were exfiltrated (github repos and oauth tokens, in this case, not prod runtimes, but similar impact). I don't think there's much you can extrapolate from this post unless there's a very direct RCA which says otherwise and pins the cause directly on agentic coding habits.
these platforms are fundamentally honeypots and are going to be targeted, and securing untrusted user code from deployment -> production is brutal.
these incidents obviously shouldn't happen, but it has more to do with the practices / priorities of the team than whether or not they use AI to code.
some more thoughts on this "human coherence" take since yesterday:
- agentic review doesn't solve this coherence/elegance problem, because review evaluates whether the presented diff is free of bugs, and rarely evaluates whether it's a globally optimal strategy. and if you do rely on agentic review to decide whether this is the optimal approach, you're going to waste a ton of tokens going back and forth across agents, which is just token churn handing money to the labs. this decision needs to happen earlier in the dev process.
- forcing developers to write their own PR descriptions is a great habit to ensure coherence. I've found myself opening PRs with the assistance of Claude Code where I can't distill _exactly_ what's important about the PR in my description, which forces me to go back to the code again.
- there are cases where it makes sense to trade off coherence in favor of velocity, it's ultimately a judgement call. the way that we think about it is that great software is produced by consistent, high-quality, high-velocity effort. the risk that you run with reduction of coherence is that the cost of fixing each bug or adding a new features grows exponentially with the complexity of your code (or, not even exponentially, any type of increase which is greater than the reduction in token cost is problematic). there's also the argument that "future models will get better," where you're implicitly making the bet that the rate of increase in model performance is greater than the rate of complexity in your codebase, which is not something I'd be comfortable betting on.
- our tiny human brains are really, really good at sensing that complexity is getting out of control when things become ugly, inelegant, or incoherent, and LLMs do not have this innate ability. they might in the future!
I'm an avid user of coding agents, but there's something I've been struggling to put into words which I think is encapsulated well by these screenshots. these are two PRs to fix the same bug: the first is a +627 / -291 line change, from Claude, the second is a +3 / -2 line change, from Matt, a human (we closed the former and merged the latter).
there's a certain coherence in a human's mental model of the codebase which makes some bugs trivial for humans and still really difficult (or token-expensive) for agents. this goes beyond just better memory of the codebase. humans will build a mental map and a probability matrix for which parts of the codebase are most likely to surface a particular bug, given the codebase's coherence, and start by tackling those before going another layer deeper.
without matt's contribution we might well have merged the +627 / -291 PR, in turn losing coherence, and becoming more token-dependent as a result. after you lose coherence it's game over.
i've found myself pair programming with agents on tiny PRs after a huge amount of independent research / agent swarms to tackle a given problem, and I think this is pretty effective; you can combine the things that agents are naturally good at (parallelization, verifiable problems) with your human coherence. often this results in 15-20 PRs, not all of which get merged.
I'm an avid user of coding agents, but there's something I've been struggling to put into words which I think is encapsulated well by these screenshots. these are two PRs to fix the same bug: the first is a +627 / -291 line change, from Claude, the second is a +3 / -2 line change, from Matt, a human (we closed the former and merged the latter).
there's a certain coherence in a human's mental model of the codebase which makes some bugs trivial for humans and still really difficult (or token-expensive) for agents. this goes beyond just better memory of the codebase. humans will build a mental map and a probability matrix for which parts of the codebase are most likely to surface a particular bug, given the codebase's coherence, and start by tackling those before going another layer deeper.
without matt's contribution we might well have merged the +627 / -291 PR, in turn losing coherence, and becoming more token-dependent as a result. after you lose coherence it's game over.
i've found myself pair programming with agents on tiny PRs after a huge amount of independent research / agent swarms to tackle a given problem, and I think this is pretty effective; you can combine the things that agents are naturally good at (parallelization, verifiable problems) with your human coherence. often this results in 15-20 PRs, not all of which get merged.
Our European takeover continues. We're so excited to have JB on the @hatchet_dev team!
please join us in welcoming jean-baptiste souvestre to the team! based in paris (ooh-lala), JB joins our growing engineering squad in europe. 🥖🇪🇺🍷
we asked him some questions to get to know him better, here's what he said:
➡️ light mode or dark mode?
🎤 what is a light mode?
➡️ we're told you're a cat man. tell us about your cats.
🎤 so i have 3 cats, kiki, chablis, and croche-pattes, all of whom are scoundrels in one way or the other. they love wet food, waking up at 4pm, and going to bed at 5.
➡️ how did you first come across hatchet?
🎤 i wanted to introduce hatchet to my last company because i liked the tech that much, but i figured it might just be more fun to work on it directly.
➡️ if i were in paris for one (1) day and wanted to get the best croissant, where would you suggest i go?
🎤 that's mamiche, in the 10th arrondissement, for sure.
➡️ what's something that surprised you in your first week here?
🎤 honestly can't decide between the tech maturity at such a young startup and the amount of custom frog emojis one can add to slack.
➡️ last, but not least: claude code or codex?
🎤 claude, but just because it was a french first name.
bienvenue à bord, JB! 🚀
Alexander Belanger retweeted
back in february, when we first released the @hatchet_dev CLI, we were pretty adamant that it was the CLI, and not an MCP, that would be the primary entrypoint for agents using Hatchet. and this has definitely been the case! but the rise of coding agents and the latest MCP spec are making MCPs much more usable across a wider range of services by supporting serverless request/response. and, frankly, we've been using MCP servers much more -- shoutout to @Posthog and @Figma.
so we're releasing a new and updated MCP which does everything you'd expect -- debugging runs, triggering workflows and runs, and inspecting configuration of workflows and tasks -- so you can use either the CLI or the MCP (or the regular SDKs) to accomplish what you need.
back in february, when we first released the @hatchet_dev CLI, we were pretty adamant that it was the CLI, and not an MCP, that would be the primary entrypoint for agents using Hatchet. and this has definitely been the case! but the rise of coding agents and the latest MCP spec are making MCPs much more usable across a wider range of services by supporting serverless request/response. and, frankly, we've been using MCP servers much more -- shoutout to @Posthog and @Figma.
so we're releasing a new and updated MCP which does everything you'd expect -- debugging runs, triggering workflows and runs, and inspecting configuration of workflows and tasks -- so you can use either the CLI or the MCP (or the regular SDKs) to accomplish what you need.
new: hatchet mcp
trigger, inspect, and replay runs from your coding agent
docs: docs.hatchet.run/reference/c…
Alexander Belanger retweeted
me, just now: hmm i wonder what accounts the @hatchet_dev account follows
the @hatchet_dev account:
happy friday, everyone! our low level systems person @vqtbl4q has been permitted to take over our socials for the day, i think this was a great idea
I think this is really exciting, but we've started just putting "durable" in front or behind everything without really having a precise definition. I think lots of people are confused about durable objects versus durable execution vs durable queues and statefulness and everything in between. here's an attempt to demystify things...
the first thing to understand is that AI agents are stateful by nature. when you start taking apart a coding agent like Pi or Claude Code, you’ll notice at least 3 aspects of state:
- filesystem/scratch space: Claude Code operates on files. It’s goal is to edit the filesystem to produce a diff.
- context: the sequence of tokens which have been fed to the LLM
- session: the history of individual tool calls along with session state (for example, whether the session is paused or resumed)
this shouldn't come as a huge surprise to anyone. when you run a coding agent locally, these three bits of state are also local and live on the local filesystem!
folks who are building agentic products aren’t usually building local agents. they're building agents which run as part of a software service, hosted somewhere on the cloud. even the most popular local coding agents also have a cloud agents or background agents product.
cloud agents need to run on a machine somewhere in the cloud (duh). but machines in the cloud aren’t like local machines. they sit behind many, many layers of virtualization, and the past 15 years of DevOps have taught us to treat our cloud machines like cattle, not pets. so we expect to run a fleet of machines, which should be resilient to random failures or interrupts and all manner of distributed systems problems.
Pi Durable is referencing this problem: "we need a harness that runs anywhere, can be reached from different surfaces, supports infinitely long conversations, survives catastrophic internal and external failures, and lets multiple humans steer the same agents"
this all comes down to agent statefulness.
0/
People of Pi: We've shipped Pi 1.0 with Pi Durable. Go make them yours.
earendil.com/posts/pi-1-0/
a certain class of agents are more effective when they can write to disk. coding agents are a big one, but we're seeing this more and more with general-purpose agents, too. I think of the process of writing to disk as building the "inner harness."
we'd like to preserve the state of this disk while we’re waiting for feedback, or our sessions are paused (unless your agent is able to one-shot every command, which is highly unlikely). suddenly, we need to think about how to get the state of the filesystem on one machine moved to a different machine.
remember when I mentioned that agents are like managing thousands of little databases? the natural extension to this analogy is to create incremental backups of disk as the agent makes progress.
but most filesystems don’t work this way. you'd need to mount a filesystem on each machine which is running an agent. many container runtimes barely support mounting the same disk in two places, let alone mounting many disks across many places. the disks also need to be virtualized, which can impact performance quite a bit.
the other naive solution is just to snapshot and archive the current state of the filesystem after each agent turn. There are two ways of doing this:
- Just use git. Each session turn is responsible for producing a commit, and resuming a session is just a git clone away. This breaks down for anything that’s not a coding agent, and becomes extremely slow and expensive (ever been hit with an AWS data transfer cost bill?) when running at scale.
- Just run tar and upload the archive to S3. Rolling this yourself is even slower than the git approach.
if you’re just getting started, and trying to get your MVP to production, I actually don’t think these are unreasonable approaches. they just don’t work at scale. it turns out that we need rather specialized tools to even begin solving this problem.
I’m particularly excited about how companies like @archildata are tackling this problem. they mount an “infinite,” S3-backed filesystem with strong RW consistency guarantees. but instead of each write going directly to S3, they put a bunch of fast disks close to your agents to make performance tractable. 3/
so, there's always an "outer harness" or "session manager," which is its own agent with its own statefulness requirements. in this bucket, you see durable workflows, durable objects, and Pi Durable (which can plug into anything).
and there's an "inner harness," which also has its own statefulness requirements. in this bucket, you see filesystems, sandbox drives / snapshotting, and S3 tarballs. and Pi Durable as well.