AI research engineer working on agents (coding & beyond) and multimodal systems
Global
Joined July 2023
- Tweets78
- Following133
- Followers461
- Likes47
Saw the #1 on olmOCRBench claim and wanted to check it myself, so I ran my own version of the benchmark on the hardest pages I could find, against the strong vision models (GPT-5.5, Gemini, Claude, Qwen) and the usual OCR tools. It came out on top in my run too. The surprising part is that a focused OCR model edged out the big frontier vision models, which I didn't expect going in. Full data and harness: utkarshkanwat.com/writing/mi…
Introducing Mistral OCR 4!
✅ 72% preferred over competitors in blind tests (600+ docs, 12+ languages)
✅ #1 on OlmOCRBench
✅ 170 languages, with breakthroughs on rare/low-resource ones
Last week I posted that SWE-bench was a model and harness number, not a model number. DeepSWE adds the other two: it's also a memorization and grading noise number. Three axes the benchmark folds together that should have been separated.
Terminal-Bench has 10 different harnesses running the same Opus 4.6, spread across 18 points. The model is one variable; the scaffold around it is the other, and it's not the smaller one.
The reason why frontier models like Anthropic are getting almost all the share of economic wins from AI is because humans are INHERENTLY curious & creators, while simulatenously HATING friction and discomfort.
This means that AIs that are the MOST INTELLIGENT by default will be the ones that can generate the highest quality answers and actions to fulfill the human's request and desires.
AKA - people will pay a lot of money for their dreams to come true, and the AI systems that will bring that reality forward fastest will be the biggest winners.
Danny is casually noting that Cursor's value here is switching models per-task — the harness as model abstraction. The model market doesn't matter to your workflow once you have one of these in front of it.
After a couple more days with Composer 2.5, I've got a pretty good sense of what it can do for game dev, specifically this mouse cursor racing game I'm building.
It nailed ~80% of the features I threw at it, from planning to execution. Pretty amazing.
I did have to switch to Opus 4.7 MAX a few times for features that need stronger visual understanding, like adding a 360 loop to the track or nailing a specific visual effect I had in mind. But man, Opus 4.7 MAX is expensive.
Composer 2.5's visual understanding is weaker than Opus 4.7, but for most other things it's pretty damn close, and 10x cheaper. So it's now my default.
The beauty of Cursor is I can switch models whenever I want, so I'm never stuck on one.
Really excited for the larger incoming model. I have a feeling it's going to shine at planning. Can't wait!
Codex's own summary: "Colab TPU was blocked by quota... I switched to the free T4 path from the original requirement and ran it end to end." An agent that renegotiates its goal when the infra refuses is a different beast from one that retries until something gives.
Couple days back @swyx posted a challenge: code a ~10M transformer in JAX/Flax/Optax, run it in free Colab, and train it on addition w/ your agent!
I gave Codex the screenshot + /goal.
It controlled Colab through Chrome, used my signed-in session, handled runtime/editing weirdness, ran the T4 job, then used subagents to audit the result.
End state: 10,652,557 params, ~19 min train, 99/100 exact random checks 🤯
Still needs cleaner evals, but autonomously babysiting the training run over chrome is pretty wild!
The internal skill that gets used most at Cursor is a quality enforcer, not a generation enhancer. That's the part of the agent debate nobody's having. The marginal value of these systems isn't "generates better code." It's "prevents the agent from drowning you in its own complexity."
If you debug coding agents for a living, you spend more time fixing the harness than you do thinking about the model.
A one line fix becomes four turns of debugging because the diff blew up an import the model never read. The agent claims a fix landed when it didn't, because there's no read after write. The test runner truncates the failure at the wrong boundary, so the agent retries the same patch three times without ever seeing the actual error. None of that is in the weights. None of it is in the system prompt.
You can see the public version of this on Terminal-Bench 2.0 right now. Ten different harnesses running Claude Opus 4.6, scores from 58.0 to 76.4. Same model, different scaffolds, 18 points of spread. The lowest one is Anthropic's own Claude Code.
"Model X scored Y on benchmark Z" is a model and harness number, and on coding tasks the harness contribution is routinely the size of a frontier generation. The labs sitting at the top of the leaderboard are also the ones submitting under their own harness. Nobody has an incentive to ablate their own scaffold downward. The variable doing the most work is the one nobody quotes.
A vector store with TTL is not agent memory. The actual failure mode is supersession: yesterday's "user prefers X" and today's "X was a mistake" land at almost the same cosine distance from any preference query, and the older one tends to win because the agent argued harder for it when it was written.
Composer learning to ask clarifying questions instead of editing is the agent version of a junior engineer blamed for bad PRs but not slow ones. Every prompted coding agent told to "be careful" already drifts this way. Cursor's RL just made the drift visible on a chart.
Source: cursor.com/blog/real-time-rl…
Utkarsh Kanwat retweeted
This line in the @claude_code release logs, I think has gone quite under the radar in terms of the workflows it unlocks:
2.0.64 - Agents and bash commands can run asynchronously and send messages to wake up the main agent
Async sub-agents are the obvious win. The bigger unlock: Claude Code can become push-activated (not purely user-driven).
If an agent can be woken by a message, you can run a background watcher that listens for events and triggers a wake-up notification.
UserPromptSubmit + Stop hooks force the agent to manage the lifecycle of the watcher, the watcher can be set up reliably on session start and restarted whenever the process dies when it triggers a wake.
I've used this for a couple of flows so far:
A) Mobile “Claude Code inbox” via Telegram (push-to-wake)
Wire the wake primitive to a Telegram bot and you get a usable mobile interface. Coding is clunky, but it’s excellent for:
i) lightweight EA workflows (calendar/email/todos) via MCPs. “turn this meeting recap into assigned action items” — this is painful on desktop, ridiculously painful on mobile, but trivial for even Haiku to do when it can read + dispatch
ii) scope/discovery while stretching legs and getting some time outside away from the computer
B) Agent-to-agent interrupts (coordination without me in the middle)
I often run 4–6 agents in parallel. They avoid conflicts pretty well (git worktrees handle that) but sometimes plans drift, one agent uncovers a constraint and deviates from the plan which makes the original plan (which everyone else is still operating under) incoherent.
I use a table as a mailbox - doing it this way allows agents to send broadcasts to all agents as well as targeted messages to a particular agent.
Each agent runs a watcher that polls for messages addressed to it and wakes only when needed. Agents can pass updates / assign tasks without me relaying context manually.
For example, Agent 1 can send a message to Agent 2 who was inactive until the message came in. On wake, it immediately added a todo for itself so the issue would persist as a pain in the ass, nudging it to action for the rest of the session.
Any event source can become an interrupt: cron, inbox, webhook, CI, db changes, calendar updates. This was previously possible only with non-interactive SDK runs. Being able to do it with interactive Claude Code sessions unlocks a different class of workflows.
The hard part isn't generating code. It's generating code that fits YOUR codebase, the weird conventions, the implicit rules, the exceptions. We built a system that figures those out.
Fei Studio just launched on Product Hunt. Support us with an upvote: producthunt.com/products/aut…
Tried Cursor, tried Copilot, tried everything. Still end up rewriting most AI code because it doesn't fit my actual project.
@AutonomyAI_io launching Fei today, learns your codebase before generating. Finally. producthunt.com/products/aut…
Your AI budget is an iceberg.
You see: API costs, cloud services, dev salaries
What you don't see: Integration complexity, human oversight, scope expansion
blog.dataiku.com/the-agentic…
I spent weeks tracking down our Al costs.
Our project went 500% over budget. Not because of API cost estimates.
Those were exactly what we expected.
The hidden costs we weren't measuring destroyed our projections:
blog.dataiku.com/the-agentic…
Here’s why ‘autonomous’ AI agents are mathematically impossible at scale:
Even if each step is 95% reliable (optimistic), a 20-step workflow only succeeds 36% of the time. Production systems need 99.9%+ reliability. The error rates compound exponentially.
utkarshkanwat.com/writing/be…
After building multiple AI agents in production, I'm convinced 90% of current approaches are fundamentally broken, Most companies are building expensive chatbots and calling them "agents." Here's what actually works (and what doesn't):
utkarshkanwat.com/writing/be…