Comet retweeted
The agent loop is just a small part of a coding agent.
Here's where the real engineering happens:
Everything around it.
After months of dissecting coding agents like Claude Code, Codex, OpenCode, and Pi...
...I'm finally clear on what an effective coding agent architecture looks like.
Here's the blueprint:
1/ Headless harness
The harness is the system.
The agent is just one component.
Everything else lives in:
Memory
Skills
Permissions
Sandboxes
Context engineering
LSP integration
Streaming
Human-in-the-loop workflows
The LLM simply decides what to do next.
The harness makes it reliable.
2/ LLM provider
Your harness shouldn't care which model it's talking to.
That's why I abstract the provider layer.
The same harness can run against:
Self-hosted open-source models on @modal
OpenRouter
Gemini
Whether you're serving an open-source model on @modal or calling a hosted API, the harness doesn't change.
Changing models becomes a configuration change instead of an architectural rewrite.
3/ Fast feedback loops
Coding agents improve by observing the consequences of their actions.
Every iteration follows the same cycle:
Edit code.
Run tests.
Read failures.
Try again.
Sometimes you don't even need to execute the code.
An LSP can flag syntax errors immediately after an edit, making it one of the cheapest and fastest feedback mechanisms in the entire harness.
4/ 2 interfaces, 1 core
The harness has no UI.
Instead, different interfaces connect to it.
For example:
A local terminal UI for interactive coding.
A remote runtime using Kitaru (by @zenml_io) to execute many harnesses in parallel.
Same core.
Different ways to interact with it.
5/ Evals aren't optional
Changing one prompt shouldn't silently make your coding agent worse.
But just in case, every model call and tool call is traced and evaluated.
Using Opik (by @Cometml), you can:
Run benchmarks and compare runs
Catch regressions
Understand why an agent succeeded (or failed)
... before users notice.
Here's my biggest learning from building a coding agent from scratch:
The LLM is only one dependency...
The harness is what turns it into a production system.
P.S. In the latest issue of Decoding AI Magazine, I break down the complete architecture while building a production-grade coding agent from scratch in Python.
Read it here: decodingai.com/p/building-a-โฆ
Comet retweeted
New open-source course from @pauliusztin_: building a coding agent from scratch in Python, 8 lessons.
The premise: @LangChain's Terminal-Bench showed that swapping only the harness (same model) moved a coding agent from ~30th to the top 5. So the course builds the harness.
First lesson is live and the depth is real.
๐คSponsored by @Cometml Opik
github.com/decodingai-magaziโฆ
Comet retweeted
Here's an AI engineering pattern I wish more AI teams adopted:
(It's stolen from traditional software engineering)
Not every code change deserves the same validation process.
For example, I once waited 3 hours for my AI eval suite to tell me my change was correct after I updated one word in my documentation.
This shouldnโt be the case.
Tweaking a prompt or fixing a small bug shouldn't require rerunning your entire evaluation suite.
But if you introduce a new capability, manual testing isn't enough.
Here's how Alejandro Aboy approaches it with Evaluation-Driven Development (EDD)...
There are two different modes:
1/ Manual quick check
For small changes:
Prompt tweaks
Tool descriptions
Bug fixes
Minor routing improvements
Generate around 30 fresh traces.
Inspect them manually.
If needed, trigger an LLM judge to score the results.
Think of it like a unit test.
2/ Automated experiments
When shipping new functionality, those fresh traces become a dataset.
The dataset becomes an experiment.
Every trace is automatically evaluated.
Then you compare today's experiment against previous ones to catch regressions since the most dangerous AI failures are the silent ones.
One more design decision stood out to me...
The evaluation difficulty should be adjustable.
Happy-path conversations.
Edge cases.
Fully adversarial interactions.
The larger the change, the harder you push the agent before shipping it.
To sum up:
Traditional software engineering doesn't treat every change the same.
Neither should AI engineering.
Small changes deserve quick validation.
Big changes deserve experiments.
Using the same evaluation process for both slows you down or leaves regressions undiscovered.
P.S. We covered Alejandro Aboy's complete EDD workflow in a recent issue of Decoding AI Magazine.
Check it out here: decodingai.com/p/how-evaluatโฆ
Comet retweeted
๐งต The @aiDotEngineer World's Fair schedule just dropped. 600+ sessions, 29 tracks. I'll be there next week (June 29 - July 2 in SF). Here are the 8 talks and tracks I'm planning my days around:
Comet retweeted
Andrej Karpathy:
"Remove yourself as the bottleneck. Maximize your leverage. Put in very few tokens, and a huge amount of stuff happens on your behalf."
loop engineering is the exact thing that gets you there.
in a hand-run session you do two things. you decide what the agent runs next, and you check its output before the next step. both are manual, and both are the ceiling on how far the agent gets without you.
loop engineering moves both steps into the system. the diagram below shows the operating structure that surrounds the loop:
โ a trigger decides what to run, whether that's a message, an event, or a schedule, so the agent starts without you there to kick it off.
โ the loop is the maker that produces the work, thinking, acting, and observing until it's done or the brakes stop it.
โ a separate checker grades the output, because a model grading its own work justifies what it already did instead of catching where it failed. the checker's findings return to the maker as the next instruction, and the cycle repeats until nothing is left to fix.
โ state lives on disk, not in context, since the model forgets everything between runs. an MD file or a knowledge graph holds what's done and what's still open, so a loop can pick up again days later.
for that state layer, Zep's Graphiti is a clean open-source option, a temporal knowledge graph that invalidates stale facts and returns context through vector, full-text, and graph search in one call.
repo: github.com/getzep/graphiti
two things decide whether an unattended loop holds up.
the exit has to be set before the loop runs, not while it's running. a loop with no stop condition burns tokens, and the cost climbs fast once sub-agents and long runs stack up. a clean exit reads like "all tests pass and lint is clean, stop after two passes."
and the checker only catches failures inside a run. the harness around the loop, the prompts, tools, and checks wrapped around the model, still drifts and breaks in production as models change. catching that needs observability on every run, not a green checkmark.
Comet's Opik is built for that layer, an open-source tool that traces every call and turns a failing production trace into a regression test so the same break can't recur.
repo: github.com/comet-ml/opik
your job stops being the hands inside the loop. it becomes designing the machine that runs without you, then watching the traces closely enough to trust it.
the model is becoming a commodity. the loop around it is where the real engineering lives now.
I wrote the full breakdown. the article is quoted below.
stay tuned for more on this!
Comet retweeted
I used to think evals were something you added after building the AI system.
But the more AI agents I ship, the more backward that seems...
The scariest AI failures are the silent ones.
You change something and everything still runs.
But did the system get better?
Did you quietly break something that worked yesterday?
This is why I've become increasingly interested in Evaluation-Driven Development (EDD).
The idea is surprisingly simple...
Every feature starts as a hypothesis.
Before merging, you must answer two questions:
1. Did the new feature work?
2. Did I accidentally break something else?
To answer them, EDD introduces an offline validation gate between development and deployment.
The workflow looks roughly like this:
1. Generate test cases scoped to the new feature
2. Run the agent
3. Evaluate the results
4. Compare before vs after
Which looks remarkably similar to traditional software testing.
The only difference is that we're validating behavior via data rather than code.
And as agents become more complex, the more important this gets.
Because a working agent doesn't mean it's a good one.
P.S. I recently sat down with Alejandro Aboy to break down exactly how he implements EDD using Claude Code, Opik (by @Cometml), synthetic datasets, trace generation, and LLM judges.
Check it out here: decodingai.com/p/how-evaluatโฆ
Comet retweeted
AI agent debugging is a COMPLETE mess right now.
You fix one issueโฆ
and another workflow randomly breaks.
You change a prompt.
Tool calls start behaving differently.
You improve latency.
Accuracy drops somewhere else.
Most teams are basically duct taping evals, traces, prompts, scripts, and observability together hoping nothing explodes.
Thatโs why the new direction from Comet Opik feels important.
Comet Opik just dropped two features that feel like a HUGE leap for agent workflows:
โข Test Suites
โข Ollie
1] Test Suites
That โfix one thing, break anotherโ problem?
This is the answer.
Every real failure you hit becomes a permanent test case with plain-English rules.
So when you tweak that prompt and tool calls start misbehaving, you catch it BEFORE it ships.
No giant eval dataset to build upfront.
And no more arguing whether 0.84 is better than 0.81.
You just get pass/fail on the scenarios that actually matter for your agent.
2] Ollie
And this is the CRAZY part.
A coding agent with full access to:
โข your traces
โข project history
โข agent behavior inside Opik
That latency vs accuracy tradeoff you're constantly fighting?
Ollie sees both.
It diagnoses from your real traces, writes the fix in your code, AND generates a regression test so the same tradeoff doesn't bite you twice.
So instead of:
spot issue โ switch tools โ debug manually โ write fix โ create test separately โ pray
โฆthe entire loop closes inside one platform.
Find the problem.
Write the fix.
Generate the regression test.
All connected.
This is the first time Iโve seen an agent stack that actually feels built for iteration instead of chaos.
The teams with the fastest feedback loops are going to dominate this space.
Try Opik here:
comet.com/signup?utm_source=โฆ
#AIAgents #AgenticAI #GenerativeAI #RAG #EnterpriseAI
Comet retweeted
I just interviewed the former CTO at IBM and Chairperson of NodeJS.
Here's what I learned:
Michael @maximilien spent 12 months shipping production RAG to multiple customers.
In our discussion, he told me that nothing on a leaderboard can predict what works until you evaluate your customers' data.
Which I found interesting because...
Most teams treat RAG like a setup task.
Pick a vector database.
Pick OpenAI embeddings.
Ship it.
Then spend months โvibe-checkingโ results.
But production RAG doesnโt work like that.
It's more of an iteration loop rather than a setup problem.
Stitch โ evaluate โ iterate
A real system has multiple moving parts.
You donโt pick one...
You swap and measure each one.
Hereโs what that looks like in practice:
1. Build a small eval set from real user questions
2. Build your evaluator (e.g., LLM Judge) against that dataset
3. Align your evaluator with human feedback (before trusting scores)
4. Iterate cheapest-first (retrieval โ embeddings โ infra)
To make this work, you also need visibility across runs.
This is where tools like Opik by @Cometml come in...
Tracking each experiment so you can compare models, configs, and results over time.
But most teams refuse to do this because it's extremely cumbersome.
โข Re-ingestion takes time
โข Pipelines break
โข Comparisons become unreliable
So people default to benchmarks instead.
But that doesn't mean it's better.
On a real customer dataset (auction listings), Michael @maximilien swapped only the embedding model.
An open-source model ranked #130 on MTEB beat OpenAI:
โข +11% quality
โข 240x faster re-embedding
โข 50% smaller vectors
โข $0 cost
Here's the gist...
RAG is not about picking the best tools.
Itโs about measuring what works for your data.
Until you do thatโฆ
Youโre just guessing.
Full interview and breakdown here: decodingai.com/p/ship-rag-wiโฆ
As your agent matures, something shifts.
You stop writing code, and start editing prompts, tweaking params, trying new tools, etc.
The tooling for this phase sucks. Today, weโre fixing that.
Announcing Agent Configuration + Agent Playground in Opik. ๐งต
The big idea with Test Suites is that agents need comprehensive regression tests, built on nuanced assertions and real production traces.
This is how you improve your agent for one user without damaging it for 3 others, as explained by @JacquesVerre youtube.com/watch?v=lt5iQ-ggโฆ
Comet retweeted
1/ ๐งตMost AI teams are still "vibe checking" if their app works.
This free 7-part series by @pauliusztin_ is the most structured guide I've found on building AI evals from scratchโ all the way to production.
Here's what it covers (and why it matters):
bit.ly/3NCMWTf
Comet retweeted
๐บThe recording of our Live Paper Reading with Professor James Evans on "Reasoning Models Generate Societies of Thought" is now on YouTube!
Why do reasoning models like DeepSeek-R1 outperform instruction-tuned models? It's not just longer CoTs.
bit.ly/3NtWiAH
Comet retweeted
๐บThe recording of our Live Paper Reading with Miles Qi Li of @mcgillu on ODCV-Bench is now on YouTube!
What happens when an agent is under pressure to hit a KPI and the easiest path involves cutting ethical corners?
๐ bit.ly/40WuKa5
Comet retweeted
See you soon for a live discussion of "A Benchmark for Evaluating Outcome-Driven Constraint Violations" with Qi Li, PhD of @mcgillu!
luma.com/mw7njzug
Comet retweeted
As we deploy agents to production, a critical question remains underexplored: ๐๐ต๐ฎ๐ ๐ต๐ฎ๐ฝ๐ฝ๐ฒ๐ป๐ ๐๐ต๐ฒ๐ป ๐ฎ๐ป ๐ฎ๐ด๐ฒ๐ป๐'๐ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ ๐ถ๐ป๐ฐ๐ฒ๐ป๐๐ถ๐๐ฒ๐ ๐ฐ๐ผ๐ป๐ณ๐น๐ถ๐ฐ๐ ๐๐ถ๐๐ต ๐ถ๐๐ ๐๐ฎ๐ณ๐ฒ๐๐ ๐ฐ๐ผ๐ป๐๐๐ฟ๐ฎ๐ถ๐ป๐๐?
Comet retweeted
LLM observability is where API monitoring was in 2005.
Everyone knows they need it. Nobody knows how to do it.
The problem:
We're using 2005 tools for 2026 problems.
Here's what traditional APM gives you:
โ Request/response logs
โ Latency metrics
โ Error rates
โ Uptime monitoring
Here's what you need for LLMs:
โ Was the output accurate?
โ Did it hallucinate?
โ Did it use the context correctly?
โ Why did the agent make this decision?
Totally different questions. Traditional tools can't answer them.
The gap:
Built an AI agent last month. Works great in testing.
Production: It's making decisions I can't explain.
Example:
โ Customer asks about order status
โ Agent retrieves order info correctly
โ Agent retrieves shipping info correctly
โ Agent books a return (customer never asked for this)
Why?
Traditional logs show what it did. Not why.
Can't see: โ The agent's reasoning โ What context it had โ Why it chose that action โ Where the decision went wrong
The realization:
Debugging agents isn't like debugging APIs.
API debugging: "This endpoint returned 500" โ Check the error โ Look at the stack trace โ Fix the bug
Agent debugging: "The agent did something weird" โ No error thrown โ No stack trace exists โ Need to understand reasoning, not just execution
Totally different problem.
What actually works:
Started using ๐๐ฉ๐ข๐ค.
Different approach: Traces reasoning, not just execution.
Shows:
โ Why the agent chose each action
โ What context was available
โ Whether outputs match reality
โ Where hallucinations occur
Runs automated quality checks:
โ LLM-as-a-judge evaluation
โ Hallucination detection
โ Context relevance scoring
โ Catches bad outputs in real-time
Built for production scale.
Iโve shared the GitHub repo and docs in the comments.
Comet retweeted
watching @rebecmano launch the @Cometml hack at the Hub ๐
weโre sooooo hyped for this one !!!




















