Comet (@Cometml) header
Comet (@Cometml) profile picture

Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring

New York, NY
Joined October 2017
Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)
The agent loop is just a small part of a coding agent. Here's where the real engineering happens: Everything around it. After months of dissecting coding agents like Claude Code, Codex, OpenCode, and Pi... ...I'm finally clear on what an effective coding agent architecture looks like. Here's the blueprint: 1/ Headless harness The harness is the system. The agent is just one component. Everything else lives in: Memory Skills Permissions Sandboxes Context engineering LSP integration Streaming Human-in-the-loop workflows The LLM simply decides what to do next. The harness makes it reliable. 2/ LLM provider Your harness shouldn't care which model it's talking to. That's why I abstract the provider layer. The same harness can run against: Self-hosted open-source models on @modal OpenRouter Gemini Whether you're serving an open-source model on @modal or calling a hosted API, the harness doesn't change. Changing models becomes a configuration change instead of an architectural rewrite. 3/ Fast feedback loops Coding agents improve by observing the consequences of their actions. Every iteration follows the same cycle: Edit code. Run tests. Read failures. Try again. Sometimes you don't even need to execute the code. An LSP can flag syntax errors immediately after an edit, making it one of the cheapest and fastest feedback mechanisms in the entire harness. 4/ 2 interfaces, 1 core The harness has no UI. Instead, different interfaces connect to it. For example: A local terminal UI for interactive coding. A remote runtime using Kitaru (by @zenml_io) to execute many harnesses in parallel. Same core. Different ways to interact with it. 5/ Evals aren't optional Changing one prompt shouldn't silently make your coding agent worse. But just in case, every model call and tool call is traced and evaluated. Using Opik (by @Cometml), you can: Run benchmarks and compare runs Catch regressions Understand why an agent succeeded (or failed) ... before users notice. Here's my biggest learning from building a coding agent from scratch: The LLM is only one dependency... The harness is what turns it into a production system. P.S. In the latest issue of Decoding AI Magazine, I break down the complete architecture while building a production-grade coding agent from scratch in Python. Read it here: decodingai.com/p/building-a-โ€ฆ
2
7
12
563
Comet retweeted
New open-source course from @pauliusztin_: building a coding agent from scratch in Python, 8 lessons. The premise: @LangChain's Terminal-Bench showed that swapping only the harness (same model) moved a coding agent from ~30th to the top 5. So the course builds the harness. First lesson is live and the depth is real. ๐ŸคSponsored by @Cometml Opik github.com/decodingai-magaziโ€ฆ
1
1
3
159
Here's an AI engineering pattern I wish more AI teams adopted: (It's stolen from traditional software engineering) Not every code change deserves the same validation process. For example, I once waited 3 hours for my AI eval suite to tell me my change was correct after I updated one word in my documentation. This shouldnโ€™t be the case. Tweaking a prompt or fixing a small bug shouldn't require rerunning your entire evaluation suite. But if you introduce a new capability, manual testing isn't enough. Here's how Alejandro Aboy approaches it with Evaluation-Driven Development (EDD)... There are two different modes: 1/ Manual quick check For small changes: Prompt tweaks Tool descriptions Bug fixes Minor routing improvements Generate around 30 fresh traces. Inspect them manually. If needed, trigger an LLM judge to score the results. Think of it like a unit test. 2/ Automated experiments When shipping new functionality, those fresh traces become a dataset. The dataset becomes an experiment. Every trace is automatically evaluated. Then you compare today's experiment against previous ones to catch regressions since the most dangerous AI failures are the silent ones. One more design decision stood out to me... The evaluation difficulty should be adjustable. Happy-path conversations. Edge cases. Fully adversarial interactions. The larger the change, the harder you push the agent before shipping it. To sum up: Traditional software engineering doesn't treat every change the same. Neither should AI engineering. Small changes deserve quick validation. Big changes deserve experiments. Using the same evaluation process for both slows you down or leaves regressions undiscovered. P.S. We covered Alejandro Aboy's complete EDD workflow in a recent issue of Decoding AI Magazine. Check it out here: decodingai.com/p/how-evaluatโ€ฆ
4
5
19
1,266
Comet retweeted
๐Ÿงต The @aiDotEngineer World's Fair schedule just dropped. 600+ sessions, 29 tracks. I'll be there next week (June 29 - July 2 in SF). Here are the 8 talks and tracks I'm planning my days around:
1
2
9
1,327
Andrej Karpathy: "Remove yourself as the bottleneck. Maximize your leverage. Put in very few tokens, and a huge amount of stuff happens on your behalf." loop engineering is the exact thing that gets you there. in a hand-run session you do two things. you decide what the agent runs next, and you check its output before the next step. both are manual, and both are the ceiling on how far the agent gets without you. loop engineering moves both steps into the system. the diagram below shows the operating structure that surrounds the loop: โ†’ a trigger decides what to run, whether that's a message, an event, or a schedule, so the agent starts without you there to kick it off. โ†’ the loop is the maker that produces the work, thinking, acting, and observing until it's done or the brakes stop it. โ†’ a separate checker grades the output, because a model grading its own work justifies what it already did instead of catching where it failed. the checker's findings return to the maker as the next instruction, and the cycle repeats until nothing is left to fix. โ†’ state lives on disk, not in context, since the model forgets everything between runs. an MD file or a knowledge graph holds what's done and what's still open, so a loop can pick up again days later. for that state layer, Zep's Graphiti is a clean open-source option, a temporal knowledge graph that invalidates stale facts and returns context through vector, full-text, and graph search in one call. repo: github.com/getzep/graphiti two things decide whether an unattended loop holds up. the exit has to be set before the loop runs, not while it's running. a loop with no stop condition burns tokens, and the cost climbs fast once sub-agents and long runs stack up. a clean exit reads like "all tests pass and lint is clean, stop after two passes." and the checker only catches failures inside a run. the harness around the loop, the prompts, tools, and checks wrapped around the model, still drifts and breaks in production as models change. catching that needs observability on every run, not a green checkmark. Comet's Opik is built for that layer, an open-source tool that traces every call and turns a failing production trace into a regression test so the same break can't recur. repo: github.com/comet-ml/opik your job stops being the hands inside the loop. it becomes designing the machine that runs without you, then watching the traces closely enough to trust it. the model is becoming a commodity. the loop around it is where the real engineering lives now. I wrote the full breakdown. the article is quoted below. stay tuned for more on this!
83
536
23
2,648
455,145
I used to think evals were something you added after building the AI system. But the more AI agents I ship, the more backward that seems... The scariest AI failures are the silent ones. You change something and everything still runs. But did the system get better? Did you quietly break something that worked yesterday? This is why I've become increasingly interested in Evaluation-Driven Development (EDD). The idea is surprisingly simple... Every feature starts as a hypothesis. Before merging, you must answer two questions: 1. Did the new feature work? 2. Did I accidentally break something else? To answer them, EDD introduces an offline validation gate between development and deployment. The workflow looks roughly like this: 1. Generate test cases scoped to the new feature 2. Run the agent 3. Evaluate the results 4. Compare before vs after Which looks remarkably similar to traditional software testing. The only difference is that we're validating behavior via data rather than code. And as agents become more complex, the more important this gets. Because a working agent doesn't mean it's a good one. P.S. I recently sat down with Alejandro Aboy to break down exactly how he implements EDD using Claude Code, Opik (by @Cometml), synthetic datasets, trace generation, and LLM judges. Check it out here: decodingai.com/p/how-evaluatโ€ฆ
6
2
11
542
AI agent debugging is a COMPLETE mess right now. You fix one issueโ€ฆ and another workflow randomly breaks. You change a prompt. Tool calls start behaving differently. You improve latency. Accuracy drops somewhere else. Most teams are basically duct taping evals, traces, prompts, scripts, and observability together hoping nothing explodes. Thatโ€™s why the new direction from Comet Opik feels important. Comet Opik just dropped two features that feel like a HUGE leap for agent workflows: โ€ข Test Suites โ€ข Ollie 1] Test Suites That โ€œfix one thing, break anotherโ€ problem? This is the answer. Every real failure you hit becomes a permanent test case with plain-English rules. So when you tweak that prompt and tool calls start misbehaving, you catch it BEFORE it ships. No giant eval dataset to build upfront. And no more arguing whether 0.84 is better than 0.81. You just get pass/fail on the scenarios that actually matter for your agent. 2] Ollie And this is the CRAZY part. A coding agent with full access to: โ€ข your traces โ€ข project history โ€ข agent behavior inside Opik That latency vs accuracy tradeoff you're constantly fighting? Ollie sees both. It diagnoses from your real traces, writes the fix in your code, AND generates a regression test so the same tradeoff doesn't bite you twice. So instead of: spot issue โ†’ switch tools โ†’ debug manually โ†’ write fix โ†’ create test separately โ†’ pray โ€ฆthe entire loop closes inside one platform. Find the problem. Write the fix. Generate the regression test. All connected. This is the first time Iโ€™ve seen an agent stack that actually feels built for iteration instead of chaos. The teams with the fastest feedback loops are going to dominate this space. Try Opik here: comet.com/signup?utm_source=โ€ฆ #AIAgents #AgenticAI #GenerativeAI #RAG #EnterpriseAI
13
9
46
933
I just interviewed the former CTO at IBM and Chairperson of NodeJS. Here's what I learned: Michael @maximilien spent 12 months shipping production RAG to multiple customers. In our discussion, he told me that nothing on a leaderboard can predict what works until you evaluate your customers' data. Which I found interesting because... Most teams treat RAG like a setup task. Pick a vector database. Pick OpenAI embeddings. Ship it. Then spend months โ€œvibe-checkingโ€ results. But production RAG doesnโ€™t work like that. It's more of an iteration loop rather than a setup problem. Stitch โ†’ evaluate โ†’ iterate A real system has multiple moving parts. You donโ€™t pick one... You swap and measure each one. Hereโ€™s what that looks like in practice: 1. Build a small eval set from real user questions 2. Build your evaluator (e.g., LLM Judge) against that dataset 3. Align your evaluator with human feedback (before trusting scores) 4. Iterate cheapest-first (retrieval โ†’ embeddings โ†’ infra) To make this work, you also need visibility across runs. This is where tools like Opik by @Cometml come in... Tracking each experiment so you can compare models, configs, and results over time. But most teams refuse to do this because it's extremely cumbersome. โ€ข Re-ingestion takes time โ€ข Pipelines break โ€ข Comparisons become unreliable So people default to benchmarks instead. But that doesn't mean it's better. On a real customer dataset (auction listings), Michael @maximilien swapped only the embedding model. An open-source model ranked #130 on MTEB beat OpenAI: โ€ข +11% quality โ€ข 240x faster re-embedding โ€ข 50% smaller vectors โ€ข $0 cost Here's the gist... RAG is not about picking the best tools. Itโ€™s about measuring what works for your data. Until you do thatโ€ฆ Youโ€™re just guessing. Full interview and breakdown here: decodingai.com/p/ship-rag-wiโ€ฆ
3
4
18
717
Comet retweeted
As your agent matures, something shifts. You stop writing code, and start editing prompts, tweaking params, trying new tools, etc. The tooling for this phase sucks. Today, weโ€™re fixing that. Announcing Agent Configuration + Agent Playground in Opik. ๐Ÿงต
3
9
4
28
28,888
Comet retweeted
Shared by a customer. Ollie just made their slack bot 52% faster and 98% cheaper. With test suites no regressions either
1
1
13
322
Comet retweeted
The big idea with Test Suites is that agents need comprehensive regression tests, built on nuanced assertions and real production traces. This is how you improve your agent for one user without damaging it for 3 others, as explained by @JacquesVerre youtube.com/watch?v=lt5iQ-ggโ€ฆ
1
3
19
592
Comet retweeted
1/ ๐ŸงตMost AI teams are still "vibe checking" if their app works. This free 7-part series by @pauliusztin_ is the most structured guide I've found on building AI evals from scratchโ†’ all the way to production. Here's what it covers (and why it matters): bit.ly/3NCMWTf
1
2
4
382
Comet retweeted
๐Ÿ“บThe recording of our Live Paper Reading with Professor James Evans on "Reasoning Models Generate Societies of Thought" is now on YouTube! Why do reasoning models like DeepSeek-R1 outperform instruction-tuned models? It's not just longer CoTs. bit.ly/3NtWiAH
1
1
2
414
Comet retweeted
๐Ÿ“บThe recording of our Live Paper Reading with Miles Qi Li of @mcgillu on ODCV-Bench is now on YouTube! What happens when an agent is under pressure to hit a KPI and the easiest path involves cutting ethical corners? ๐Ÿ”— bit.ly/40WuKa5
1
1
3
285
Comet retweeted
As we deploy agents to production, a critical question remains underexplored: ๐˜„๐—ต๐—ฎ๐˜ ๐—ต๐—ฎ๐—ฝ๐—ฝ๐—ฒ๐—ป๐˜€ ๐˜„๐—ต๐—ฒ๐—ป ๐—ฎ๐—ป ๐—ฎ๐—ด๐—ฒ๐—ป๐˜'๐˜€ ๐—ฝ๐—ฒ๐—ฟ๐—ณ๐—ผ๐—ฟ๐—บ๐—ฎ๐—ป๐—ฐ๐—ฒ ๐—ถ๐—ป๐—ฐ๐—ฒ๐—ป๐˜๐—ถ๐˜ƒ๐—ฒ๐˜€ ๐—ฐ๐—ผ๐—ป๐—ณ๐—น๐—ถ๐—ฐ๐˜ ๐˜„๐—ถ๐˜๐—ต ๐—ถ๐˜๐˜€ ๐˜€๐—ฎ๐—ณ๐—ฒ๐˜๐˜† ๐—ฐ๐—ผ๐—ป๐˜€๐˜๐—ฟ๐—ฎ๐—ถ๐—ป๐˜๐˜€?
1
1
1
418
LLM observability is where API monitoring was in 2005. Everyone knows they need it. Nobody knows how to do it. The problem: We're using 2005 tools for 2026 problems. Here's what traditional APM gives you: โ†’ Request/response logs โ†’ Latency metrics โ†’ Error rates โ†’ Uptime monitoring Here's what you need for LLMs: โ†’ Was the output accurate? โ†’ Did it hallucinate? โ†’ Did it use the context correctly? โ†’ Why did the agent make this decision? Totally different questions. Traditional tools can't answer them. The gap: Built an AI agent last month. Works great in testing. Production: It's making decisions I can't explain. Example: โ†’ Customer asks about order status โ†’ Agent retrieves order info correctly โ†’ Agent retrieves shipping info correctly โ†’ Agent books a return (customer never asked for this) Why? Traditional logs show what it did. Not why. Can't see: โ†’ The agent's reasoning โ†’ What context it had โ†’ Why it chose that action โ†’ Where the decision went wrong The realization: Debugging agents isn't like debugging APIs. API debugging: "This endpoint returned 500" โ†’ Check the error โ†’ Look at the stack trace โ†’ Fix the bug Agent debugging: "The agent did something weird" โ†’ No error thrown โ†’ No stack trace exists โ†’ Need to understand reasoning, not just execution Totally different problem. What actually works: Started using ๐Ž๐ฉ๐ข๐ค. Different approach: Traces reasoning, not just execution. Shows: โ†’ Why the agent chose each action โ†’ What context was available โ†’ Whether outputs match reality โ†’ Where hallucinations occur Runs automated quality checks: โ†’ LLM-as-a-judge evaluation โ†’ Hallucination detection โ†’ Context relevance scoring โ†’ Catches bad outputs in real-time Built for production scale. Iโ€™ve shared the GitHub repo and docs in the comments.
11
10
1
81
4,807
watching @rebecmano launch the @Cometml hack at the Hub ๐Ÿ˜Ž weโ€™re sooooo hyped for this one !!!
3
6
708
We have seen the Opik meme coin popping up here. It gave me a laugh, but just to be clear, we have nothing to do with it and are not involved in any way ๐Ÿ˜„ No fees are being redirected to โ€œmyโ€ wallet as they claim
27
3
5
27
9,253