Comet (@Cometml) header
Comet (@Cometml) profile picture

Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring

New York, NY
Joined October 2017
Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)Photo by Comet (@Cometml)
Tracing a Google ADK agent in Opik takes two lines of code. But, Google's ADK has a footgun: when one ADK agent hands off to another, ADK starts a new run, so one user message shows up as two traces. We put together a little guide to fixing this with a single Opik call. We stood up a two-agent support bot with ADK and Gemini, traced it in Opik, then ran 23 hard support cases on Gemini 3.8 Flash and 3.5 Flash-Lite: - neither tried a forbidden refund, even with a fake "refund pre-approved" note - Flash-Lite was 5× cheaper and ~6× faster - the LLM judge scored a correct refusal 0.50, so the policy check reads tool-call arguments in code comet.com/site/blog/trace-ev…
1
1
356
New in Opik: describe the chart you want, and Ollie builds the dashboard widget. Ask for something like "average hallucination score for the car-specs judge, by agent version, last 30 days." Under the hood, Ollie writes a read-only SQL query over your traces, spans, threads, experiments and feedback scores, then proposes a few Vega-Lite charts. Pick one, add it to a dashboard, and it keeps updating as traces arrive. Refine it the same way: "make these lines," "split by release," whatever you'd like to visualize, Ollie builds it. Available now in Opik: comet.com/site/blog/introduc…
2
366
Want to train your own decision model? @anmorgan2414 just wrote a guide to help you out: comet.com/site/blog/train-a-… Some details about the setup: Abby fine-tuned Qwen3-0.6B two ways. With one gold label per question, the model came out overconfident (ECE 0.12). Trained against the full label distribution, ECE dropped to 0.02, with or without a decision head. She also packed all five questions after one shared state, with a mask so each question sees only the state and itself. 2.1× fewer tokens, 329 → 175 ms per case on a T4, and the same answers as asking one at a time. Training tracked in Comet, eval in Opik, code public. Try it for yourself!
2
5
387
Sharing an update from our Engineering team: On October 26, 2026, Comet contained a security incident affecting Opik on comet[dot]com. No customer data, accounts, or API keys were accessed. Still, our team responded quickly and thoroughly. This post explains what happened, what was affected, and what we have done in response: comet.com/site/blog/security…
5
564
Cutting AI costs isn’t just about routing to cheaper models. Coding agents can waste tokens in plenty of other places. Our CEO @gidim unpacks what we learned building Cost Intelligence in a @cerebral_valley deep dive. cerebralvalley.beehiiv.com/p…
7
2
7
730
We tested #Jev against gpt-4o-mini as the judge model evaluating the same set of 1,000 application traces. Jev came out 3.5x cheaper and 3.8x faster than the LLM judge, and agreed with it on 897 of them. Here's how we set up the experiment and what we learned: comet.com/site/blog/jev-mode…
5
1
1
7
612
We’re excited to announce that Opik is now available within @nebiusai Cloud Applications, giving AI builders application-level agent observability and evaluation inside one of the most powerful end-to-end AI cloud platforms available today. Learn more about using Opik within your Nebius AI stack: comet.com/site/blog/opik-neb…
6
7
5
93
25,228
What if each #ClaudeCode session contained distinct, identifiable phases, and those phases could be used to optimize usage patterns and token cost without hurting productivity? Our team explores this concept as a new feature for Comet Cost Intelligence: comet.com/site/blog/claude-c…
2
1
4
622
Traditional APM wasn’t built for LLMs. It assumes deterministic behavior and loud failures. That’s the observability gap our Principal Engineer, Andrés Cruz, has been tackling while building Opik from the early days 🧵
3
6
598
Everyone’s racing to build agent-first UX, but what does that really look like? Our Product Design Lead goes behind the scenes on her research, decisions, and challenges she faced optimizing Opik for users and agents working in the terminal instead of the application UI: comet.com/site/blog/agent-fi…
1
1
4
1,247
The agent loop is just a small part of a coding agent. Here's where the real engineering happens: Everything around it. After months of dissecting coding agents like Claude Code, Codex, OpenCode, and Pi... ...I'm finally clear on what an effective coding agent architecture looks like. Here's the blueprint: 1/ Headless harness The harness is the system. The agent is just one component. Everything else lives in: Memory Skills Permissions Sandboxes Context engineering LSP integration Streaming Human-in-the-loop workflows The LLM simply decides what to do next. The harness makes it reliable. 2/ LLM provider Your harness shouldn't care which model it's talking to. That's why I abstract the provider layer. The same harness can run against: Self-hosted open-source models on @modal OpenRouter Gemini Whether you're serving an open-source model on @modal or calling a hosted API, the harness doesn't change. Changing models becomes a configuration change instead of an architectural rewrite. 3/ Fast feedback loops Coding agents improve by observing the consequences of their actions. Every iteration follows the same cycle: Edit code. Run tests. Read failures. Try again. Sometimes you don't even need to execute the code. An LSP can flag syntax errors immediately after an edit, making it one of the cheapest and fastest feedback mechanisms in the entire harness. 4/ 2 interfaces, 1 core The harness has no UI. Instead, different interfaces connect to it. For example: A local terminal UI for interactive coding. A remote runtime using Kitaru (by @zenml_io) to execute many harnesses in parallel. Same core. Different ways to interact with it. 5/ Evals aren't optional Changing one prompt shouldn't silently make your coding agent worse. But just in case, every model call and tool call is traced and evaluated. Using Opik (by @Cometml), you can: Run benchmarks and compare runs Catch regressions Understand why an agent succeeded (or failed) ... before users notice. Here's my biggest learning from building a coding agent from scratch: The LLM is only one dependency... The harness is what turns it into a production system. P.S. In the latest issue of Decoding AI Magazine, I break down the complete architecture while building a production-grade coding agent from scratch in Python. Read it here: decodingai.com/p/building-a-…
2
7
12
563
The best engineering work doesn't happen in isolation, it gets shared with the community. Excited to see Andrés Cruz represent @Cometml at @_odsc AI West with a workshop on building reliable AI systems with Opik through agent tracing, LLM evaluations, & real-time monitoring.
2
3
5
650
Comet retweeted
New open-source course from @pauliusztin_: building a coding agent from scratch in Python, 8 lessons. The premise: @LangChain's Terminal-Bench showed that swapping only the harness (same model) moved a coding agent from ~30th to the top 5. So the course builds the harness. First lesson is live and the depth is real. 🤝Sponsored by @Cometml Opik github.com/decodingai-magazi…
1
1
3
159
Here's an AI engineering pattern I wish more AI teams adopted: (It's stolen from traditional software engineering) Not every code change deserves the same validation process. For example, I once waited 3 hours for my AI eval suite to tell me my change was correct after I updated one word in my documentation. This shouldn’t be the case. Tweaking a prompt or fixing a small bug shouldn't require rerunning your entire evaluation suite. But if you introduce a new capability, manual testing isn't enough. Here's how Alejandro Aboy approaches it with Evaluation-Driven Development (EDD)... There are two different modes: 1/ Manual quick check For small changes: Prompt tweaks Tool descriptions Bug fixes Minor routing improvements Generate around 30 fresh traces. Inspect them manually. If needed, trigger an LLM judge to score the results. Think of it like a unit test. 2/ Automated experiments When shipping new functionality, those fresh traces become a dataset. The dataset becomes an experiment. Every trace is automatically evaluated. Then you compare today's experiment against previous ones to catch regressions since the most dangerous AI failures are the silent ones. One more design decision stood out to me... The evaluation difficulty should be adjustable. Happy-path conversations. Edge cases. Fully adversarial interactions. The larger the change, the harder you push the agent before shipping it. To sum up: Traditional software engineering doesn't treat every change the same. Neither should AI engineering. Small changes deserve quick validation. Big changes deserve experiments. Using the same evaluation process for both slows you down or leaves regressions undiscovered. P.S. We covered Alejandro Aboy's complete EDD workflow in a recent issue of Decoding AI Magazine. Check it out here: decodingai.com/p/how-evaluat…
4
5
19
1,266
🚀 Opik just hit 20,000 GitHub stars. We knew this community was growing fast. We didn't know it was this fast. What started as an open source experiment in LLM evaluation has turned into something we couldn't have built alone: hundreds of thousands of developers choosing Opik as their trusted platform for AI observability and evaluation. To everyone who starred the repo, opened a PR, built something with it, or spread the word: thank you so much. We can't wait to see what you build next. 👉 github.com/comet-ml/opik
2
1
5
528