Open-Source AI Observability and Evaluation
Notebook or Container
Joined February 2023
- Tweets1K
- Following319
- Followers1.8K
- Likes1.5K
What does the OpenInference code donation mean for AI tracing?
@arizeai's @mikeldking and OTel contributor @ne_skazu discuss open source collaboration and GenAI semantic conventions on @Dynatrace's Observation Deck.
Watch the episode: youtube.com/watch?v=FSrw6Fnm…
Xandar Song (@axiomofjoy) from the Phoenix team introduces Harbor, an open source framework for testing agents on realistic tasks with real data in isolated sandboxes.
Give the agent a task, run it in a fresh environment, then score what it changed after it finishes.
He shows how we ran Harbor at scale on @daytonaio and used Phoenix to inspect the results.
A prompt change steered agents toward a read-only SQL tool for aggregate questions. SQL use went up, tool calls went down, and the task reward stayed high.
youtube.com/watch?v=kRJjSOYq…
Join us Nov 5 to see how Phoenix MCP lets agents query traces with SQL and code mode, with live demos.
We'll also share our eight-question benchmark, which showed roughly 17x lower average cost than fetching spans page by page.
Register here:
luma.com/arizeai-query-phoen…
Our Head of Open Source, @mikeldking talks through a problem every agent builder runs into: tool lists start out helpful, then start filling the context window.
Code mode gives the agent a sandbox where it can find the tools it needs, call them in code & return a smaller result.
I recently gave my first conference talk at @aiDotEngineer representing @arizeai . It was great getting to meet fellow AI nerds and talk about the things we are building. It genuinely gave me energy being around true innovators.
There's so much hype around self-improving agents and RL environments but I've only recently started to feel this is possible when I started playing with code mode and the harbor framework plugin with @ArizePhoenix. It's been a journey over the last few months and I think it fundamentally has changed how I think we should build agentic systems.
Have you setup code mode and RL / hill-climbing evals that work for you? What should I try next?
youtu.be/dAoOYFx7ZYw?si=6bKO…
Mikyo walks through how we built this into Phoenix, including shell access, sandboxed Python, SQL, and browser UI actions.
In Harbor evals, code mode used 43% fewer tokens and lowered cost by 31%.
youtube.com/watch?v=dAoOYFx7…
We love community contributions! Thank you, schlaepf, for helping make Phoenix's Traces chart faster to load.
The chart shows how many traces your app recorded over time and how many had errors. Their fix reduces the work needed to calculate those counts.
In their PostgreSQL test with about 2.3 million traces in one project, calculating a day's counts took 40 milliseconds instead of 3.9 seconds, with the same results.
The fix is merged and awaiting release
github.com/Arize-ai/phoenix/…
Learn how to investigate prompt injection in AI agents with Phoenix traces and evaluations, add guardrails to block unauthorized data access, and set up online evaluations and alerts in Arize AX.
arize.com/blog/indirect-prom…
We built an indirect prompt injection lab to test whether pages could trick an agent into accessing another customer's data. We used fictional customer records. The agent retrieved another customer's data in 28 of 100 runs. We traced and evaluated tool calls in @ArizePhoenix
We love community contributions. Thank you Saif Ali Shaik for writing the @scalekitinc Tracing guide for the Phoenix docs.
If your agent runs tools through Scalekit's AgentKit, you can now trace those calls in Phoenix. Learn more through the guide
arize.com/docs/phoenix/integ…
We love community contributions. Thank you @Kenny_V for the Atomic Agents Tracing guide.
Atomic Agents is a lightweight, modular Python framework for building agentic AI apps. See how to trace it in Phoenix:
arize.com/docs/phoenix/integ…
Decision models as evaluators
Phoenix evals (TS) can now run any classification evaluator on a decision model, like gpt-6-luna via OpenAI's Decisions API. The evaluator returns a label and a probability per option, no text generated. Each call is traced as a decision span.
Switching the model is a one-line change to your evaluator. The docs cover which models work, what comes back in the result, how it differs from a chat-model evaluator, and when to stick with a language model.
arize.com/docs/phoenix/sdk-a…
Phoenix now supports decision tracing, so you can debug the step before the tool call. You see which tool the decision model picked, which alternatives it considered, and how confident it was.
That's where a lot of tool-calling failures start.
We improved a shopping agent's first-tool match rate from 78% to 95% by putting a decision model in charge.
Unfortunately, it also produced 21 empty or unfinished replies in 300 turns, compared with zero for the original agent.
Apparently picking the right tool and finishing the job are different skills.
We wrote up how we did it. Decision spans in Phoenix, a few annotations, and a confidence threshold told us when to trust the router and when to let the chat model take over.
arize.com/blog/ai-agent-tool…
Why make an LLM read thousands of spans to count errors?
In our benchmark, querying traces with SQL + code mode cost ~17x less than fetching spans page by page.
See how Arize Phoenix MCP does it, live in Claude Code and Cursor.
Register here:
luma.com/arizeai-query-phoen…
Changing one tool description reduced Codex’s unnecessary skill loads from 51 to 0 and lowered the cost of our benchmark run from $11.85 to $8.34
The walkthrough shows how to benchmark AI agents and tools with Harbor and Arize Phoenix
We benchmarked 5 agent configurations across 18 tasks and 270 sandboxed runs with @ArizePhoenix.
Three configurations answered every task correctly. But Codex loaded an unnecessary skill in 51 of 54 attempts.
The benchmark score said everything was fine, but our agent traces told a different story.
Full walkthrough: arize.com/blog/how-to-test-a…
Check out the video walkthrough to see how we tested a tool description change with Harbor and Phoenix
youtu.be/OuGQ5zuWHuk?si=MH96…
You can now trace Claude Managed Agents in Arize Phoenix. Learn more: claude.com/resources/article…
nitter.cf/ClaudeDevs/status/2110…
And you can learn more about tracing Claude Managed Agents with Phoenix in our docs:
arize.com/docs/phoenix/cookb…
