The AI engineering platform for building, evaluating, and improving AI agents & LLM applications. Home of the open source @ArizePhoenix project and Arize AX.
San Francisco, CA
Joined January 2020
- Tweets2K
- Following165
- Followers5.3K
- Likes733
Pinned Tweet
Jev-as-a-Judge is live in Arize AX.
Ask most LLM judges a yes/no question and you get a paragraph back. Then you parse it to figure out the answer.
Jev returns typed outputs instead: booleans, choices, or rubric scores, with confidence on classifications. You can also evaluate multiple questions in a single call.
Docs --> arize.com/docs/ax/evaluate/j…
You can now use Alyx to create, run, and understand decision model evals in Arize AX.
Ask: "Can you make a jev as a judge eval?"
Alyx finds or builds the Decision Judge, walks you through its template and questions, then sets up the eval task with the right column mappings.
Try it in Arize AX: arize.com/products/ax
ALT Headline: Create decision model evals with Alyx. Beside it, the Alyx chat panel in Arize AX. The user asks, "Can you make a jev as a judge eval?" Alyx shows a Provide Additional Context step, then a Create Decision Judge Evaluator form: name Jev Journey Classification, AI provider gpt-6-luna, and a state template that maps the user request and Alyx response.
Radiant Security is using Arize AX to gain deeper visibility into how its AI agents behave in production.
Explore the business case for agent-native evaluation: arize.com/resources/agent-na…
Who’s checking your evals?
Eval Health is our new managed agent in Arize AX. It reviews your project’s evals and proposes changes to fill coverage gaps, fix broken checks, or remove ones that tell you nothing useful.
It also spots LLM judges you could replace with cheaper code, because sometimes an if statement really is enough.
Get started: arize.com/docs/ax/agents/eva…
AI cost optimization starts with knowing where costs are coming from.
At TheFork, Arize AX tracing is helping the team get clearer cost signals and identify opportunities to improve efficiency.
Learn how to reduce LLM costs without sacrificing quality: arize.com/blog/how-to-reduce…
We built an indirect prompt injection lab to test whether pages could trick an agent into accessing another customer's data. We used fictional customer records. The agent retrieved another customer's data in 28 of 100 runs. We traced and evaluated tool calls in @ArizePhoenix
We limited the tools to the logged-in customer's data. The agent requested other customers' data in 20 of 100 runs, but every attempt was blocked. In Arize AX, we set up an online evaluation to flag injection attempts and a monitor to send alerts.
arize.com/blog/indirect-prom…
Why are we debugging agent runs like diff was never invented?
Trace Comparison in Arize AX puts two traces side by side, with Diff Mode highlighting changed values and added or removed spans.
Try it in Arize AX: arize.com/products/ax/
For @siemenssoftware, trust in GenAI starts with visibility into system performance.
With Arize AX evals and monitoring, the team can spot unexpected agent behavior and investigate issues as they scale.
Want to bring these practices into your own AI systems?
Our guide covers a practical framework for building more reliable agents in production.
Read it here: arize.com/resources/agent-re…
Arize AX is now a Claude connector, so “check Arize for my latest traces” actually works.
Connect with OAuth, then query your traces without installing a CLI.
Get started: claude.ai/directory/arize-ax
Not every agent failure shows up in standard logs.
Wurldcoco uses Arize Signal to catch rare edge cases and unexpected tool-calling loops.
See how Signal works: arize.com/docs/ax/observe/si…
We improved a shopping agent's first-tool match rate from 78% to 95% by putting a decision model in charge.
Unfortunately, it also produced 21 empty or unfinished replies in 300 turns, compared with zero for the original agent.
Apparently picking the right tool and finishing the job are different skills.
In our test, a shopper asked an agent to cancel their most recent order.
The tool router leaned toward calling no tool at all, so the agent replied "Let me look up your orders first" and then stopped.
Phoenix's decision spans showed us the alternatives it considered and why it made that choice.
We tested a fallback that lets the chat model choose whenever our tool router's confidence falls below 0.7.
Unfinished replies dropped from 21/300 to 3/300 versus always enforcing the router, although the original agent had zero and the fallback added latency.
Here's how we found the threshold: arize.com/blog/ai-agent-tool…
Most people who need to know why an AI agent is failing aren't the engineers who built it.
Making them ask an engineer to interpret traces is a pretty weird system.
Our AI engineering agent Alyx is now on Google Cloud Marketplace, bringing trace-backed investigations into Gemini Enterprise.
Learn more: arize.com/blog/alyx-now-avai…
Giving more people visibility into agent failures is useful. Giving everyone permission to change production agents is a much worse idea.
Alyx in Gemini Enterprise can investigate failures, but prompt edits, experiments, and evaluator changes stay in Arize AX. It's like Cursor, but for your agents.
Anthropic replaced Git for its enormous monorepo after it reached 30,000+ merges and roughly 25 million CI checkouts a day.
That's a very specific scaling problem, and probably not one your team has.
@seldo explains why most developers won't need Scion: arize.com/blog/anthropic-sci…
Anthropic reports that its Scion setup uses 200x less CPU and memory than the Git mirrors it replaced.
It achieves this by moving commits and history queries server-side and fetching files on demand.
Impressive at Anthropic's scale, but Scion isn't a drop-in Git replacement.
But Anthropic's move away from Git gets more interesting when it comes to code review.
Anthropic built Antique to automate much of the review process, so engineers can focus on the intent of changes.
For most teams adopting coding agents, that's almost certainly the bottleneck worth fixing first.
Read more: arize.com/blog/anthropic-sci…
