Pinned Tweet
Introducing Sterling: Inference from verified providers.
The model you’re paying for might not be the one answering you. Every API model is served by a provider. Some serve a smaller version. Some break tool calls.
Sterling tests all providers and routes to the verified ones.
A correctly served model does not necessarily mean a securely routed inference request.
The OpenRouter/MiniMax incident exposes a critical blind spot in multi-provider inference infrastructure: requests can pass through upstream providers, introducing additional data-handling and retention risks.
Model fidelity and routing provenance are two distinct verification challenges.
Knowing which model processed a request is only part of the picture.
Knowing where that request traveled, who had access to it, and whether privacy guarantees held across every hop matters just as much.
Verified inference needs to account for both.
The scariest agent failures don't throw errors. They execute successfully.
Imagine a refund agent.
The user requests a $50 refund. The model selects the correct function, `issue_refund`, but passes $500 as the amount.
The JSON is valid. The schema checks out. The payment processor executes the refund.
Every API in the chain reports success.
Except the agent just refunded 10x the requested amount.
This is the problem with treating tool-call validity as tool-call correctness.
There are three separate things to verify:
- Selection: Did the model choose the right function?
- Arguments: Do the values reflect the user's actual intent?
- Outcome: Did the action produce the intended result?
For state-changing actions, you also need authorization boundaries, idempotency, and reconciliation.
The hardest bugs aren't always the ones that crash. Sometimes they're the ones that complete.
Your agent's performance drops. Your application code hasn't changed.
Where do you start investigating?
One approach using Sterling:
1. Find the model your agent is using.
2. Compare its verification results across providers.
3. Filter by date to inspect historical performance.
4. Download the JSON reports.
5. Compare the results using your own scripts or an AI assistant.
If a provider's results deteriorated over that period, you've identified something worth investigating.
It doesn't automatically tell you why the change occurred. But it gives you evidence beyond API uptime and error logs.
Explore: sterling.mira.network/verifi…
Imagine debugging a regression caused by a deployment you never made.
1/Your prompts haven't changed
2/Your model ID is identical
3/The API still works
But the outputs have shifted.
This is one of the stranger problems with relying on externally hosted models: the provider controls a serving environment you can't fully inspect.
TrackLLM monitors black-box LLM endpoints for behavioral changes over time.
Its public monitoring data includes a detected change in January 2026 on a Crusoe-hosted DeepSeek endpoint, identified through shifts in token log probabilities.
That doesn't establish what caused the change. But it establishes a useful question for production teams:
If your application's behavior changes and your own code hasn't, what evidence do you have about the inference endpoint?
You can now download Sterling’s verification reports as JSON, CSV or PDF.
Every provider behind a model is re-tested every few hours, and the report keeps every run: how each one scored on each check, whether it got better or worse over time, and when its price or limits changed.
Pull any date range, or give it to your agent and ask which provider changed and when.
Explore the reports: sterling.mira.network/verifi…
A 95% success rate sounds pretty good.
Until your agent has to succeed ten times in a row.
Assuming ten independent steps, each with a 95% probability of success:
`P(all succeed) = 0.95^10 ≈ 59.9%`
So a workflow built from individually reliable steps can have a surprisingly low end-to-end success rate.
Real agents are more complicated. Failures may be correlated, retries can recover some steps, and one bad result can affect everything downstream.
But that's exactly why evaluating agent reliability only at the individual model-call level is insufficient.
The interesting metric is whether the workflow actually finishes its job.
Two inference endpoints score 85% on the same benchmark.
Are they serving the same model?
There's no way to establish that from the score alone.
A benchmark measures performance on a particular set of tasks. Model identity testing asks whether two endpoints produce equivalent output distributions.
Here's why that distinction matters. 🧵
At ICLR 2025, researchers introduced Model Equality Testing.
Instead of comparing aggregate benchmark scores, they collected response samples from API endpoints and reference models, then used statistical two-sample tests to compare their output distributions.
They examined 31 commercial Llama endpoints during summer 2024.
11 produced distributions statistically distinguishable from the reference weights.
That doesn't establish why the endpoints differed, or whether they still differ today.
But it demonstrates why benchmark scores alone are insufficient evidence of model equivalence.
Consider two models that answer 85 out of 100 benchmark questions correctly.
Their scores are identical.
But they might disagree on completely different questions, respond differently to tool-use instructions, or generate very different distributions of completions.
Aggregate accuracy compresses all of that into one number.
If you're evaluating model capability, benchmark scores are useful.
If you're evaluating whether an API behaves like a reference model, you need a different experiment.
Paper: proceedings.iclr.cc/paper_fi…
307
1,000 inference runs.
`temperature: 0`
80 distinct completions.
That's what Thinking Machines Lab observed when testing Qwen3-235B with greedy decoding.
The interesting culprit wasn't randomness in token sampling.
It was batch-dependent numerical behavior in the inference kernels.
Different server loads can change batching and numerical results, occasionally flipping token selection.
When the researchers used batch-invariant kernels, all 1,000 completions were identical.
A useful reminder that reproducibility is an inference-systems property, not just a sampling parameter.
thinkingmachines.ai/blog/def…
Here's a strange finding from long-context research:
You can make information harder for a model to retrieve without removing or changing it.
Just move it to the middle of a long prompt.
In *Lost in the Middle*, researchers found that several language models performed best when relevant information appeared near the beginning or end of their input.
Move that information toward the middle, and performance often deteriorated.
Same information. Different position. Different results.
The practical implication for long-context evaluations is interesting:
Don't just measure retrieval accuracy at 32K, 64K, or 128K tokens.
Measure it across positions within each context length.
A single long-context benchmark score can hide a surprisingly uneven reliability profile.
Same model. Same benchmark. Different API providers. Different results.
When Artificial Analysis tested gpt-oss-120b across inference providers in August 2025, the AIME 2025 results varied sharply between implementations.
Then several providers fixed their deployments, and their scores improved.
What happened? 🧵
Here's the interesting part.
After providers made implementation fixes, Artificial Analysis updated the benchmark results.
Several previously underperforming endpoints improved substantially.
The issue wasn't necessarily the underlying model's intelligence. The deployment itself was part of the problem.
This is why benchmarking a model and benchmarking an endpoint aren't the same thing.
1
2
426
If you're deploying through a third-party inference API, model selection is only half the decision.
You also need to evaluate how faithfully the provider serves it.
That's the problem Sterling is built around: independently testing provider behavior and using verification results to inform routing.
Explore the results: sterling.mira.network/verifi…
1
341
An inference endpoint isn't simply healthy or unhealthy.
It can serve the expected model but mishandle tool calls. Honor tool schemas but ignore request parameters. Accept a long prompt without reliably supporting the declared context.
That's why Sterling tests provider-model pairs across five dimensions:
1. Model identity
2. Tool-call behavior
3. Parameter compliance
4. Silent failures
5. Declared context support
Each dimension catches a different class of failure. 🧵
Declared Context Support
A provider advertising 128K context doesn't automatically establish that it can reliably process that context.
There are two separate questions:
• Does the endpoint accept inputs approaching its declared limit?
• Can the model actually use information throughout that context?
Boundary tests and long-context retrieval probes help expose different limitations.
Maximum context length ≠ effective context utilization.
1
274
These dimensions are independent.
A provider might preserve model behavior but mishandle tool calls. Another might support tools correctly while falling short on declared context.
That's why Sterling evaluates provider-model pairs across all five dimensions, publishes verification results, and uses those signals to inform routing.
Behavioral verification isn't cryptographic proof. It's additional evidence that traditional infrastructure monitoring doesn't provide.
Explore the results: sterling.mira.network/verifi…
1
274









