Shreshth Kharbanda retweeted
We’re launching Inference Stack, one-click deploy with everything you need to own your inference
It ships with:
@vllm_project for deploying models
@huggingface for finding models
@_llm_d_ for deployment-level routing
@LiteLLM for inference
@grafana for metrics
Try it out here: inferencestack.io
Shreshth Kharbanda retweeted
You can now compare models × harnesses for your agent from the terminal with @LiteLLM Lens CLI
Building an agent means testing different combinations: Claude Agent SDK, OpenAI Agents SDK, LangGraph or your own custom agent, across different models.
That often means writing an eval loop, collecting results in a spreadsheet and repeating the work when you change a model or harness.
With Lens CLI, tell your local coding agent which combinations to evaluate. Run them against the same tasks and compare:
- Pass rates by model and harness
- Total cost and cost per successful task
- Cost vs. quality
- Latency
- Failure breakdowns: wrong answers, tool misuse and errors
Shreshth Kharbanda retweeted
Self host your own gateway and use providers you trust. You should know where your requests are going.
the OpenRouter situation for anyone curious:
-someone managed to access outputs sent to MiniMax
-researcher discovered this by sending a secret links
-40 to 130 hours later someone opened the secret links
-oop, this means that someone accessed your outputs!!
-AltasCloud/Novita/StreamLake were forwarding requests to MiniMax???? what the helly?
ugh.
you'll never hear about APIs without an API Gateway. if you're building AI products without an AI gateway, why?
bare minimum if you have agents in production
Introducing Lens Profiles: Turn Production Traffic Into Task-Level Model Evals
With Lens Profiles, you can easily:
- Discover tasks automatically: Understand what customers use your agents for without manually labeling every run.
- Find cost drivers: Attribute gateway-recorded spend to specific tasks and identify where optimization matters most.
- Compare model deployments: See request volume, average cost, and p95 latency for each task.
- Build relevant evals: Use task categories and real production runs to evaluate models against your customers’ actual workloads.
More details: docs.litellm.ai/blog/lens-pr…
Implementation PRs for reference:
Gateway cost attribution: github.com/BerriAI/lens/pull…
Traffic and spend by profile: github.com/BerriAI/lens/pull…
Automatic task discovery and classification: github.com/BerriAI/lens/pull…
Discovery configuration and similarity map: github.com/BerriAI/lens/pull…
Shreshth Kharbanda retweeted
As a developer, how do you define "improving your agent"?
Presumably at some point your agent harness has all the capabilities it needs to accomplish its goal.
What comes next?
Shreshth Kharbanda retweeted
If you sell an AI agent, every customer request costs you money.
Join us live on Friday to learn how you can manage costs with our new Customer Management portal.
Nov 20, 10:00 AM PT
Link to register: crowdcast.io/c/w9airc43f37a
Shreshth Kharbanda retweeted
We’re launching @LiteLLM for Startups.
We’re starting with @ycombinator companies and opening it up to other VC-backed AI startups with 5+ people.
This stack includes our Rust AI Gateway + Agent tracing + Evals.
You’ll get a direct Slack channel with our engineers. For teams in San Francisco, I’ll come to your office to help get set up.
If you’re building an AI company, we’d love to help ❤️
Shreshth Kharbanda retweeted
We’re launching agent evaluations in the @LiteLLM SDK for developers who want to compare models and harnesses on their own tasks.
Before you had to write the eval loop, create temporary workspaces and collect the results in a spreadsheet.
Now you can reuse your agent, define typed tasks with a success check, and call litellm.agent_eval(). You get a scorecard with pass rate, cost, tokens and duration.
athletes being replaced by ai next
Did you know how much prep professional players need to do?
- studying opponent’s plays
- tuning rackets based on humidity / temperature
- match data analysis
- meal and rest planning
Many require a team to help them prep.
So how tf does this relate to Devin Bots?
1
3
147
how are folks using devin token-efficiently? i absolutely love the UX but feel like it’s so easy to run up a huge tab compared to just multiple claude code subscriptions
recently been thinking, what's the point of unit tests anymore? if no one is reading and reviewing the assertions, they add no value. TDD was valuable when engineers started by asserting the intention then worked backwards.
now just build better guardrails (more code review, exhaustive QAing, it's all cheap)
At @costgraphai , we deleted most of our tests, 251k lines of it.
For months since then, we've not needed them. Velocity increased and QA time is faster, CI bill dropped significantly.
Synthesising test datasets is easier now, value for unit tests isn't there anymore. Skills can do CV (continuous verification) as part of a release workflow from engineers environments.
Maybe TDD is dying, probably dead. A new era begins.
Shreshth Kharbanda retweeted
I’ve tried a bunch of swarm frameworks and none of them work that well
We want to build a good swarm framework into our SDK at @LiteLLM
To do this, we want to work with devs that are actually building swarms in prod
If this is you please reply or dm me, would love to chat
Shreshth Kharbanda retweeted
Replying to @LiteLLM
@LiteLLM Global Control Plane API lets you manage gateways across hybrid clouds. This helps with fault tolerance & data residency.
Read more: litellm.ai/blog?post=global-…
Shreshth Kharbanda retweeted
You can now use your Cursor subscription through the @LiteLLM Gateway!
Read more below ↓


















