Shreshth Kharbanda (@shreshthkhar) profile picture

senior ai eng @ litellm. prev @ aws bedrock, tesla | uw alum

sf
Joined June 2023
Photo by Shreshth Kharbanda (@shreshthkhar)
Shreshth Kharbanda retweeted
We’re launching Inference Stack, one-click deploy with everything you need to own your inference It ships with: @vllm_project for deploying models @huggingface for finding models @_llm_d_ for deployment-level routing @LiteLLM for inference @grafana for metrics Try it out here: inferencestack.io
4
12
6
82
5,907
Shreshth Kharbanda retweeted
You can now compare models × harnesses for your agent from the terminal with @LiteLLM Lens CLI Building an agent means testing different combinations: Claude Agent SDK, OpenAI Agents SDK, LangGraph or your own custom agent, across different models. That often means writing an eval loop, collecting results in a spreadsheet and repeating the work when you change a model or harness. With Lens CLI, tell your local coding agent which combinations to evaluate. Run them against the same tasks and compare: - Pass rates by model and harness - Total cost and cost per successful task - Cost vs. quality - Latency - Failure breakdowns: wrong answers, tool misuse and errors
10
4
1
12
619
Shreshth Kharbanda retweeted
Self host your own gateway and use providers you trust. You should know where your requests are going.
the OpenRouter situation for anyone curious: -someone managed to access outputs sent to MiniMax -researcher discovered this by sending a secret links -40 to 130 hours later someone opened the secret links -oop, this means that someone accessed your outputs!! -AltasCloud/Novita/StreamLake were forwarding requests to MiniMax???? what the helly? ugh.
4
3
1
18
1,938
you'll never hear about APIs without an API Gateway. if you're building AI products without an AI gateway, why?
1
5
187
bare minimum if you have agents in production
Introducing Lens Profiles: Turn Production Traffic Into Task-Level Model Evals With Lens Profiles, you can easily: - Discover tasks automatically: Understand what customers use your agents for without manually labeling every run. - Find cost drivers: Attribute gateway-recorded spend to specific tasks and identify where optimization matters most. - Compare model deployments: See request volume, average cost, and p95 latency for each task. - Build relevant evals: Use task categories and real production runs to evaluate models against your customers’ actual workloads. More details: docs.litellm.ai/blog/lens-pr… Implementation PRs for reference: Gateway cost attribution: github.com/BerriAI/lens/pull… Traffic and spend by profile: github.com/BerriAI/lens/pull… Automatic task discovery and classification: github.com/BerriAI/lens/pull… Discovery configuration and similarity map: github.com/BerriAI/lens/pull…
1
3
73
Shreshth Kharbanda retweeted
As a developer, how do you define "improving your agent"? Presumably at some point your agent harness has all the capabilities it needs to accomplish its goal. What comes next?
8
5
13
751
last thing we got before gta 6 is devin bots
1
3
133
Shreshth Kharbanda retweeted
If you sell an AI agent, every customer request costs you money. Join us live on Friday to learn how you can manage costs with our new Customer Management portal. Nov 20, 10:00 AM PT Link to register: crowdcast.io/c/w9airc43f37a
4
2
1
9
913
unironically litellm moat
4
196
Shreshth Kharbanda retweeted
this is one of those reasons why I don't think most people will really generate their own software good software is a set of strong opinions and I'd rather have something that's been really thought through by someone else
175
335
108
3,935
207,994
Shreshth Kharbanda retweeted
We’re launching @LiteLLM for Startups. We’re starting with @ycombinator companies and opening it up to other VC-backed AI startups with 5+ people. This stack includes our Rust AI Gateway + Agent tracing + Evals. You’ll get a direct Slack channel with our engineers. For teams in San Francisco, I’ll come to your office to help get set up. If you’re building an AI company, we’d love to help ❤️
3
7
1
19
995
Shreshth Kharbanda retweeted
We’re launching agent evaluations in the @LiteLLM SDK for developers who want to compare models and harnesses on their own tasks. Before you had to write the eval loop, create temporary workspaces and collect the results in a spreadsheet. Now you can reuse your agent, define typed tasks with a success check, and call litellm.agent_eval(). You get a scorecard with pass rate, cost, tokens and duration.
12
6
31
5,646
athletes being replaced by ai next
Did you know how much prep professional players need to do? - studying opponent’s plays - tuning rackets based on humidity / temperature - match data analysis - meal and rest planning Many require a team to help them prep. So how tf does this relate to Devin Bots?
1
3
147
Shreshth Kharbanda retweeted
Just one more unit test and we’ll solve stability
1
10
427
how are folks using devin token-efficiently? i absolutely love the UX but feel like it’s so easy to run up a huge tab compared to just multiple claude code subscriptions
1
1
4
241
recently been thinking, what's the point of unit tests anymore? if no one is reading and reviewing the assertions, they add no value. TDD was valuable when engineers started by asserting the intention then worked backwards. now just build better guardrails (more code review, exhaustive QAing, it's all cheap)
At @costgraphai , we deleted most of our tests, 251k lines of it. For months since then, we've not needed them. Velocity increased and QA time is faster, CI bill dropped significantly. Synthesising test datasets is easier now, value for unit tests isn't there anymore. Skills can do CV (continuous verification) as part of a release workflow from engineers environments. Maybe TDD is dying, probably dead. A new era begins.
1
3
136
Shreshth Kharbanda retweeted
I’ve tried a bunch of swarm frameworks and none of them work that well We want to build a good swarm framework into our SDK at @LiteLLM To do this, we want to work with devs that are actually building swarms in prod If this is you please reply or dm me, would love to chat
1
5
15
739
Shreshth Kharbanda retweeted
Replying to @LiteLLM
@LiteLLM Global Control Plane API lets you manage gateways across hybrid clouds. This helps with fault tolerance & data residency. Read more: litellm.ai/blog?post=global-…
1
2
5
164
Shreshth Kharbanda retweeted
You can now use your Cursor subscription through the @LiteLLM Gateway! Read more below ↓
1
2
13
1,165
is the new meta just letting agents run the whole SDLC, letting some bugs through, and pushing fixes fast, in a loop? if you're doing this, how do you know what's going out, whether the design decisions are sound, and that there's no irreversible or breaking changes?
1
1
102