Sam Sgro (@samsgro) profile picture

Software Architect

Mountain View, CA
Joined February 2009
Photo by Sam Sgro (@samsgro)Photo by Sam Sgro (@samsgro)Photo by Sam Sgro (@samsgro)Photo by Sam Sgro (@samsgro)Photo by Sam Sgro (@samsgro)Photo by Sam Sgro (@samsgro)Photo by Sam Sgro (@samsgro)Photo by Sam Sgro (@samsgro)
Sam Sgro retweeted
I got access to Jev earlier today (thank you @hackgoofer). I have run ~5,000 requests so far, (which cost me around $2!), across classification, model routing, intent, steering, and many other things. tl;dr, Jev enables a new intelligent decision-making primitive, separate from deterministic code and LLM calls. This allows a class of decision-making that was neither suited to dumb, unintelligent code, nor to slow, expensive LLMs. It is super fast and cheap, and I think I will likely end up making a few Jev calls to every LLM call I make in my product. I think probably any company using LLM requests today can probably add a Jev call pre and/or post LLM calls to quite literally make their product much better for free, and have better tool calling behavior in many cases. I happened to have a personal benchmark for this as Iโ€™d been working on a ton of proactivity and classification tasks. I have been using the deepseek flash and more recently gpt 5.6 luna family of models as reasonably smart classifiers with low latency. Think questions like: - Did this conversation output contradict something theyโ€™ve mentioned before? - Should we send a followup message to this user based on our rules? - Itโ€™s been a few seconds of silence. Should we proactively send a message? In the past, Iโ€™ve been forced to write a bunch of what I call decision chains, mostly because an LLM classification call is very expensive in TIME (avg 4s), and less importantly can cost quite a bit if run on every message. Imagine a normal chat app. If you added 4s to every response to figure out if the response is good before sending it out, that ends up being pretty bad. So instead, I usually have to write some code that is a crude heuristic that runs quickly and decides whether to run the classifier. Obviously this sucks because you call the classifier many times that you donโ€™t want to, which makes your p95 bad, and you also miss cases with the heuristic, and you also have to manage all of these weird chains. With Jev, itโ€™s cheap enough, and fast enough (p50 ~150ms, p95 ~350ms in my testing!) that you can easily run it every turn. Heck you can reasonably run it before generation AND post generation, for any application that isnโ€™t realtime voice, and still feel snappy. But this is just one use case. Think: smarter model routing, better context packing, better responses, better observability for intent/tags/safety, smarter retries and so much more. By simply thinking about the inputs and outcomes you want to enforce, you can use Jev to supercharge most model calls and reduce bad user outcomes. The more โ€œquirksโ€ a model has, the more valuable it ends up being. Itโ€™s a bit weird and unintuitive using Jev. Generally, you want to decrease the # of questions you ask a classifier, or it makes more mistakes. In fact, you might want to ask your questions kind of in a compound way, because the reasoning happens in a shared scratchpad of sorts. Adding questions muddies the scratchpad and makes it take longer. With Jev, you feel incentivized to go the other way, to formulate your query as a set of independent questions. It doesnโ€™t feel like adding more questions decreases your performance on others. You can go a bit deeper to improve tool calls. Many model tools are things like turning on settings, or other things. You can easily improve models that are not very good at tool calling with Jev, by simply figuring out when to run them. You can do a pre-LLM call to figure out when to unfurl different tool definitions, in order to make your main LLM run better, you could run a background task with Jev + another LLM to reduce tool and context burden on your main LLM, and free it to be responsive. Iโ€™ve only scratched the surface of my testing but very excited!
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? Iโ€™ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev โ€ข 20-200x faster โ€ข 40-400x cheaper (w/ output tokens free) โ€ข Frontier composable intelligence optimized for decisions AFAICT the shortest path to AI-based economic revolution
127
179
44
3,067
502,309
Retro 2024 ๐ŸŽˆ Celebrated my daughter's 3rd birthday ๐ŸŽฎ 14 games finished: PS5 + Switch ๐Ÿ—ƒ๏ธ 1 lang learned: C3. Code a lot Scala 3 ๐Ÿง  learned a lot about AI: 290 POCs ๐Ÿ’ป 3,162 contributions in GitHub ๐Ÿ“‘ 18 blog posts ๐Ÿ“– 12 books read ๐Ÿ“˜ 1 book published: Continuous Modernization
1
2
157
Blog Post: Proprietary Systems and Distributed Monoliths diego-pacheco.blogspot.com/2โ€ฆ
1
67
Continuous Modernization! My paperback copy just arrived. Thanks for publishing us BPB. #Architecture #SOA #Microservices #Services #Modernization #Teams #Culture
1
3
174
It's great to see Continuous Modernization as Amazon's #1 hot release. ๐Ÿ“˜๐Ÿ”ฅ
1
3
110
Very happy to share that my next book is finally out! Continuous Modernization is all about how software development teams can improve their lives every single day. Modernize your software, change your way of working, and transform your company. amazon.com/dp/B0DHS63NQH/
4
83
After 7 months of hard work, my third book is finally out. Introducing: Continuous Modernization: The never-ending discipline of improving microservices, monoliths, distributed monoliths, individuals, and teams at scale. amazon.com/dp/B0DHS63NQH/refโ€ฆ
1
6
1,132
Use A Work Journal To Recover Focus Faster And Clarify Your Thoughts fev.al/posts/work-journal/
2
3
890
Very cool to see our book made available to ACM members!
Now available for ACM Members: "Principles of Software Architecture Modernization: Delivering Engineering Excellence With the Art of Fixing Microservices, Monoliths, and Distributed Monoliths," by @diego_pacheco & @samsgro.
2
96
Now available for ACM Members: "Principles of Software Architecture Modernization: Delivering Engineering Excellence With the Art of Fixing Microservices, Monoliths, and Distributed Monoliths," by @diego_pacheco & @samsgro.
1
5
1
16
617
The Bitter Lesson - general methods win in AI. incompleteideas.net/IncIdeasโ€ฆ
2
53
Blog post: Refactoring: Making sense of the invisible diego-pacheco.blogspot.com/2โ€ฆ
1
1
127
My first book is out! If you love thinking about distributed systems and software architecture, hopefully it has something to say to you.
#1 on Amazon - Distributed Systems & Computing book category ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ Principles of Software Architecture Modernization: Delivering Engineering Excellence with the art of fixing microservices, monoliths, and distributed monoliths. diego-pacheco.blogspot.com/2โ€ฆ
4
129
๐—›๐—ผ๐˜„ ๐—ง๐—ผ ๐——๐—ผ ๐—” ๐—š๐—ฟ๐—ฒ๐—ฎ๐˜ ๐—ฆ๐—ผ๐—ณ๐˜๐˜„๐—ฎ๐—ฟ๐—ฒ ๐——๐—ฒ๐˜€๐—ถ๐—ด๐—ป In his latest talk at Clojure Conj 2023., Rick Hickey, a creator of Clojure, talked about some great techniques to create a great software design in practice. Here are the most important ones: ๐Ÿญ. ๐—”๐—ฑ๐—ฑ ๐—ฎ ๐—ด๐—น๐—ผ๐˜€๐˜€๐—ฎ๐—ฟ๐˜† to your work where you define things and use it consistently ๐Ÿฎ. ๐—”๐˜€๐—ธ๐—ถ๐—ป๐—ด ๐—ด๐—ผ๐—ผ๐—ฑ ๐—พ๐˜‚๐—ฒ๐˜€๐˜๐—ถ๐—ผ๐—ป๐˜€ to formulate what you seek ๐Ÿฏ. ๐—ง๐—ต๐—ฒ ๐—ฆ๐—ผ๐—ฐ๐—ฟ๐—ฎ๐˜๐—ถ๐—ฐ ๐— ๐—ฒ๐˜๐—ต๐—ผ๐—ฑ - examine an idea and try to find the truth, questioning its underlying assumptions ๐Ÿฐ. ๐—ฅ๐—ฒ๐—ณ๐—น๐—ฒ๐—ฐ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—œ๐—ป๐—พ๐˜‚๐—ถ๐—ฟ๐˜† is a framework where we have the status and agenda of something passed through understanding (what do you know?) and activity (where are you at?) ๐Ÿฑ. ๐—ฃ๐—  ๐—ง๐—ผ๐—ฝ ๐—ฆ๐˜๐—ผ๐—ฟ๐˜†/๐˜๐—ถ๐—ฐ๐—ธ๐—ฒ๐˜ - design stories contribute to building a "top" story. All stories need to have: a title, description, problem statement, and approach. ๐Ÿฒ. ๐——๐—ฒ๐˜€๐—ถ๐—ด๐—ป ๐—ฃ๐—ต๐—ฎ๐˜€๐—ฒ๐˜€ are: Describe a situation, Diagnose problems, Delimit problem, Direction approach,ย Design tactics, and build it. He describes here in more detail how to implement every phase. Check the lecture in the comments. _______ If you like my posts, please follow me, @milan_milanovic, and hit the ๐Ÿ”” on my profile to get a notification for all my new posts. Grow with me ๐Ÿš€! #technology #softwareengineering #programming #techworldwithmilan #softwaredesign
8
68
367
79,058
How can you tell that something is written in Rust? . . . . . . . . . . Don't worry, they'll tell you
30
40
11
588
81,351