cofounder & CEO @PortexAI | evals & post-training @ https://nitter.cf/t.co/RXTUewHMGG
Joined December 2009
- Tweets1.9K
- Following725
- Followers27.1K
- Likes2.6K
Lucas Nuzzi retweeted
🚀 Terminal-Bench-Science is now featured in its first open-weight model release!
Kimi K3.1 Pro scores 38.1%, the strongest open-weight result on Terminal-Bench-Science to date.
Still, Terminal-Bench-Science shows BY FAR the biggest gap to SOTA (~30 points), a promising sign that it’s one of the most discriminative measures of frontier capability out there.
Introducing Kimi K3.1 Pro: our most capable 3T-class open-weight model.
🔹 1M-token context, 2.8T total / 104B active params (MoE), Multimodal.
🔹 Full-project coding: games in Unreal, Unity & Godot.
🔹 Excels at parametric 3D modeling in Blender & CAD.
🔹 More reliable at finance work: Accurate modeling with hallucination down by 65%.
🔹 Matching the strongest proprietary models across coding, agentic and knowledge benchmarks.
Kimi K3.1 Pro and K3.1 Pro Highspeed (2–3× faster) are now live on Kimi.ai, Kimi Work, Kimi Code, and the Kimi API.
Open weights by Nov 4, 2026.
🔗 API: platform.kimi.ai
🔗 Tech blog: kimi.ai/blog/kimi-k3-1-pro
black october is probably the biggest IP transfer in the history of mankind thus far. there’s literally no precedent
and whether you like it or not, all of this data will eventually make its way to open weight models
Lucas Nuzzi retweeted
There's a meta question here:
Why did all these get published yesterday? Is it reasonable to suggest that some people thought it was a "good day to bury bad news" - be one item in a list of shit, not the main news story.
nitter.cf/fiveoutofnine/status/2…
So basically, today PHP got two critical bugs that let anyone run code on your server without even logging in, which is bad, and then the details leaked before the patch was even out, which is worse, and it works on WordPress and MediaWiki and Magento and Joomla and Laravel, and Censys says like 1.5 million servers are running a version with no fix, so the NSA was like "take your system offline," and the FBI was like "take your system offline," and CISA and the UK and CERT-EU were all like "take your system offline," and then the Prime Minister of Japan, the actual Prime Minister, posted at 5:55 in the morning telling 48.8 million people to take their system offline, so everyone took their system offline, like the Dutch cybersecurity agency took its own website offline, and the Dutch parliament went offline, and the Port of Rotterdam went offline, and Finland and France and Denmark started pulling government sites, and Wikipedia went down, and Archive.org went down, and Asana and GOG and Something Awful went down, and the University of Vienna's webmail went down for some reason, and that's just PHP. Because GitLab also dropped a CVSS 10.0, which is the highest score they give out, for a bug that's been there since 2014, so that's every self-hosted GitLab for twelve years, and then LiteLLM 1.107.0 got hijacked and shipped a credential stealer and a remote access trojan in every single wheel and the Docker images, including latest, and you didn't even have to import it, just installing it was enough, and it's the second time this year, and then Railway paused every deployment on the entire platform because someone got into the environment variables, so now everyone has to rotate every secret they've ever stored there, and TiDB Cloud said someone was walking around inside with stolen credentials for basically a month and an extortion group says they took 10.6 terabytes, and Zeabur got breached for the second time in two months and is just shutting down, like the whole company, and Node.js patched remote code execution and a TLS bypass, and CISA says Plex is being actively exploited, and someone took over 300 Tor exit relays to steal SSH passwords, and SafePay ransomware has hit over 600 critical infrastructure orgs, and India's biggest bank filed a data loss incident, and Japan has been in an official state of emergency in cyberspace since October 9, which honestly explains the 5:55 AM post. Oh, and eMudhra, a certificate authority that's issued over 60 million certificates, whose entire job is proving websites are who they say they are, was supposed to be using private keys that are random numbers over a hundred digits long, and one of them was 4. Just 4. And the others were like 66 and 237. Anyway, it's Wednesday.
Coordinating and aligning agent swarms will be the biggest challenge of our existence.
If you run a swarm, you need to read this and put the right evals and observability in place.
i've seen dozens of darknet marketplaces for stolen datasets that indirectly flow to the labs through shady data brokers
so your app might see zero downtime while in the background someone is turning your data into an rl environment
i suspect this is happening at scale today..
Everyone using LLM judges or classifiers needs to read this.
“In practice, we recommend building a separate judge for each failure mode. With a narrower task and more specific guidance”
Lucas Nuzzi retweeted
New blog post 🎉: How To Use Decision Models For Evals
In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores.
Lots of cool interactive charts here.
it's time for most open-source projects to move away from git or it'll be hard to stay competitive against closed-source.
who's building oss alternatives to git?
Lucas Nuzzi retweeted
Today we’re releasing Muse Spark Open, the most capable US open-weight model to date.
Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning.
Key Model Specs:
→ 816B total / 42B active parameters
→ 1M token context window
→ Natively multimodal across text, image, video, audio
willing to bet simulations are the next frontier in eval infrastructure, and AI more broadly
good evals are still foundational for sims to be impactful and I highly recommend @HamelHusain's AI Evals course
Recently had the chance to interview @benhylak on how he is thinking about simulations in our AI Evals Course 😁
Simulations are incredibly useful for evals when you do them well. Ben discusses what he learned from implementing simulations for evals at several companies. Some fun topics came up, my favorite being "simulation awareness" - when models know they are in a simulation.
Below is a chapter summary:
Why input/output tests miss agent failures
Replay production traces to see what a change breaks
Recreate the data and tools around the agent
How agents detect simulations and change their behavior
Choose scenarios that test the change you made
Check answers, costs, and tool behavior across runs
Replay known failures to verify a fix
Common mistakes that make simulations misleading
Check that the simulation reproduces production behavior
When to start using simulations
You can also watch this on YT: youtube.com/watch?v=eLkESCLv…
Lucas Nuzzi retweeted
Succeeding in selling data/envs is hard without VC because the lab promises you $ Xm in a few months to give data for them right now. But you have to evaluate the data you make by testing on their frontier models (Fable, Astra are super pricey). And you have to pay the market rate API cost while they make 80% margin off the rollouts you’re running for them (isn’t this kind of funny?). And if the data is bad they don’t have to pay you for the data, so you’re fronting the costs while also putting in a ton of manpower.
That three-month period when you’re broke and at the whim of one of a few AI labs is not easy, but exchanging some equity for upfront capital for tokens and compute makes it slightly more doable. Respect to the founders who make it happen.
Lucas Nuzzi retweeted
We've been monitoring a decline in HuggingFace downloads across the top open LLMs we track for interconnects. It looks like there was a slight change in counting of downloads, as we saw a ~30% download rate drop for many top models, but this is continuing over time. What else do people think it may be?
being early can be painful.. we wrote about dataset valuation techniques over a year ago and many VCs thought we were crazy...
now it's literally the foundation of how labs acquire data and the fastest-growing subsector in AI
research.portexai.com/data-v…
Lucas Nuzzi retweeted
The next frontier lab featuring Terminal-Bench-Science, with a big performance jump from 12.4% (Gemini 3.8 Flash) to 57.6% by @GoogleDeepMind's brand-new Gemini 4 Argon - #3 on our leaderboard.
If you're a scientist, I'd be curious to hear about your experience using this model over the coming weeks!
eval automation is a huge deal. it's what allows agents to recursively self-improve.
and now you can use claude to build swarms of agents continuously finding cheaper ways to do their own jobs. anthropic’s example here improved performance and at roughly 1/5 the token cost.
Claude Code can help with eval design + hillclimbing.
check out this article i just published for details:
claude.dev/blog/automating-e…
i’ve been tracking tb-science since the early days and can say with confidence that it’s one of the best benchmarks to come out in 2026.
several of the tasks are great proxies for scientific creativity, which is notoriously hard to measure. huge congrats to @StevenDillmann and the team.
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains.
An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. v0.1 has 70 tasks. Claude Opus 5 solves only ~30%.
1/n 👇
in some sense spacex buying cursor gave it permission to cannibalize its own moat: the task-level data that made the company so valuable.
now the deal is done. cursor can put the crown jewels to work in routers & reward models instead of preserving them as acquisition leverage.
If the task you’re working on is not repeatable, your prompt should be the transcript of a ~20 min unhinged ramble
you’ll probably get 10x better responses than your usual “you are a professional.. etc”
One pattern I find useful for working with LLMs is a nice long ramble session. Sometimes the LLM needs more bits to understand what you're trying to achieve, but you're too lazy to type them. In these cases I like to lean back, switch to /voice and just ramble for like 10 minutes, total mess, anything goes, full stream of consciousness. Sometimes I declare it up top, something like "switching to speech recognition sorry for any typos...". Sometimes I turn it into a small interview of a few turns. But I find that the LLMs are somehow very good at reconstructing long incoherent rambles and often their echo of your own tangle of thoughts comes out quite a bit cleaner than what you started with. The result is that you improve the mind meld and have to correct things less from that point on.
farewell to the Amp Editor, my daily driver for a solid few months up until 5.3-codex
hard to overstate how much has changed since then
agent companies must now have 0 attachment to their products and embrace change in order to meet the current pace
