Pinned Tweet
Sharing my latest initiative - the Nerfwatch Index
What is Nerfwatch Index?
Like many, I have often questioned whether models from the big AI labs are dialed back (i.e. 'nerfed') over time. I presume this might be to practically manage ongoing model delivery commitments within finite compute. Maybe it's just to reduce costs over time? Either way- it's challenging to prove, but we sometimes just sense things 'back off a bit' as we use them.
I'm contributing some of my subscription capacity ongoing for a while to test for this phenomena (started just before Sonnet 5.5 launched), and will see whether it is indeed true or not.
How does it work?
Nerfwatch Index tracks whether closed AI models maintain their original performance over time. Twice daily, models face the same 20 tests, with the outputs judged against strict quality standards (measurable / empirical), despite challenging constraints and conditions.
Link: nerfwatch.googz.com
Do you believe the big labs nerf their models over time?
A 9B from @SenseTime_AI that runs 24 of its 32 layers twice, so each token costs about 14.6B worth of compute.
Weights should fit a 10GB card at Q4. The question is what the second pass costs in speed, and whether it actually plans better than Ornith 1.5 9B or NeoHorse 1 9B, still the 9Bs to beat on my personal-agent tests.
Putting it on the RTX 3080 next. Results soon.
🔊New model just arrived for our 16GB RTX NVIDIA GPUs!
This 9B beats Qwen3.5-9B on seven of the eight benchmarks in its launch chart—and ties the eighth.
SenseNova Loop-9B-Preview from SenseTime is a new model with
🔓 Open weights
📜 Apache 2.0 license
🛠️ Permits commercial use, modification and redistribution under its terms
You can download, run and adapt on your own hardware. I have to create some quants.
The architecture is interesting.
It runs a block of layers twice using the same weights.
🧠 32 distinct decoder layers
🔄 56 layer executions
More computation without storing a second copy of those layers.
And SenseTime’s reported results go beyond beating the same-size Qwen.
Against Qwen3.5-27B
💼 WorkBuddy: 66.89 vs 60.53
⚙️ AutomationBench: 13.72 vs 10.17
📊 RaccoonPPT: 70.70 vs 59.08
There are limits. It trails the 27B on Terminal Bench and banking tasks. These are developer-reported results against selected models, not proof of “best 9B.”
Local inference instructions are available, but I want consumer-GPU tests showing memory use, latency and quality after quantization.
Open weights let us test that ourselves.
🔗 Link in ALT.
ALT https://huggingface.co/sensenova/SenseNova-Loop-9B-Preview
Results are in: LFM3-1.7B Uncensored vs the original @liquidai LFM3-1.7B on my personal-agent suite (RTX 3080, reasoning on, 51K context, same protocol, 3 runs each).
"No accuracy loss" holds for the simple stuff:
- Tool calls: 30/30 every run (original 29.7)
- Context recall: level, slightly up
But anything that needs judgement slipped:
- Light agent brief (A1): 8.7 to 8.0, mostly scope errors
- Change-of-plan recovery: down 12 points, the biggest drop
- PA Core total: 198.7 to 171.0 (out of 300)
Its best agent run scored lower than the original's worst.
Removing refusals seems to loosen more than refusals. For tool calls and chat it's fine. For a personal agent, I'd stick with the original.
I'll see if this holds up as well as the original on my personal-agent suite.
I just ran Mythos-V2-9B, another uncensored fine-tune, so I'm curious whether removing refusals costs a 1.7B anything. 'No accuracy loss' is a big claim at this size.
(After all, who doesn't want an assistant that never says no?!)
I'll see if this holds up as well as the original on my personal-agent suite.
I just ran Mythos-V2-9B, another uncensored fine-tune, so I'm curious whether removing refusals costs a 1.7B anything. 'No accuracy loss' is a big claim at this size.
(After all, who doesn't want an assistant that never says no?!)
LFM3-1.7B just got an uncensored version, and it's small enough to run on almost anything.
• Quants come in at 1.1 GB and 1.6 GB
• Refusals dropped from 61.7% to 0%
• Claims no accuracy loss
THE SAFETY LAYER LASTED LESS THAN A WEEK. CAN OPEN MODELS EVER STAY LOCKED DOWN?
Open Weights: huggingface.co/AtomicChat/LF…
Part-time builders matter! 😢
Day 25/
We are doubling all usage limits for all Pro 100/200/500 tiers and Business Premium, effective today.
We heard your kind feedback and won’t be reducing usage limits for Pro 200, so your Work and Codex usage will stay the same. We’re also going to continue offering extended limits for dots for now, and usage of your dot will still continue to be separate from your default usage.
And we'll of course also keep shipping better models and other efficiency improvements that increase your effective usage over time.
Only a Sith deals in absolutes.
"Local inference is a terrible allocation" sounds sensible until you look at what it assumes: that compute should be pooled and allocated centrally, and individuals owning their own is waste. That's a collectivist premise, and it runs against how people actually live and behave. We want our own tools, our own data, our own say.
And the practical case stands on its own:
- Cost over time: a GPU you already own costs nothing per token, while an always-on agent adds up fast in the cloud.
- Privacy: your calendar, inbox and notes never leave your machine.
- Always available: no outage, rate limit or connection needed.
2B models can now run real agent loops on a 10GB card. Cloud has its place. So does ownership.
sam: local inference is a terrible allocation of scarce resources
“a lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data centre
but from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time
you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment
somehow many people's intuition is the opposite of that”
Tested Mythos-V2-9B, the uncensored Qwen3.5-9B fine-tune, on my personal-agent suite (RTX 3080 10GB, Q4_K_M, reasoning off).
The thinking is there. A1 (light agentic) scored 9/10 on every run, ahead of NeoHorse 1 9B's 8.3.
Tool calls are where it slips: T1: 27/30 (NeoHorse and Ornith 1.5 both 30/30)
Poor tool calling leads to further issues:
- PA1 failed all 3 runs on malformed tool-call JSON
- PA2: 88 (vs NeoHorse 92, Ornith 88.4)
- PA4: 46.3 avg. Ahead of Ornith but behind NeoHorse.
It's built for storytelling and NPCs, and I'd happily use it there. For an agent that runs your day, NeoHorse is the 9B to beat right now.
Got a new version of NeoHorse coming soon @jarvis_pei ?
The most exciting ‘all rounder’ small model releases in recent weeks:
@liquidai LFM3 1.7B
@OpenBMB MiniCPM5 2B
@SparkLLM Spark X2.6 1.7B
@PrismML Bonsai 2 2B/4B
@NVIDIAAI Nemotron 3.5 Bolt 4B
These run on just about anything.
You have no excuse - build something amazing that packs local AI!
Any small-model wonders that I’m missing?
Small models are going through a real growth spurt right now.
On my personal-agent tests (PA Core, reasoning on, RTX 3080), three models at 2B and under now land within 21 points of each other, and rubbing shoulders with older 4B/9B weights.
Newest on the bench: Spark X2.6 1.7B from @SparkLLM
- PA total: 172.3/300
- Tool calls: 30/30 in every run
- Context recall: 9.3 at 8K, 8.7 at 16K, best of the three!
- Schedule and draft (PA2): 98 and 100 in two runs, one 54 pulled the mean to 84
- Daily organization (PA1): 35.7, the weak spot
Under 2B, three different strengths. @liquidai LFM3 plans, @SparkLLM Spark reads, @OpenBMB MiniCPM5 is the balanced one.
Leave thinking on. With it off, Spark is a much weaker model on these tasks.
Model and GGUFs: huggingface.co/XHToken/Spark…
3 runs, same protocol, Q8_0 quant, sampler and 51K context as the LFM3 and MiniCPM5 runs.
This is why I get excited with each new small model increasing in capability. Becoming super capable for everyday tasks and, on my old RTX3080, @SparkLLM Spark X2.6 1.7B absolutely screams! Instant responses..
Nice work @Tylerwtzhang and team!
Spark-X2.6-1.7B has landed: an open-source 1.7B model built for capable local agents.@SparkLLM
🤖 modelscope.ai/collections/XH…
🧠 Think only as much as the task needs: Switch between no-thinking, low-thinking, and high-thinking modes to balance answer quality, latency, and inference cost.
🛠️ Built for everyday agent work: Designed for tool use, multi-step planning, coding, and reasoning workflows in a compact local model.
⚡ Made for constrained hardware: The 1.7B scale makes it a practical starting point for private, responsive, resource-conscious local deployment.
🔧 Ready for the local AI stack: Use it with supported runtimes and quantized formats to fit different hardware and serving setups.
📦 Open to adapt: Start with the chat model for agent workflows, or use the Base checkpoint for continued training and domain customization.
📜 Apache-2.0 License.
Great ChatGPT QoL feature - this circular dial. Especially for Plus users contending with 5H limits.
(Thanks @thsottiaux, but over the next few days please consider removing the 5H all together!)
Round two on small models for personal agents on the RTX3080, this time with reasoning on.
Same protocol for all three: 51K context, my PA Core agent tests, each model on its recommended sampler.
PA Core total /300:
- Qwen 3.5 4B: 206
- LFM3-1.7B: 193
- MiniCPM5 2B: 181
The numbers don't tell the whole story.
Where each one is stronger:
- Qwen 3.5 4B is the all-rounder. Best on multi-step agent tasks, perfect tool calling, best briefings. If you have the VRAM for a 4B, it's the pick for an assistant that plans and acts.
- LFM3-1.7B is the agent-workflow surprise. It nearly matches the 4B on schedule-and-draft tasks (95 vs 96) and finished all 18 agent loops. Great for a lightweight agent that runs tools and drafts things on small hardware. It's the slowest-responding model when reasoning is on, however.
- MiniCPM5 2B is the reader of the 2Bs. Strong at pulling the right details out of long personal context, solid briefings, and the fastest of the three with reasoning (12 sec response for A1 compared to LFM3's 33 sec). Great for summarising, recall and quick answers.
With reasoning off, LFM3 and MiniCPM5 are dead level. With it on, LFM3 pulls ahead on agent work and MiniCPM5 keeps the edge on context and speed.
Still not a Gemma 4 12B for my use case, but the gap keeps shrinking. Great work @LiquidAI_ & @OpenBMB!
(Background of testing in quoted-post comments. Thanks @Franzferdinan57 and @FreelanceMike90 for asking about this)
LFM3-1.7B from @liquidai on my personal-agent suite (RTX3080), reasoning off.
PA Core Compact /300:
- Gemma 4 12B: 234.7
- NeoHorse 1 4B: 172.7
- LFM3-1.7B: 164.3
- MiniCPM5 2B: 164.0
- Qwen 3.5 2B: 159.7
- Qwen 3.5 4B: 138.4
At 1.7B it's level with MiniCPM5 as the best 2B I've tested, ahead of Qwen 3.5 4B and right behind NeoHorse 4B.
Tool use was a perfect 30/30 every run. Schedule and draft (PA2) is its standout at 78.3.
Where it isn't there yet: it didn't finish a change-of-plan loop (PA4, 0/3), and it drifted on scope in A1 (7/10).
So it's not a 12B replacement for my use case yet, but it's excellent for its size. One day I hope a 1B can do what Gemma 4 12B does on these tests, and releases like this keep closing the gap.
Really nice work from the @liquidai team!
LFM3-1.7B from @liquidai on my personal-agent suite (RTX3080), reasoning off.
PA Core Compact /300:
- Gemma 4 12B: 234.7
- NeoHorse 1 4B: 172.7
- LFM3-1.7B: 164.3
- MiniCPM5 2B: 164.0
- Qwen 3.5 2B: 159.7
- Qwen 3.5 4B: 138.4
At 1.7B it's level with MiniCPM5 as the best 2B I've tested, ahead of Qwen 3.5 4B and right behind NeoHorse 4B.
Tool use was a perfect 30/30 every run. Schedule and draft (PA2) is its standout at 78.3.
Where it isn't there yet: it didn't finish a change-of-plan loop (PA4, 0/3), and it drifted on scope in A1 (7/10).
So it's not a 12B replacement for my use case yet, but it's excellent for its size. One day I hope a 1B can do what Gemma 4 12B does on these tests, and releases like this keep closing the gap.
Really nice work from the @liquidai team!
LFM3 just landed from @liquidai, and the 1.7B text model is going straight onto my personal agent test bench!
256K context, about 1GB at 4-bit, and pitched for agent work. Strong claims at this size.
Can it handle a real personal-agent loop (clean tool calls, recalling context, planning a day, and adapting when the plan changes)?
Running my PA Core suite on the 10GB RTX 3080 and 16GB M2 Air. Results soon.
Method linked below.
Reasoning off, temp 0, 20K context. PA scores keep partial credit for unfinished loops. C1 and speed left out because conditions differ. Directional only (not a lab-grade ranking). nitter.cf/PaulGug/status/2097943…
LFM3 just landed from @liquidai, and the 1.7B text model is going straight onto my personal agent test bench!
256K context, about 1GB at 4-bit, and pitched for agent work. Strong claims at this size.
Can it handle a real personal-agent loop (clean tool calls, recalling context, planning a day, and adapting when the plan changes)?
Running my PA Core suite on the 10GB RTX 3080 and 16GB M2 Air. Results soon.
Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.
> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding
> 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages.
> 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language.
> 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒
Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.
1/ 🧵
Fair challenge @Tylerwtzhang, so I re-ran it.
Spark X2.6 4B, reasoning off vs on, 3 runs per test. Same samplers, 20K context. Qwen 3.5 4B and NeoHorse 1 4B as peers, tested both ways too.
Spark X2.6 4B
T1 tool use: 27 → 30/30, every run
A1 and C1: already 10/10 with reasoning off
PA Core: 158.7 → 138.3 /300
Finished agent loops: 4/9 → 2/9
You were right about tool calls. They were well-formed and reasoning made them perfect.
The agent loops look like a planning issue rather than a thinking-budget one. 7 of 8 unfinished runs used all 10 tool calls without submitting a final answer. Only 1 hit the 4,096 token cap.
Peers, PA Core /300:
Qwen 3.5 4B: 138.4 → 219.9 (PA2 26 → 90.3)
NeoHorse 1 4B: 172.7 → 189.7
Reasoning cost 1.8 to 10x more time per task across all three.
Happy to share the full traces and run notes if they'd help. Spark's tool calling is clean, and I'd love to see the next version close out those longer loops.
Method: nitter.cf/PaulGugAI/status/20979…
Reasoning on = medium effort, 4,096 output cap per turn (off: 512 to 900). 3 runs per test, so treat it as directional.
Added GPT-6.1 Terra to the Nerfwatch Index today. First run: 15/20, right inside GPT-5.6 Terra's usual 13 to 17 range. 😐
Nerfwatch isn't an intelligence test, it's a daily consistency check. But a "significant step up" should show up somewhere.
Same score at half the price. Anyone else finding it suspiciously similar?
ALT nerfwatch.googz.com
The "50% lower" is measured against GPT-5.6 Terra, which cost $2 in / $12 out. That's the same input and more output than Sol 6 and 6.1 ($2 / $10).
So 6.1 Terra at $1 / $5 isn't really a cut. It's the mid-tier going back to a mid-tier price.
Always check what a "discount" is measured against.
Prices from Oct 15 (see in comments):
