Pinned Tweet
🚨 €135 FOR A COMPLETE LOCAL AI COMPUTER. NOT JUST THE GPU. THE ENTIRE MACHINE! 🤯
And the story is even crazier.
🖥️ HP Z220, Xeon E3-1270 v2, 32GB DDR3 — €50. Bought 5 years ago. Collecting dust in my garage ever since.
🔥 AMD RX 580 2048SP, 8GB VRAM — €49 delivered this week.
💾 256GB NVMe — €30 delivered this week.
🔧 PCIe-to-NVMe adapter — €6 on Amazon.
TOTAL: €135!
We installed Linux, built llama.cpp b12318 with Vulkan, and benchmarked five quantized AI models.
📊 REAL MEASURED RESULTS
Prefill / Decode (tokens/sec):
• Qwen2.5 3B Q4_K_M — 397 / 56.8
• Nemotron 3 Nano 4B Q4_K_M — 270 / 39.1
• Nemotron 3.5 Bolt E4B Q4_K_XL — 441 / 29.3
• Ternary Bonsai 2 9B Q2_0 — 172 / 26.9
• Ternary Bonsai 2 27B Q2_0 — 48 / 9.3
Yes, even a 27B model runs on this €135 machine! Part of it is offloaded to system RAM.
Test protocol: 512 prompt tokens, 128 generated tokens, maximum GPU offload, flash attention OFF. These are benchmark workloads, not maximum context-window measurements.
GPU peak: 120W, 46°C. No thermal throttling.
🤯 AND HERE’S THE SURPRISE:
We moved the same RX 580 from a much more powerful 12-core, 64GB quad-channel DDR3 workstation into this old 4-core, 32GB dual-channel machine.
Performance barely changed!
For these small models, the GPU matters far more than the expensive workstation around it.
No cloud. No subscriptions. Your hardware. Your data.
This is a real, affordable experimental AI laboratory. And possibly a useful starting point for practical local AI applications.
🎄 Here’s an idea: build one for your kids as a New Year’s gift. Their very own computer for experimenting with language models, learning AI, and building things!
And now imagine what models we might be running on just 8GB of VRAM with llama.cpp by New Year’s Eve.
THE LOCAL AI REVOLUTION MIGHT ALREADY BE COLLECTING DUST IN YOUR GARAGE. 🚀
🚨 Strata v0.2.17 by @coldniko is out! Strongly recommend updating.
🔹 Quality fix: disabled an optimization that could degrade responses.
🔹 21 engine crash/wrong-output issues fixed, plus tool-calling fixes.
🔹 Better RAM usage: 47 vs 40 GiB for experts on 64 GiB systems.
🔗 github.com/Niko1221/Strata/r…
1
14
2,091
New version is up! The update was focused on stability 👀
See more: github.com/Niko1221/Strata/r…
1
144
🚀 Strata vs optimized llama.cpp — RTX 5090, 24GB VRAM.
Same 54.5GB AI model. Same hardware.
⚡ Prefill: up to ~14× faster
🧠 8 agents: ~17× higher throughput
🔋 ~4× better GPU energy efficiency
We tested the latest Strata against a tuned llama.cpp.
Results & caveats 👇
3/4 📊 RESULTS — Optimization helped llama.cpp. A lot. But the remaining gap is substantial.
First, credit where it’s due.
The original llama.cpp configuration had no optimized GPU MoE expert cache. After enabling and tuning it:
📈 32K prefill: ~170 → 237 tokens/s (+39%)
📈 32K generation: ~30 → 41 tokens/s (+37%)
📈 4K generation: ~29 → 43 tokens/s (+49%)
✅ Eight agents with ~39K-token prompts each became possible, rather than failing because their requests exceeded the available per-slot context.
This is a meaningful improvement from configuration changes alone. Our same-version control confirmed that the gain was mainly due to expert-cache tuning, not simply a newer binary.
Now compare with Strata v0.1.62:
⚡ Cold prefill:
• ~39K tokens: Strata 2,866 vs llama.cpp 237 tok/s — 12.1×
• ~70–75K tokens: Strata 3,160 vs llama.cpp 225 tok/s — ~14×
⏱️ Time to first token:
• ~39K prompt: 13.7s vs 165.9s
• Long ~150–170K prompt: 47.6s vs 790.3s
The longer prompts have different token counts, so the second comparison is indicative rather than perfectly normalized.
🧠 Single-stream generation:
• 32K: Strata ~249 client-reported tok/s
• llama.cpp ~41 tok/s
But there is a major accounting caveat: Strata’s client-reported generation includes effects from speculative decoding, and its internal engine rates are materially lower. These numbers should not be treated as a clean 6× comparison of raw decode kernels.
👥 Eight-agent workload:
• Strata: 126.5 aggregate generated tok/s
• llama.cpp: 7.2–8.3 aggregate tok/s
That’s approximately 15–18× more delivered output in the measured workload.
However, Strata’s result benefits from prompt reuse, while llama.cpp’s eight-agent measurements include unique and shared-prefix variants. A fully cache-normalized eight-agent comparison was not performed.
🔋 Thirty-minute sustained operation:
• Strata: 1,285 successful requests
• Optimized llama.cpp: 270 successful requests
• Both: zero errors
GPU energy per generated token:
• Strata: 1.20 joules
• llama.cpp: 4.94 joules
That’s roughly 4.1× better measured GPU energy efficiency for Strata.
CPU and RAM energy consumption were not included, so this is not a whole-system power comparison.
🎯 Accuracy check: 5/5 injected fact-retrieval markers for both engines.
The conclusion is straightforward: tuning dramatically improved llama.cpp, but Strata remained substantially faster for this specific oversized-MoE workload.
1
172
4/4 🧠 WHAT DOES THIS ACTUALLY PROVE?
First, an important clarification: this is an EXPRESS benchmark, not an exhaustive scientific study.
We wanted to understand the scale of the performance differences under realistic hardware constraints, not spend weeks establishing rankings precise to the last percentage point.
Why?
Because local AI inference is evolving incredibly quickly.
Strata, llama.cpp and specialized forks can introduce major architectural improvements within days. A comprehensive benchmark may already be outdated by the time its analysis is finished.
Our approach is deliberately practical:
🔹 Use the latest available versions.
🔹 Apply meaningful optimizations.
🔹 Run identical or closely matched workloads.
🔹 Measure performance, context handling, concurrency and stability.
🔹 Publish commands, versions, logs, model hashes and known limitations.
🔹 Repeat when important new versions arrive.
What did we learn?
1️⃣ Expert-cache optimization matters enormously.
llama.cpp gained 30–50% in several workloads simply by changing how MoE experts were cached and scheduled.
2️⃣ Architectural differences remain significant.
Strata’s expert placement, prompt processing, speculation and conversation management deliver a large advantage on this particular machine and model.
3️⃣ Caching can change the entire picture.
llama.cpp repeated an identical ~39K-token prompt in just 0.168 seconds.
Its warm conversation follow-up took 1.53 seconds, close to Strata’s 1.43 seconds.
But a warm-cache result is not a cold-prefill benchmark. Prefix reuse is also vulnerable to cache eviction, and llama.cpp’s cached conversation state does not persist across restarts.
4️⃣ Performance rankings are workload-specific.
On long contexts of 128K tokens and above, the aggressively optimized llama.cpp configuration actually regressed in generation speed relative to its previous settings. There is no universally optimal cache size.
5️⃣ No single benchmark should be treated as a permanent verdict.
We are not claiming that llama.cpp is universally slow, or that Strata is universally superior.
On this system, with this 54.5GB MoE model and 24GB GPU, Strata is clearly the more effective engine for the workloads measured.
Yesterday we also compared Basalt, our RTX 50-series-specialized Strata fork. That is a separate comparison, with different trade-offs.
Our goal is not to crown a permanent winner. It’s to make powerful local AI practical, affordable and measurable — and to share what actually works.
📂 Complete reports, exact configurations and reproducibility details:
Original comparison:
github.com/ViktorIXAI/strata…
Optimized llama.cpp re-test:
github.com/ViktorIXAI/strata…
All measurements are from our own hardware. No vendor-supplied benchmark figures were substituted for our test results.
And yes — if either engine improves significantly next week, we’ll test again. 🚀
152
🚀 Basalt v0.5.0 is out!
This morning, we published our Strata vs. Basalt benchmarks. And they’re already outdated! 😅
What’s new:
⚡ 92 tok/s in conversations, 171 tok/s structured output
🔥 3,454 tok/s prefill
🧠 1M-token context, FP8 KV cache
🔧 NVFP4 experts, DFly2 speculative decoding
RTX 5090 + 5060 Ti.
All figures are from the release author’s benchmarks. We’ll also be using the new Basalt version and sharing our own test results.
github.com/jesdga95/basalt/r…
OpenAI Dots & GrokBot alternative running fully locally!! It uses PolyStrata (Strata engine) to serve and swap the models it needs and, of course being fast 😎
Check out more here: github.com/simplexai-labs/li…
The Strata vs. llama.cpp vs. Basalt comparison benchmarks are almost complete!
Next, we’ll test the models running entirely on CPUs, without any GPU acceleration. That should be interesting, especially since CPU-only local AI was exactly what I spent a lot of time experimenting with at the beginning of this year.
Curious to see how much things have improved since then! 🔥
This quoted post is unavailable.
We tested the latest version of Strata and compared it with Basalt, a fork of Strata specifically optimized for NVIDIA RTX 50-series (Blackwell) GPUs.
nitter.cf/uctive/status/21233238…
🚨 Strata vs Basalt — benchmark COMPLETE.
Same Qwen3.8-Flash-Next 54.5GB IQ3_S MoE. Same RTX 5090 Laptop (24GB).
⚡ Strata: multi-agent throughput & efficiency
🚀 Basalt: single-stream decode
🔥 Cached prefill: up to 275× faster
llama.cpp excluded: our agent misconfigured its MoE GPU cache and context slots. Retesting.
🧵👇
162
🚀 Strata v0.2.16 is OUT — a major update for local AI!
@coldniko continues to impress with the pace of development. Just two and a half days after the previous release, another major update has arrived, bringing significant improvements in performance, stability, and AI agent capabilities.
🔥 1. Strata’s own inference engine — another major milestone!
The transition to Strata’s own compute kernels began back in v0.2.14. Now, with v0.2.16, the developers report that the engines for NVIDIA, AMD, Intel Arc, and Apple Silicon run on Strata’s own code, without llama.cpp or ggml matrix kernels.
Strata’s own implementation is now used for image processing as well.
What’s particularly impressive is that despite this major architectural overhaul, the new engines produced identical output tokens to the previous release on most tested GPU configurations!
⚡ 2. Serious performance improvements!
• RTX 2080 Ti: approximately 23% faster processing of 26,000-token prompts compared with v0.2.15.
• Tesla V100: processing time for a similarly long prompt dropped from 17.46 to 15.07 seconds compared with engine v0.1.58.
• NVIDIA Blackwell: a massive CUDA 12 first-response delay was fixed — from 173 seconds down to just 0.34 seconds! That’s over a 500× reduction in latency in this specific test.
And here’s something I particularly appreciate: optimizations aren’t limited to the latest GPUs. Older NVIDIA generations are getting meaningful improvements too.
🧠 3. Critical bugs affecting answer accuracy have been fixed!
• Mac M1–M4: fixed a serious UD-Q4_K_XL quantization bug. Previously, some model layers were computed incorrectly, even though the resulting answers could sound perfectly convincing.
• Fixed an issue where UD-IQ1_M models could produce garbage text.
• Fixed a major Metal memory leak: memory growth across four test rounds dropped from 16.9 GB to just 0.23 GB.
• Fixed premature engine shutdowns when multiple chats were running concurrently on Mac.
🐧 4. Linux and Windows are now more reliable!
• Improved memory management when loading large models on Ubuntu. Previously, systemd-oomd could silently terminate Strata. In tests with other applications running, successful model loads improved from 0 out of 5 to 6 out of 7 attempts.
• Fixed a startup crash affecting certain Windows user accounts.
• Tesla P100 now downloads a prebuilt engine instead of spending around 6.5 minutes compiling it.
• Fixed the strata command in default PowerShell.
• Improved detection of multiple GPUs.
🤖 5. AI agents are becoming more capable and resilient!
• The chat now displays what the model is doing: thinking, calling tools, writing files.
• If an agent calls an unavailable tool, the turn no longer simply ends. Instead, the agent receives an error listing the available tools and can continue working. This succeeded in 20 out of 20 live tests.
• Improved local API compatibility.
• Initial hardware detection on an RTX 5070 dropped from approximately 23 seconds to just 2 seconds.
• Better offline installation, download management, and handling of models already stored locally.
🧪 6. Impressive testing!
✅ 14,976 inference requests.
✅ 23,919 tool calls.
✅ 21.4 hours of extended testing across five computers.
✅ Zero garbage answers or unplanned shutdowns during the long-running tests.
✅ 9,750 automated tests passed on Linux.
✅ 6,124 automated tests passed on Windows.
And all of this in a single release!
What interests me most isn’t any individual performance record. It’s the steady development of an independent inference stack, improvements in computational accuracy, and growing reliability for autonomous AI agents.
Local AI is gradually evolving from an experimental technology into something genuinely useful for solving real-world problems.
Massive respect to @coldniko and everyone contributing to the project!
🔗 Strata v0.2.16 — official release:
github.com/Niko1221/Strata/r…
#LocalAI #Strata #OpenSource #LLM #AI
We tested the latest version of Strata and compared it with Basalt, a fork of Strata specifically optimized for NVIDIA RTX 50-series (Blackwell) GPUs.
nitter.cf/uctive/status/21233238…
🚨 Strata vs Basalt — benchmark COMPLETE.
Same Qwen3.8-Flash-Next 54.5GB IQ3_S MoE. Same RTX 5090 Laptop (24GB).
⚡ Strata: multi-agent throughput & efficiency
🚀 Basalt: single-stream decode
🔥 Cached prefill: up to 275× faster
llama.cpp excluded: our agent misconfigured its MoE GPU cache and context slots. Retesting.
🧵👇
55
🚨 Strata vs Basalt — benchmark COMPLETE.
Same Qwen3.8-Flash-Next 54.5GB IQ3_S MoE. Same RTX 5090 Laptop (24GB).
⚡ Strata: multi-agent throughput & efficiency
🚀 Basalt: single-stream decode
🔥 Cached prefill: up to 275× faster
llama.cpp excluded: our agent misconfigured its MoE GPU cache and context slots. Retesting.
🧵👇
🤖 5/6 — MULTI-AGENT PERFORMANCE & ENERGY
Now the workload that matters most for my local AI experiments:
Multiple agents sharing the same GPU.
We tested simultaneous requests with approximately 39K-token prompts.
AGGREGATE THROUGHPUT
Concurrency → Strata / Basalt
2 requests:
18.8 / 31.0 tok/s
4 requests:
51.2 / 51.0 tok/s
8 requests:
126.5 / 101.6 tok/s
At 8 concurrent requests:
🔵 Strata: 126.5 aggregate tok/s
🟠 Basalt: 101.6 aggregate tok/s
Strata delivered approximately 25% higher aggregate throughput.
All eight requests completed successfully on both engines.
TOTAL EXECUTION TIME
Strata: 16.2 seconds.
Basalt: 20.2 seconds.
IMPORTANT LIMITATION
The workload rotated three prompt variants.
Some prompts repeated, allowing KV-cache reuse.
We also tested Strata with four completely unique ~39K-token prompts.
Results:
• 19.0 aggregate tok/s
• 32.13s mean TTFT
• 53.8s total wall time
This demonstrates how strongly the apparent concurrency advantage depends on cache reuse.
No claim of parallel 3,300 tok/s prefill per agent is justified.
STRATA’S KV-CACHE TELEMETRY
During the c=8 test:
• 26 requests tracked
• 14 requests reused context
• 672,733 reused tokens out of 1,310,176 prompt tokens
• 51.3% cumulative reuse
• 16 park operations
• 9 restores
• 8 evictions
Representative expert-cache hit rates reached approximately 90–93%.
PCIe accounted for approximately 0.1–0.3% of measured decode time in those requests.
The GPU was not spending most of its time waiting for expert transfers over PCIe.
30-MINUTE SUSTAINED LOAD
Both engines ran for approximately 1,802 seconds.
Completed requests:
🔵 Strata: 1,285
🟠 Basalt: 829
Zero errors for both.
Strata completed approximately 55% more requests.
GPU ENERGY PER OUTPUT TOKEN
Strata: 1.20 J/token
Basalt: 1.73 J/token
Strata used approximately 31% less measured GPU energy per answer token.
Average GPU power:
Strata: 164.4W
Basalt: 153.0W
Peak temperatures:
Strata: 76°C
Basalt: 74°C
No thermal throttling detected.
Both engines remained stable throughout the 30-minute load, with throughput drift below 2%.
GPU energy measurements exclude CPU, RAM and other system components.
CONCLUSION
Strata performed better in the tested sustained multi-agent workload.
Basalt maintained strong single-stream performance and stable operation.
For systems running multiple local agents continuously, throughput per watt and effective KV reuse may be more important than peak single-request generation speed.
1
1
324
🏁 6/6 — FINAL VERDICT: STRATA OR BASALT?
After comparing Strata 0.1.62 and Basalt 0.4.1 on the same oversized MoE model and hardware, I would not declare one universal winner.
They have different strengths.
🔵 CHOOSE STRATA FOR MULTI-AGENT SYSTEMS
My preferred choice when building autonomous local AI systems with several agents sharing one GPU.
Particularly for:
• Processing large documents
• Long-context prefill
• Multi-agent workflows
• Frequent conversation switching
• Reusing previously processed contexts
• Continuous workloads
• GPU energy efficiency
• Monitoring per-agent KV-cache behavior
MEASURED ADVANTAGES
~3,200–3,300 tok/s cold long-context prefill.
Successful processing of a ~189K-token prompt in the tested configuration.
126.5 aggregate tok/s with 8 concurrent requests in the shared-prompt workload.
1,285 completed requests during the 30-minute sustained test.
1.20 joules per output token, measured at the GPU.
The ability to inspect cache reuse, parked conversations, restores, evictions and per-request activity is especially valuable for developing autonomous agents.
For my particular multi-agent workload, Strata is the stronger candidate.
🟠 CHOOSE BASALT FOR INTERACTIVE ASSISTANTS
Basalt is especially interesting for a persistent primary AI assistant.
Particularly for:
• Single-stream interactive chat
• Fast answer generation
• Long-running conversations
• Persistent context across server restarts
• Disk-backed conversation caching
• Production assistants using established conversation histories
MEASURED ADVANTAGES
139–172 tok/s engine-reported single-stream decode.
Approximately 3,000 tok/s cold prefill in a measured 50K-token test.
Persistent conversation caching with a configured 20GiB disk budget.
Its existing production configuration remained stable under sustained load.
For interactive work where a single assistant responds quickly and must preserve conversation state, Basalt is a strong option.
⚖️ WHAT MATTERS MOST?
Different workload → different winner.
If the goal is maximum single-stream generation speed, Basalt has the advantage in these measurements.
If the goal is processing long prompts and running multiple agents efficiently, Strata performed better in the tested workloads.
If persistent conversation caching across server restarts is essential, Basalt has an architectural advantage in the tested configuration.
If detailed monitoring of per-agent KV reuse is important, Strata provides more directly useful telemetry.
ONE MORE IMPORTANT POINT
These results were obtained with one RTX 5090 Laptop, 24GB VRAM, 64GB RAM and one specific IQ3_S MoE model.
Other models, GPUs, cache states and configurations may produce different rankings.
This is not a universal benchmark of every possible deployment.
And llama.cpp?
We attempted the comparison, but our automated test configuration did not properly enable and tune its GPU MoE expert cache. The concurrency setup also allocated insufficient context per slot.
Rather than publish an unfair ranking, we excluded llama.cpp from the final comparison and initiated optimized retesting.
The failed experiments remain documented.
That is the point of open benchmarking: publish not only the impressive numbers, but also the mistakes, limitations and corrections.
FULL TECHNICAL REPORT
Exact configurations, raw measurements, methodology, cache experiments, failed tests and limitations:
github.com/ViktorIXAI/strata…
Open benchmarks. Real workloads. Reproducible results.
And a simple principle: measure first, conclude second.
1
3
312
This matches my experience exactly.
Whenever I hit a dead end with ChatGPT—or rather, realized we were going in circles without solving the problem—I always turned to Gemini. Not Grok, not Claude. Gemini.
I would bring it all the context and knowledge accumulated during those endless ChatGPT attempts, and Gemini would solve the problem faster than anyone else.
This happened repeatedly, so it wasn’t a one-off. I rarely used Gemini, but when I did, I came with all the accumulated experience and a very specific problem to solve.
The paradox was that, until today at least, actually having a conversation with Gemini was almost impossible compared to ChatGPT.
Something is going on at Google
I think they might have hit RSI
They’ve released new frontier models FOUR TIMES in less than a month. And each is an actual MAJOR improvement
Carbon is the most intelligent model I’ve ever used
While everyone is talking about Anthropic and OpenAI, Google quickly caught up
What I absolutely love about these models too is they were clearly trained in a completely different way than Claude or ChatGPT. 0 distillation.
They talk, think, write, and code COMPLETELY differently than any other model
I can always count on Carbon to give me a novel insight no other model has given me
If you haven’t incorporated Gemini into your tool belt yet, you absolutely need to
1
3
521
Don’t rush! This could be fun, but maybe not worth the money. Let’s wait for more reviews. 🤣✌️
nitter.cf/bnjmn_marie/status/212…
I kind of already want my money back!
I gave Neon High my research plan for some quantization and inference work (made with Opus 5.5)
Prompt: "check this research plan and tell me what you think and suggestions" + my plan
This is what I got:
1
85














