Qwen3.8-Flash-Next beats DeepSeek V4 Flash in benchmarks. That looked interesting, so I wanted to test it on 2Γ RTX PRO 6000s.
Snag: a 172 GiB FP8 checkpoint with 192 GB of VRAM. The weights might fit, but where do the KV cache, Mamba state, draft model, and CUDA graphs go?π
I found a recipe using SGLang. Itβs fast, a few metrics:
β’ Decode: 200β300 tok/s, tested with real coding tasks
β’ 262,144 maximum tokens per request
β’ 606,912 logical KV tokens shared across the engine
The real trick is that it doesnβt offload the whole model. It offloads the weird part.
Qwen has a massive PLE n-gram lookup table:
20M rows Γ 2,560 dimensions Γ FP8
= 51.20 GB
= 47.68 GiB
Instead of putting that table in VRAM, SGLang keeps it in pinned system RAM and splits it across the two
TP ranks, with about 23.84 GiB per rank.
When the model needs an embedding, the GPUs fetch only the few rows referenced by the current tokens
over PCIe. The whole 47.68 GiB table never moves.
That leaves the compute-heavy parts in VRAM.
The attached video illustrates what the memory placement looks like and how one request moves through tokenization, scheduling, prefill, PCIe row fetches, NCCL communication, MTP-assisted decoding, and finally back to the client.
DeepSeek V4 Flash needs 2Γ RTX 6000s
Qwen3.8-27B FP8 needs just 2Γ RTX 4090s (2,873 tok/s prefill, 74 tok/s decode)
I was curious how they compared, so I ran the same Pi agentic task on both. Both solved it in one shot with no follow-ups
Here are the outputs side by sideπ
Metaβs first good open-weight model since Llama.
Mark Zuckerbergβs Muse Glimmer 30B just front-ran Qwen3.8 27B.
I ran a real agentic task locally on a single RTX 4090, making 28 LLM calls:
- 817 tok/s prefill
- 86 tok/s decode (max 263)
- 17.24% DFlash acceptance
Seriously fast for a 30B model on one consumer GPU.
7
-- Weekend learning --
Hermes writes "notes" for its next session, Prime rewrites its own instructions mid-job.
Neither touches a single model weight.
Weekend learning
Self-improving AI agents sound cool, but what does βimprovingβ actually mean?
Hermes by @NousResearch and Prime by @PrimeIntellect both claim they do it, so I scanned their source code and broke down how each one really works.
Hereβs the simple version π§΅
1
I spent 2 days tuning a vLLM serving profile for DeepSeek V4 Flash 0731 so my 7-person team could use Hermes Agent comfortably on 2Γ RTX PRO 6000 Blackwell GPUs. At 100K C4, prefill was 21Γ faster and decode 14Γ faster than my previous setup. Here is what I changed π
TUNED VALUES
I kept the custom vLLM Gilded Gnosis v20 build, true TP=2, native DSpark K=5, B12X Blackwell kernels, and FP8 KV cache.
These are the server values I ended up with:
β’ MAX_NUM_BATCHED_TOKENS=4096, down from 8192. This was the biggest measured KV-capacity win.
β’ MAX_NUM_SEQS=24, down from 32. This reduced scheduler, activation, and graph overhead.
β’ MAX_CUDAGRAPH_CAPTURE_SIZE=auto. It resolved to 144 rows instead of 192.
β’ MAX_MODEL_LEN=262144, up from 131072. This is the limit for one request, not the total KV pool.
β’ GPU_MEMORY_UTILIZATION=0.975, down from 0.985. This leaves safer transient headroom.
β’ PREFIX_CACHE=1, enabled for real Hermes workloads.
β’ KV_OFFLOADING_SIZE=0. Everything stays on the GPUs.
Reducing MAX_NUM_BATCHED_TOKENS from 8192 to 4096 moved logical KV capacity from about 138K to 286K. The full tuned profile reached 474,850 logical tokens, up from 138,227 before.
DeepSeek V4's sparse attention design saves KV memory. It uses FP8, compression, and short sliding windows. It keeps full token history only where needed, instead of storing full F16 K/V data for every token in every layer.
This lets 8.12 GiB of KV memory per GPU support up to 474,850 tokens. The exact limit also depends on vLLM and the server settings. TP=2 splits one KV pool across both GPUs, so the total is still 474,850 tokens, not twice that.
I also changed the benchmark from the original 128 output tokens to 7,000 output tokens per request. A 128-token response is too short for this test. Prefill dominates the timing, decode may not settle, and KV pressure can stay hidden. A 7K output shows sustained decode and what happens when several long Hermes jobs overlap.
NEW BENCHMARK
I reran 20 cells using the same fixed 13-record Spec-Bench subset:
β’ Context: 2K, 32K, 64K, and 100K
β’ Concurrency: 1, 4, 8, 16, and 24
β’ Output: exactly 7,000 tokens for every request
β’ Cache: no warm or cross-request prompt cache. Prefix caching stayed enabled for the real server, but benchmark requests did not share cached prefixes.
β’ Total: 212 requests and 1,484,000 generated tokens
All speeds are tokens per second. At concurrency > 1, decode is aggregate throughput across all requests, not the speed of each user.
Concurrency order: C1 β C4 β C8 β C16 β C24
2K context
Prefill: 5,059 β 6,066 β 6,976 β 7,463 β 7,153
Decode: 401 β 790 β 1,263 β 1,070 β 1,863
32K context
Prefill: 9,214 β 9,038 β 8,740 β 8,532 β 8,426
Decode: 401 β 383 β 748 β 936 β 1,060
64K context
Prefill: 9,796 β 9,078 β 8,773 β 8,510 β 3,696
Decode: 282 β 347 β 574 β 592 β 341
100K context
Prefill: 9,944 β 9,053 β 8,787 β 5,749 β 5,621
Decode: 247 β 405 β 374 β 342 β 331
CONCLUSION
100K C4 is the sweet spot. Four 102,400-token prompts with 7K outputs need 437,600 KV tokens, so they fit inside the 474,850-token pool.
At 100K C4, the old profile reached 436 prefill and 29 aggregate decode tok/s. The tuned profile reached 9,053 and 405, about 21Γ and 14Γ faster. Not every cell improved. At C1, decode fell from 392 to 247 tok/s. This tuning targets team concurrency, not single-request speed.
C8 needs 875,200 tokens, so it exceeds the pool. The server still finishes, but queueing and recomputation grow. P95 time to first token rises from 43 seconds at C4 to 89 seconds at C8, 4.4 minutes at C16, and 6.8 minutes at C24.
Two RTX PRO 6000 GPUs now feel comfortable for my 7-person team. I would allow four simultaneous 100K+7K jobs and queue the rest. Shorter jobs can use higher concurrency. For a larger long-context team, I would consider four GPUs, but I have not benchmarked that setup.
I wanted a practical way for my 5-person team to run DeepSeek V4 Flash 0731 with Hermes Agent on 2x RTX PRO 6000 Blackwell GPUs. So I ran 80 fresh tests across llama.cpp, DSpark, and custom vLLM with and without CPU KV offload. Two GPUs work, but KV is the catch. π
THE BENCHMARK
I cared about team use, not one flashy speed number. Seven people can hit the server at the same time. Hermes Agent can also create long, tool-heavy conversations. So I tested both speed and concurrency.
Hardware: 2x NVIDIA RTX PRO 6000 Blackwell 96GB over PCIe. No NVLink.
Prompts were exactly 2K, 32K, 64K, and 100K tokens. Concurrency was 1, 4, 8, 16, and 32. I used a fixed 13-record Spec-Bench subset. Each request generated 128 tokens with greedy streaming. GPU prompt caching was off. For CPU offload, unique early prompt blocks kept external cache reuse at 0%.
All speeds are tokens per second. Prefill and effective aggregate decode are shown separately.
1. LLAMA.CPP WITHOUT DSPARK
Target: DeepSeek V4 Flash 0731 UD-Q8_K_XL GGUF by Unsloth. KV cache: F16. Slots: 32, with up to 128K context per slot.
This used llama.cpp layer split. Complete model layers were divided between the GPUs. The tensor-parallel-like row split mode does not support DeepSeek V4 Flash 0731 yet. The command still uses --tensor-split 1,1. In layer-split mode, that only sets the GPU allocation ratio. It does not enable tensor parallelism.
Concurrency order: C1 β C4 β C8 β C16 β C32
2K context
Prefill: 2,379 β 1,939 β 1,803 β 1,623 β 461
Decode: 48 β 106 β 104 β 100 β 104
32K context
Prefill: 1,004 β 1,022 β 1,056 β 1,117 β 1,326
Decode: 45 β 10 β 8 β 8 β 8
64K context
Prefill: 1,259 β 1,213 β 1,225 β 1,267 β 1,389
Decode: 43 β 4 β 4 β 3 β 3
100K context
Prefill: 1,229 β 1,192 β 1,201 β 1,225 β 1,296
Decode: 40 β 2 β 2 β 2 β 2
2. LLAMA.CPP WITH DSPARK 0731
This used the same GGUF and layer split. I added the official 0731 DSpark BF16 drafter at K=5.
The KV cache had to change from F16 to Q8_0. Otherwise, the target, drafter, and 32 x 128K slots would not fit. That KV change is an important caveat in this comparison.
Concurrency order: C1 β C4 β C8 β C16 β C32
2K context
Prefill: 1,652 β 399 β 346 β 182 β 95
Decode: 47 β 36 β 19 β 11 β 6
32K context
Prefill: 1,240 β 1,186 β 1,195 β 1,222 β 1,356
Decode: 42 β 9 β 7 β 7 β 6
64K context
Prefill: 1,326 β 1,267 β 1,272 β 1,246 β 1,303
Decode: 46 β 4 β 3 β 3 β 3
100K context
Prefill: 1,245 β 1,210 β 1,207 β 1,172 β 1,201
Decode: 41 β 2 β 2 β 2 β 2
3. CUSTOM VLLM GILDED GNOSIS TP=2 WITH NATIVE DSPARK
This was not normal vLLM. The stack used:
β’ vLLM 0.11.2.dev280+gilded.gnosis.v20
β’ Gilded Gnosis v20 custom container
β’ True TP=2 across both GPUs
β’ Native DSpark at K=5
β’ Blackwell-specific B12X kernels
β’ FP4/W4A8 MoE experts
β’ FP8 KV cache
β’ Official DeepSeek V4 Flash 0731 checkpoint
Exact image:
voipmonitor/vllm:gilded-gnosis-v20-vllm1e9c9c3-sieec30ff-fi801d57a-cu132-20260731-r16
HOW IT FIT INTO 192GB OF VRAM
The cards have 192GB on the box. vLLM saw about 191 GB of usable VRAM. TP=2 does not turn that into one big memory pool. Each rank still has to fit on its own GPU.
Per GPU:
β’ Usable VRAM: about 96 GB
β’ vLLM VRAM budget: about 94 GB
β’ Weights, DSpark, graphs, activations, and runtime: about 86 GB
β’ KV cache: about 8 GB
β’ Safety margin: about 1 GB
Across both GPUs:
β’ Non-KV allocations: about 173 GB
β’ KV cache: about 15 GB
β’ Safety margin: about 3 GB
β’ Total usable VRAM: about 191 GB
Each GPU got about 8 GB of KV cache. Together, that provided one logical pool of 137,603 tokens. It was not a 275,206-token pool. You cannot simply add the two TP ranks together. One resident 128K request fit. Multiple long requests did not. That forced vLLM to preempt and recompute during the concurrent long-context tests.
The model advertises a 1M-token maximum context. This setup could not reach it. A 1M context would need about 59 GB of KV per GPU. Total usage would rise to about 145 GB per GPU, or 290 GB across TP=2. That is far beyond 192GB.
Concurrency order: C1 β C4 β C8 β C16 β C32
2K context
Prefill: 976 β 4,317 β 9,723 β 12,406 β 13,226
Decode: 208 β 161 β 360 β 416 β 581
32K context
Prefill: 12,183 β 14,056 β 10,880 β 10,975 β 10,602
Decode: 376 β 67 β 47 β 45 β 42
64K context
Prefill: 12,726 β 13,578 β 11,118 β 10,996 β 10,120
Decode: 201 β 34 β 24 β 23 β 20
100K context
Prefill: 12,627 β 9,803 β 7,666 β 6,883 β 6,877
Decode: 252 β 15 β 10 β 9 β 9
4. CUSTOM VLLM GILDED GNOSIS TP=2 WITH NATIVE DSPARK AND CPU KV OFFLOAD
This used the same custom vLLM stack, model, TP=2, DSpark K=5, FP8 KV cache, 128K maximum context, and 32-sequence limit. The change was KV_OFFLOADING_SIZE=40, which added one 40 GB shared mmap in system RAM.
GPU KV capacity remained 141,047 tokens. CPU offload did not pool the two GPUs into a larger VRAM block. It stored evicted KV in system RAM and restored it over PCIe.
Every scenario and stream got a unique tag in its first prompt block. That kept completed-request KV from turning later cells into cache-hit tests.
Concurrency order: C1 β C4 β C8 β C16 β C32
2K context
Prefill: 4,839 β 6,013 β 10,376 β 12,400 β 13,365
Decode: 215 β 305 β 385 β 493 β 615
32K context
Prefill: 13,073 β 13,977 β 14,284 β 12,726 β 12,906
Decode: 297 β 70 β 61 β 52 β 52
64K context
Prefill: 12,617 β 13,553 β 12,773 β 12,124 β 11,828
Decode: 228 β 35 β 28 β 25 β 24
100K context
Prefill: 12,748 β 12,663 β 12,318 β STALL β STALL
Decode: 256 β 20 β 17 β STALL β STALL
The sequential sweep first stalled at 100K C8 after old offload state had accumulated. I cleared only the disposable CPU KV mmap and reran that cell. It then completed at 12,318 prefill and 17 aggregate decode tok/s.
I also reran C16 and C32 from clean server starts. C16 completed 6 of 16 requests, then stopped. C32 completed 11 of 32, then stopped. Both ended with 0 running requests, 3 connector-deferred requests, and no KV-transfer or preemption progress for more than two minutes. STALL means no valid throughput number, not zero tok/s.
WHAT STOOD OUT
At 100K and C1, vLLM reached 12,627 prefill and 252 decode tok/s. That was about 10x faster in prefill and 6x faster in decode than llama.cpp. At 100K and C32, vLLM was still about 5x faster in both prefill and effective decode.
CPU offload helped the completed 100K concurrency cells. At C4, prefill rose from 9,803 to 12,663 tok/s, while aggregate decode rose from 15 to 20 tok/s. At C8, prefill rose from 7,666 to 12,318, while aggregate decode rose from 10 to 17.
But the failure mode matters more than those gains. The native 40 GiB CPU tier completed 100K C8 after a clean reset. Clean C16 and C32 runs still stalled inside the experimental offload connector. I would not deploy this exact offload setup for unrestricted long-context concurrency.
DSpark did not help this llama.cpp setup. The worst case was 2K at C32. Prefill fell from 461 to 95 tok/s. Decode fell from 104 to 6 tok/s. The draft overhead was too high. Acceptance was modest, and multi-slot batching was weak.
One detail matters here. At concurrency above one, decode means aggregate serving throughput. It is not per-stream speed. For example, vLLMβs 15 tok/s at 100K and C4 is the total across all four requests. It is not 15 tok/s for each stream.
This is also not a pure engine comparison. llama.cpp used an 8-bit GGUF target with layer split. vLLM used a custom FP8/FP4 B12X representation with true TP=2. I was comparing practical deployment options, not identical model formats.
QUALITY WITH HERMES AGENT
Speed is only half the test. I also wanted to know if the model could do useful agent work without hand-holding.
The task was to create an elegant 8-10 page editorial PDF about breakfast culture in:
β’ Paris
β’ Tokyo
β’ Istanbul
β’ Mexico City
β’ Hong Kong
β’ Copenhagen
It had to use verified facts and unique, authentic images.
Hermes Agent researched the morning rituals in each city. It sourced 18 different images from Wikimedia Commons. It built the journal in HTML and CSS, then rendered it with Chromium. The first draft was 17 pages. The agent refined it to exactly 10 pages. It also checked the content, layout, and images before finishing.
The result was Breakfast_Around_the_World.pdf, a polished 10-page A4 journal with:
β’ A cover and introduction
β’ Six city stories
β’ A cross-city comparison table
β’ References and photo credits
β’ Different imagery for every city
β’ An editable HTML source
CONCLUSION
This was the result I cared about most. DeepSeek V4 Flash 0731 worked really well with Hermes Agent. It completed the whole editorial task in one shot. I did not send a follow-up request. I did not ask for corrections. The first completed output was the final output I wanted.
Two RTX PRO 6000 GPUs can run and serve this model locally for a small team. The CPU tier can help with a few simultaneous long requests, but I would cap concurrency until the connector stall is fixed. For a larger team, I would rather move to four RTX PRO 6000 GPUs for more GPU-resident KV headroom. The four-GPU recommendation comes from the memory and concurrency behavior above. I did not benchmark the four-GPU setup.
I wanted a practical way for my 5-person team to run DeepSeek V4 Flash 0731 with Hermes Agent on 2x RTX PRO 6000 Blackwell GPUs. So I ran 80 fresh tests across llama.cpp, DSpark, and custom vLLM with and without CPU KV offload. Two GPUs work, but KV is the catch. π
THE BENCHMARK
I cared about team use, not one flashy speed number. Seven people can hit the server at the same time. Hermes Agent can also create long, tool-heavy conversations. So I tested both speed and concurrency.
Hardware: 2x NVIDIA RTX PRO 6000 Blackwell 96GB over PCIe. No NVLink.
Prompts were exactly 2K, 32K, 64K, and 100K tokens. Concurrency was 1, 4, 8, 16, and 32. I used a fixed 13-record Spec-Bench subset. Each request generated 128 tokens with greedy streaming. GPU prompt caching was off. For CPU offload, unique early prompt blocks kept external cache reuse at 0%.
All speeds are tokens per second. Prefill and effective aggregate decode are shown separately.
1. LLAMA.CPP WITHOUT DSPARK
Target: DeepSeek V4 Flash 0731 UD-Q8_K_XL GGUF by Unsloth. KV cache: F16. Slots: 32, with up to 128K context per slot.
This used llama.cpp layer split. Complete model layers were divided between the GPUs. The tensor-parallel-like row split mode does not support DeepSeek V4 Flash 0731 yet. The command still uses --tensor-split 1,1. In layer-split mode, that only sets the GPU allocation ratio. It does not enable tensor parallelism.
Concurrency order: C1 β C4 β C8 β C16 β C32
2K context
Prefill: 2,379 β 1,939 β 1,803 β 1,623 β 461
Decode: 48 β 106 β 104 β 100 β 104
32K context
Prefill: 1,004 β 1,022 β 1,056 β 1,117 β 1,326
Decode: 45 β 10 β 8 β 8 β 8
64K context
Prefill: 1,259 β 1,213 β 1,225 β 1,267 β 1,389
Decode: 43 β 4 β 4 β 3 β 3
100K context
Prefill: 1,229 β 1,192 β 1,201 β 1,225 β 1,296
Decode: 40 β 2 β 2 β 2 β 2
2. LLAMA.CPP WITH DSPARK 0731
This used the same GGUF and layer split. I added the official 0731 DSpark BF16 drafter at K=5.
The KV cache had to change from F16 to Q8_0. Otherwise, the target, drafter, and 32 x 128K slots would not fit. That KV change is an important caveat in this comparison.
Concurrency order: C1 β C4 β C8 β C16 β C32
2K context
Prefill: 1,652 β 399 β 346 β 182 β 95
Decode: 47 β 36 β 19 β 11 β 6
32K context
Prefill: 1,240 β 1,186 β 1,195 β 1,222 β 1,356
Decode: 42 β 9 β 7 β 7 β 6
64K context
Prefill: 1,326 β 1,267 β 1,272 β 1,246 β 1,303
Decode: 46 β 4 β 3 β 3 β 3
100K context
Prefill: 1,245 β 1,210 β 1,207 β 1,172 β 1,201
Decode: 41 β 2 β 2 β 2 β 2
3. CUSTOM VLLM GILDED GNOSIS TP=2 WITH NATIVE DSPARK
This was not normal vLLM. The stack used:
β’ vLLM 0.11.2.dev280+gilded.gnosis.v20
β’ Gilded Gnosis v20 custom container
β’ True TP=2 across both GPUs
β’ Native DSpark at K=5
β’ Blackwell-specific B12X kernels
β’ FP4/W4A8 MoE experts
β’ FP8 KV cache
β’ Official DeepSeek V4 Flash 0731 checkpoint
Exact image:
voipmonitor/vllm:gilded-gnosis-v20-vllm1e9c9c3-sieec30ff-fi801d57a-cu132-20260731-r16
HOW IT FIT INTO 192GB OF VRAM
The cards have 192GB on the box. vLLM saw about 191 GB of usable VRAM. TP=2 does not turn that into one big memory pool. Each rank still has to fit on its own GPU.
Per GPU:
β’ Usable VRAM: about 96 GB
β’ vLLM VRAM budget: about 94 GB
β’ Weights, DSpark, graphs, activations, and runtime: about 86 GB
β’ KV cache: about 8 GB
β’ Safety margin: about 1 GB
Across both GPUs:
β’ Non-KV allocations: about 173 GB
β’ KV cache: about 15 GB
β’ Safety margin: about 3 GB
β’ Total usable VRAM: about 191 GB
Each GPU got about 8 GB of KV cache. Together, that provided one logical pool of 137,603 tokens. It was not a 275,206-token pool. You cannot simply add the two TP ranks together. One resident 128K request fit. Multiple long requests did not. That forced vLLM to preempt and recompute during the concurrent long-context tests.
The model advertises a 1M-token maximum context. This setup could not reach it. A 1M context would need about 59 GB of KV per GPU. Total usage would rise to about 145 GB per GPU, or 290 GB across TP=2. That is far beyond 192GB.
Concurrency order: C1 β C4 β C8 β C16 β C32
2K context
Prefill: 976 β 4,317 β 9,723 β 12,406 β 13,226
Decode: 208 β 161 β 360 β 416 β 581
32K context
Prefill: 12,183 β 14,056 β 10,880 β 10,975 β 10,602
Decode: 376 β 67 β 47 β 45 β 42
64K context
Prefill: 12,726 β 13,578 β 11,118 β 10,996 β 10,120
Decode: 201 β 34 β 24 β 23 β 20
100K context
Prefill: 12,627 β 9,803 β 7,666 β 6,883 β 6,877
Decode: 252 β 15 β 10 β 9 β 9
4. CUSTOM VLLM GILDED GNOSIS TP=2 WITH NATIVE DSPARK AND CPU KV OFFLOAD
This used the same custom vLLM stack, model, TP=2, DSpark K=5, FP8 KV cache, 128K maximum context, and 32-sequence limit. The change was KV_OFFLOADING_SIZE=40, which added one 40 GB shared mmap in system RAM.
GPU KV capacity remained 141,047 tokens. CPU offload did not pool the two GPUs into a larger VRAM block. It stored evicted KV in system RAM and restored it over PCIe.
Every scenario and stream got a unique tag in its first prompt block. That kept completed-request KV from turning later cells into cache-hit tests.
Concurrency order: C1 β C4 β C8 β C16 β C32
2K context
Prefill: 4,839 β 6,013 β 10,376 β 12,400 β 13,365
Decode: 215 β 305 β 385 β 493 β 615
32K context
Prefill: 13,073 β 13,977 β 14,284 β 12,726 β 12,906
Decode: 297 β 70 β 61 β 52 β 52
64K context
Prefill: 12,617 β 13,553 β 12,773 β 12,124 β 11,828
Decode: 228 β 35 β 28 β 25 β 24
100K context
Prefill: 12,748 β 12,663 β 12,318 β STALL β STALL
Decode: 256 β 20 β 17 β STALL β STALL
The sequential sweep first stalled at 100K C8 after old offload state had accumulated. I cleared only the disposable CPU KV mmap and reran that cell. It then completed at 12,318 prefill and 17 aggregate decode tok/s.
I also reran C16 and C32 from clean server starts. C16 completed 6 of 16 requests, then stopped. C32 completed 11 of 32, then stopped. Both ended with 0 running requests, 3 connector-deferred requests, and no KV-transfer or preemption progress for more than two minutes. STALL means no valid throughput number, not zero tok/s.
WHAT STOOD OUT
At 100K and C1, vLLM reached 12,627 prefill and 252 decode tok/s. That was about 10x faster in prefill and 6x faster in decode than llama.cpp. At 100K and C32, vLLM was still about 5x faster in both prefill and effective decode.
CPU offload helped the completed 100K concurrency cells. At C4, prefill rose from 9,803 to 12,663 tok/s, while aggregate decode rose from 15 to 20 tok/s. At C8, prefill rose from 7,666 to 12,318, while aggregate decode rose from 10 to 17.
But the failure mode matters more than those gains. The native 40 GiB CPU tier completed 100K C8 after a clean reset. Clean C16 and C32 runs still stalled inside the experimental offload connector. I would not deploy this exact offload setup for unrestricted long-context concurrency.
DSpark did not help this llama.cpp setup. The worst case was 2K at C32. Prefill fell from 461 to 95 tok/s. Decode fell from 104 to 6 tok/s. The draft overhead was too high. Acceptance was modest, and multi-slot batching was weak.
One detail matters here. At concurrency above one, decode means aggregate serving throughput. It is not per-stream speed. For example, vLLMβs 15 tok/s at 100K and C4 is the total across all four requests. It is not 15 tok/s for each stream.
This is also not a pure engine comparison. llama.cpp used an 8-bit GGUF target with layer split. vLLM used a custom FP8/FP4 B12X representation with true TP=2. I was comparing practical deployment options, not identical model formats.
QUALITY WITH HERMES AGENT
Speed is only half the test. I also wanted to know if the model could do useful agent work without hand-holding.
The task was to create an elegant 8-10 page editorial PDF about breakfast culture in:
β’ Paris
β’ Tokyo
β’ Istanbul
β’ Mexico City
β’ Hong Kong
β’ Copenhagen
It had to use verified facts and unique, authentic images.
Hermes Agent researched the morning rituals in each city. It sourced 18 different images from Wikimedia Commons. It built the journal in HTML and CSS, then rendered it with Chromium. The first draft was 17 pages. The agent refined it to exactly 10 pages. It also checked the content, layout, and images before finishing.
The result was Breakfast_Around_the_World.pdf, a polished 10-page A4 journal with:
β’ A cover and introduction
β’ Six city stories
β’ A cross-city comparison table
β’ References and photo credits
β’ Different imagery for every city
β’ An editable HTML source
CONCLUSION
This was the result I cared about most. DeepSeek V4 Flash 0731 worked really well with Hermes Agent. It completed the whole editorial task in one shot. I did not send a follow-up request. I did not ask for corrections. The first completed output was the final output I wanted.
Two RTX PRO 6000 GPUs can run and serve this model locally for a small team. The CPU tier can help with a few simultaneous long requests, but I would cap concurrency until the connector stall is fixed. For a larger team, I would rather move to four RTX PRO 6000 GPUs for more GPU-resident KV headroom. The four-GPU recommendation comes from the memory and concurrency behavior above. I did not benchmark the four-GPU setup.
Hereβs how shared AI compute works across Buzz + LocalAGI, encrypted end to end.
Last post, I showed what I built with @jack's Buzz: shared compute between my Mac mini and my teammate's MacBook Pro.
That was the demo. Now under the hood: how trust is established, how E2E encryption seals each prompt, what the relay can see, and where plaintext exists. π
Setup in one line: Buzz and Bumble run on my 16 GB Mac mini. My teammate owns a 64 GB MacBook Pro and chooses to share its model (Qwen3.6 27B MTP) compute with my private LocalAGI network. My agents stay on my mini. Their intelligence can come from my teammate's Pro.
What happens when I hit send:
1. Buzz calls LocalAGI on my mini. That's localhost. Nothing has left my machine yet.
2. My mini seals the request for one specific machine: my teammate's Pro, which I approved after they chose to share it with the network. My private keys stay on my mini. My teammate's private keys stay on their Pro.
3. It crosses the internet through our relay. Being straight about this bit: there's a server in the middle, we run it, and every byte passes through it.
It exists for a boring reason. Reaching my teammate's Mac behind their router otherwise means port forwarding, or a VPN, or a static IP. Neither of us wants to set that up.
What it can't do is read any of it. That's the whole point of the encryption. The relay holds no keys, so the isolation is proven rather than promised. It moves boxes it can't open.
4. My teammate's Pro opens the request and runs the model they chose to share. That's the only other place the text exists in the clear.
This is the trust boundary in plain English: E2EE protects the prompt from the internet and relay, not from my teammate. Their Pro is an approved endpoint and has to see the request to perform inference.
5. Tokens come back the same way. Sealed on my teammate's Pro, opened on my mini, into the Buzz channel.
So the readable prompt lives on two explicitly trusted machines. I own the mini. My teammate owns the Pro. Slack isn't in that path. Neither is a cloud model provider.
The obvious question: why not have my teammate run a local server and point Buzz at it? On the same wifi, do exactly that. Across two networks you're into port forwarding or a VPN, plus TLS, plus auth, plus keeping all of it alive.
Trade-offs, honestly: the relay hop costs a little latency, my teammate's Pro has to be awake and actively sharing, I trust my teammate with the prompt content, and the relay can tell that our two machines are talking, just not what about.
Buzz gives every agent its own keypair so its identity isn't something a vendor hands out. LocalAGI gives each compute machine its own keys too. My teammate shares intelligence with my agents without handing me their laptop, login, or model endpoint.
6
Enjoy the weekend setup, buddy!
I've run Unsloth's Qwen3.6-27B GGUF for months. It's my favorite agent model.
Then NVFP4 dropped. I wanted it on vLLM, but didn't know how to tune MTP speculative decoding, how it'd behave across 2 RTX 5090s, or what long context would do.
So I benchmarked it. 3 findings π§΅
This is what we've been buildingπ
What it is: a team of AI agents on your own machine, not someone else's cloud.
π local, on your device
π€ a private E2E-encrypted mesh you trust
π public network, only if you opt in
macOS now.
π localagi.network
10
nitter.cf/zoomyzoomm/status/1956β¦
Great thread on the AI house of cards! The $200 user cost vs. $10k compute reality is wild. But hereβs another angle: As models get smarter in smaller sizes and edge devices grow powerful, 100% local AI could dodge the cost spiral and VC trap.
This Post is from a suspended account. Learn more
Even better, the moderate edge devices can form a network, sharing idle compute with those lacking hardware for AI use. This turns unused resources into value, breaking the cost cycle.
1
41








