Local AGI (@LocalAGINetwork) header
Local AGI (@LocalAGINetwork) profile picture

The privacy-first agentic AI platform.

Joined August 2025
Local AGI retweeted
Qwen3.8-Flash-Next beats DeepSeek V4 Flash in benchmarks. That looked interesting, so I wanted to test it on 2Γ— RTX PRO 6000s. Snag: a 172 GiB FP8 checkpoint with 192 GB of VRAM. The weights might fit, but where do the KV cache, Mamba state, draft model, and CUDA graphs go?πŸ‘‡ I found a recipe using SGLang. It’s fast, a few metrics: β€’ Decode: 200–300 tok/s, tested with real coding tasks β€’ 262,144 maximum tokens per request β€’ 606,912 logical KV tokens shared across the engine The real trick is that it doesn’t offload the whole model. It offloads the weird part. Qwen has a massive PLE n-gram lookup table: 20M rows Γ— 2,560 dimensions Γ— FP8 = 51.20 GB = 47.68 GiB Instead of putting that table in VRAM, SGLang keeps it in pinned system RAM and splits it across the two TP ranks, with about 23.84 GiB per rank. When the model needs an embedding, the GPUs fetch only the few rows referenced by the current tokens over PCIe. The whole 47.68 GiB table never moves. That leaves the compute-heavy parts in VRAM. The attached video illustrates what the memory placement looks like and how one request moves through tokenization, scheduling, prefill, PCIe row fetches, NCCL communication, MTP-assisted decoding, and finally back to the client.
3
3
1
8
1,683
Local AGI retweeted
DeepSeek V4 Flash needs 2Γ— RTX 6000s Qwen3.8-27B FP8 needs just 2Γ— RTX 4090s (2,873 tok/s prefill, 74 tok/s decode) I was curious how they compared, so I ran the same Pi agentic task on both. Both solved it in one shot with no follow-ups Here are the outputs side by sideπŸ‘‡
2
3
1
8
2,787
Local AGI retweeted
I spent 2 days tuning a vLLM serving profile for DeepSeek V4 Flash 0731 so my 7-person team could use Hermes Agent comfortably on 2Γ— RTX PRO 6000 Blackwell GPUs. At 100K C4, prefill was 21Γ— faster and decode 14Γ— faster than my previous setup. Here is what I changed πŸ‘‡ TUNED VALUES I kept the custom vLLM Gilded Gnosis v20 build, true TP=2, native DSpark K=5, B12X Blackwell kernels, and FP8 KV cache. These are the server values I ended up with: β€’ MAX_NUM_BATCHED_TOKENS=4096, down from 8192. This was the biggest measured KV-capacity win. β€’ MAX_NUM_SEQS=24, down from 32. This reduced scheduler, activation, and graph overhead. β€’ MAX_CUDAGRAPH_CAPTURE_SIZE=auto. It resolved to 144 rows instead of 192. β€’ MAX_MODEL_LEN=262144, up from 131072. This is the limit for one request, not the total KV pool. β€’ GPU_MEMORY_UTILIZATION=0.975, down from 0.985. This leaves safer transient headroom. β€’ PREFIX_CACHE=1, enabled for real Hermes workloads. β€’ KV_OFFLOADING_SIZE=0. Everything stays on the GPUs. Reducing MAX_NUM_BATCHED_TOKENS from 8192 to 4096 moved logical KV capacity from about 138K to 286K. The full tuned profile reached 474,850 logical tokens, up from 138,227 before. DeepSeek V4's sparse attention design saves KV memory. It uses FP8, compression, and short sliding windows. It keeps full token history only where needed, instead of storing full F16 K/V data for every token in every layer. This lets 8.12 GiB of KV memory per GPU support up to 474,850 tokens. The exact limit also depends on vLLM and the server settings. TP=2 splits one KV pool across both GPUs, so the total is still 474,850 tokens, not twice that. I also changed the benchmark from the original 128 output tokens to 7,000 output tokens per request. A 128-token response is too short for this test. Prefill dominates the timing, decode may not settle, and KV pressure can stay hidden. A 7K output shows sustained decode and what happens when several long Hermes jobs overlap. NEW BENCHMARK I reran 20 cells using the same fixed 13-record Spec-Bench subset: β€’ Context: 2K, 32K, 64K, and 100K β€’ Concurrency: 1, 4, 8, 16, and 24 β€’ Output: exactly 7,000 tokens for every request β€’ Cache: no warm or cross-request prompt cache. Prefix caching stayed enabled for the real server, but benchmark requests did not share cached prefixes. β€’ Total: 212 requests and 1,484,000 generated tokens All speeds are tokens per second. At concurrency > 1, decode is aggregate throughput across all requests, not the speed of each user. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C24 2K context Prefill: 5,059 β†’ 6,066 β†’ 6,976 β†’ 7,463 β†’ 7,153 Decode: 401 β†’ 790 β†’ 1,263 β†’ 1,070 β†’ 1,863 32K context Prefill: 9,214 β†’ 9,038 β†’ 8,740 β†’ 8,532 β†’ 8,426 Decode: 401 β†’ 383 β†’ 748 β†’ 936 β†’ 1,060 64K context Prefill: 9,796 β†’ 9,078 β†’ 8,773 β†’ 8,510 β†’ 3,696 Decode: 282 β†’ 347 β†’ 574 β†’ 592 β†’ 341 100K context Prefill: 9,944 β†’ 9,053 β†’ 8,787 β†’ 5,749 β†’ 5,621 Decode: 247 β†’ 405 β†’ 374 β†’ 342 β†’ 331 CONCLUSION 100K C4 is the sweet spot. Four 102,400-token prompts with 7K outputs need 437,600 KV tokens, so they fit inside the 474,850-token pool. At 100K C4, the old profile reached 436 prefill and 29 aggregate decode tok/s. The tuned profile reached 9,053 and 405, about 21Γ— and 14Γ— faster. Not every cell improved. At C1, decode fell from 392 to 247 tok/s. This tuning targets team concurrency, not single-request speed. C8 needs 875,200 tokens, so it exceeds the pool. The server still finishes, but queueing and recomputation grow. P95 time to first token rises from 43 seconds at C4 to 89 seconds at C8, 4.4 minutes at C16, and 6.8 minutes at C24. Two RTX PRO 6000 GPUs now feel comfortable for my 7-person team. I would allow four simultaneous 100K+7K jobs and queue the rest. Shorter jobs can use higher concurrency. For a larger long-context team, I would consider four GPUs, but I have not benchmarked that setup.
I wanted a practical way for my 5-person team to run DeepSeek V4 Flash 0731 with Hermes Agent on 2x RTX PRO 6000 Blackwell GPUs. So I ran 80 fresh tests across llama.cpp, DSpark, and custom vLLM with and without CPU KV offload. Two GPUs work, but KV is the catch. πŸ‘‡ THE BENCHMARK I cared about team use, not one flashy speed number. Seven people can hit the server at the same time. Hermes Agent can also create long, tool-heavy conversations. So I tested both speed and concurrency. Hardware: 2x NVIDIA RTX PRO 6000 Blackwell 96GB over PCIe. No NVLink. Prompts were exactly 2K, 32K, 64K, and 100K tokens. Concurrency was 1, 4, 8, 16, and 32. I used a fixed 13-record Spec-Bench subset. Each request generated 128 tokens with greedy streaming. GPU prompt caching was off. For CPU offload, unique early prompt blocks kept external cache reuse at 0%. All speeds are tokens per second. Prefill and effective aggregate decode are shown separately. 1. LLAMA.CPP WITHOUT DSPARK Target: DeepSeek V4 Flash 0731 UD-Q8_K_XL GGUF by Unsloth. KV cache: F16. Slots: 32, with up to 128K context per slot. This used llama.cpp layer split. Complete model layers were divided between the GPUs. The tensor-parallel-like row split mode does not support DeepSeek V4 Flash 0731 yet. The command still uses --tensor-split 1,1. In layer-split mode, that only sets the GPU allocation ratio. It does not enable tensor parallelism. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C32 2K context Prefill: 2,379 β†’ 1,939 β†’ 1,803 β†’ 1,623 β†’ 461 Decode: 48 β†’ 106 β†’ 104 β†’ 100 β†’ 104 32K context Prefill: 1,004 β†’ 1,022 β†’ 1,056 β†’ 1,117 β†’ 1,326 Decode: 45 β†’ 10 β†’ 8 β†’ 8 β†’ 8 64K context Prefill: 1,259 β†’ 1,213 β†’ 1,225 β†’ 1,267 β†’ 1,389 Decode: 43 β†’ 4 β†’ 4 β†’ 3 β†’ 3 100K context Prefill: 1,229 β†’ 1,192 β†’ 1,201 β†’ 1,225 β†’ 1,296 Decode: 40 β†’ 2 β†’ 2 β†’ 2 β†’ 2 2. LLAMA.CPP WITH DSPARK 0731 This used the same GGUF and layer split. I added the official 0731 DSpark BF16 drafter at K=5. The KV cache had to change from F16 to Q8_0. Otherwise, the target, drafter, and 32 x 128K slots would not fit. That KV change is an important caveat in this comparison. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C32 2K context Prefill: 1,652 β†’ 399 β†’ 346 β†’ 182 β†’ 95 Decode: 47 β†’ 36 β†’ 19 β†’ 11 β†’ 6 32K context Prefill: 1,240 β†’ 1,186 β†’ 1,195 β†’ 1,222 β†’ 1,356 Decode: 42 β†’ 9 β†’ 7 β†’ 7 β†’ 6 64K context Prefill: 1,326 β†’ 1,267 β†’ 1,272 β†’ 1,246 β†’ 1,303 Decode: 46 β†’ 4 β†’ 3 β†’ 3 β†’ 3 100K context Prefill: 1,245 β†’ 1,210 β†’ 1,207 β†’ 1,172 β†’ 1,201 Decode: 41 β†’ 2 β†’ 2 β†’ 2 β†’ 2 3. CUSTOM VLLM GILDED GNOSIS TP=2 WITH NATIVE DSPARK This was not normal vLLM. The stack used: β€’ vLLM 0.11.2.dev280+gilded.gnosis.v20 β€’ Gilded Gnosis v20 custom container β€’ True TP=2 across both GPUs β€’ Native DSpark at K=5 β€’ Blackwell-specific B12X kernels β€’ FP4/W4A8 MoE experts β€’ FP8 KV cache β€’ Official DeepSeek V4 Flash 0731 checkpoint Exact image: voipmonitor/vllm:gilded-gnosis-v20-vllm1e9c9c3-sieec30ff-fi801d57a-cu132-20260731-r16 HOW IT FIT INTO 192GB OF VRAM The cards have 192GB on the box. vLLM saw about 191 GB of usable VRAM. TP=2 does not turn that into one big memory pool. Each rank still has to fit on its own GPU. Per GPU: β€’ Usable VRAM: about 96 GB β€’ vLLM VRAM budget: about 94 GB β€’ Weights, DSpark, graphs, activations, and runtime: about 86 GB β€’ KV cache: about 8 GB β€’ Safety margin: about 1 GB Across both GPUs: β€’ Non-KV allocations: about 173 GB β€’ KV cache: about 15 GB β€’ Safety margin: about 3 GB β€’ Total usable VRAM: about 191 GB Each GPU got about 8 GB of KV cache. Together, that provided one logical pool of 137,603 tokens. It was not a 275,206-token pool. You cannot simply add the two TP ranks together. One resident 128K request fit. Multiple long requests did not. That forced vLLM to preempt and recompute during the concurrent long-context tests. The model advertises a 1M-token maximum context. This setup could not reach it. A 1M context would need about 59 GB of KV per GPU. Total usage would rise to about 145 GB per GPU, or 290 GB across TP=2. That is far beyond 192GB. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C32 2K context Prefill: 976 β†’ 4,317 β†’ 9,723 β†’ 12,406 β†’ 13,226 Decode: 208 β†’ 161 β†’ 360 β†’ 416 β†’ 581 32K context Prefill: 12,183 β†’ 14,056 β†’ 10,880 β†’ 10,975 β†’ 10,602 Decode: 376 β†’ 67 β†’ 47 β†’ 45 β†’ 42 64K context Prefill: 12,726 β†’ 13,578 β†’ 11,118 β†’ 10,996 β†’ 10,120 Decode: 201 β†’ 34 β†’ 24 β†’ 23 β†’ 20 100K context Prefill: 12,627 β†’ 9,803 β†’ 7,666 β†’ 6,883 β†’ 6,877 Decode: 252 β†’ 15 β†’ 10 β†’ 9 β†’ 9 4. CUSTOM VLLM GILDED GNOSIS TP=2 WITH NATIVE DSPARK AND CPU KV OFFLOAD This used the same custom vLLM stack, model, TP=2, DSpark K=5, FP8 KV cache, 128K maximum context, and 32-sequence limit. The change was KV_OFFLOADING_SIZE=40, which added one 40 GB shared mmap in system RAM. GPU KV capacity remained 141,047 tokens. CPU offload did not pool the two GPUs into a larger VRAM block. It stored evicted KV in system RAM and restored it over PCIe. Every scenario and stream got a unique tag in its first prompt block. That kept completed-request KV from turning later cells into cache-hit tests. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C32 2K context Prefill: 4,839 β†’ 6,013 β†’ 10,376 β†’ 12,400 β†’ 13,365 Decode: 215 β†’ 305 β†’ 385 β†’ 493 β†’ 615 32K context Prefill: 13,073 β†’ 13,977 β†’ 14,284 β†’ 12,726 β†’ 12,906 Decode: 297 β†’ 70 β†’ 61 β†’ 52 β†’ 52 64K context Prefill: 12,617 β†’ 13,553 β†’ 12,773 β†’ 12,124 β†’ 11,828 Decode: 228 β†’ 35 β†’ 28 β†’ 25 β†’ 24 100K context Prefill: 12,748 β†’ 12,663 β†’ 12,318 β†’ STALL β†’ STALL Decode: 256 β†’ 20 β†’ 17 β†’ STALL β†’ STALL The sequential sweep first stalled at 100K C8 after old offload state had accumulated. I cleared only the disposable CPU KV mmap and reran that cell. It then completed at 12,318 prefill and 17 aggregate decode tok/s. I also reran C16 and C32 from clean server starts. C16 completed 6 of 16 requests, then stopped. C32 completed 11 of 32, then stopped. Both ended with 0 running requests, 3 connector-deferred requests, and no KV-transfer or preemption progress for more than two minutes. STALL means no valid throughput number, not zero tok/s. WHAT STOOD OUT At 100K and C1, vLLM reached 12,627 prefill and 252 decode tok/s. That was about 10x faster in prefill and 6x faster in decode than llama.cpp. At 100K and C32, vLLM was still about 5x faster in both prefill and effective decode. CPU offload helped the completed 100K concurrency cells. At C4, prefill rose from 9,803 to 12,663 tok/s, while aggregate decode rose from 15 to 20 tok/s. At C8, prefill rose from 7,666 to 12,318, while aggregate decode rose from 10 to 17. But the failure mode matters more than those gains. The native 40 GiB CPU tier completed 100K C8 after a clean reset. Clean C16 and C32 runs still stalled inside the experimental offload connector. I would not deploy this exact offload setup for unrestricted long-context concurrency. DSpark did not help this llama.cpp setup. The worst case was 2K at C32. Prefill fell from 461 to 95 tok/s. Decode fell from 104 to 6 tok/s. The draft overhead was too high. Acceptance was modest, and multi-slot batching was weak. One detail matters here. At concurrency above one, decode means aggregate serving throughput. It is not per-stream speed. For example, vLLM’s 15 tok/s at 100K and C4 is the total across all four requests. It is not 15 tok/s for each stream. This is also not a pure engine comparison. llama.cpp used an 8-bit GGUF target with layer split. vLLM used a custom FP8/FP4 B12X representation with true TP=2. I was comparing practical deployment options, not identical model formats. QUALITY WITH HERMES AGENT Speed is only half the test. I also wanted to know if the model could do useful agent work without hand-holding. The task was to create an elegant 8-10 page editorial PDF about breakfast culture in: β€’ Paris β€’ Tokyo β€’ Istanbul β€’ Mexico City β€’ Hong Kong β€’ Copenhagen It had to use verified facts and unique, authentic images. Hermes Agent researched the morning rituals in each city. It sourced 18 different images from Wikimedia Commons. It built the journal in HTML and CSS, then rendered it with Chromium. The first draft was 17 pages. The agent refined it to exactly 10 pages. It also checked the content, layout, and images before finishing. The result was Breakfast_Around_the_World.pdf, a polished 10-page A4 journal with: β€’ A cover and introduction β€’ Six city stories β€’ A cross-city comparison table β€’ References and photo credits β€’ Different imagery for every city β€’ An editable HTML source CONCLUSION This was the result I cared about most. DeepSeek V4 Flash 0731 worked really well with Hermes Agent. It completed the whole editorial task in one shot. I did not send a follow-up request. I did not ask for corrections. The first completed output was the final output I wanted. Two RTX PRO 6000 GPUs can run and serve this model locally for a small team. The CPU tier can help with a few simultaneous long requests, but I would cap concurrency until the connector stall is fixed. For a larger team, I would rather move to four RTX PRO 6000 GPUs for more GPU-resident KV headroom. The four-GPU recommendation comes from the memory and concurrency behavior above. I did not benchmark the four-GPU setup.
4
3
1
32
14,917
Local AGI retweeted
I wanted a practical way for my 5-person team to run DeepSeek V4 Flash 0731 with Hermes Agent on 2x RTX PRO 6000 Blackwell GPUs. So I ran 80 fresh tests across llama.cpp, DSpark, and custom vLLM with and without CPU KV offload. Two GPUs work, but KV is the catch. πŸ‘‡ THE BENCHMARK I cared about team use, not one flashy speed number. Seven people can hit the server at the same time. Hermes Agent can also create long, tool-heavy conversations. So I tested both speed and concurrency. Hardware: 2x NVIDIA RTX PRO 6000 Blackwell 96GB over PCIe. No NVLink. Prompts were exactly 2K, 32K, 64K, and 100K tokens. Concurrency was 1, 4, 8, 16, and 32. I used a fixed 13-record Spec-Bench subset. Each request generated 128 tokens with greedy streaming. GPU prompt caching was off. For CPU offload, unique early prompt blocks kept external cache reuse at 0%. All speeds are tokens per second. Prefill and effective aggregate decode are shown separately. 1. LLAMA.CPP WITHOUT DSPARK Target: DeepSeek V4 Flash 0731 UD-Q8_K_XL GGUF by Unsloth. KV cache: F16. Slots: 32, with up to 128K context per slot. This used llama.cpp layer split. Complete model layers were divided between the GPUs. The tensor-parallel-like row split mode does not support DeepSeek V4 Flash 0731 yet. The command still uses --tensor-split 1,1. In layer-split mode, that only sets the GPU allocation ratio. It does not enable tensor parallelism. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C32 2K context Prefill: 2,379 β†’ 1,939 β†’ 1,803 β†’ 1,623 β†’ 461 Decode: 48 β†’ 106 β†’ 104 β†’ 100 β†’ 104 32K context Prefill: 1,004 β†’ 1,022 β†’ 1,056 β†’ 1,117 β†’ 1,326 Decode: 45 β†’ 10 β†’ 8 β†’ 8 β†’ 8 64K context Prefill: 1,259 β†’ 1,213 β†’ 1,225 β†’ 1,267 β†’ 1,389 Decode: 43 β†’ 4 β†’ 4 β†’ 3 β†’ 3 100K context Prefill: 1,229 β†’ 1,192 β†’ 1,201 β†’ 1,225 β†’ 1,296 Decode: 40 β†’ 2 β†’ 2 β†’ 2 β†’ 2 2. LLAMA.CPP WITH DSPARK 0731 This used the same GGUF and layer split. I added the official 0731 DSpark BF16 drafter at K=5. The KV cache had to change from F16 to Q8_0. Otherwise, the target, drafter, and 32 x 128K slots would not fit. That KV change is an important caveat in this comparison. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C32 2K context Prefill: 1,652 β†’ 399 β†’ 346 β†’ 182 β†’ 95 Decode: 47 β†’ 36 β†’ 19 β†’ 11 β†’ 6 32K context Prefill: 1,240 β†’ 1,186 β†’ 1,195 β†’ 1,222 β†’ 1,356 Decode: 42 β†’ 9 β†’ 7 β†’ 7 β†’ 6 64K context Prefill: 1,326 β†’ 1,267 β†’ 1,272 β†’ 1,246 β†’ 1,303 Decode: 46 β†’ 4 β†’ 3 β†’ 3 β†’ 3 100K context Prefill: 1,245 β†’ 1,210 β†’ 1,207 β†’ 1,172 β†’ 1,201 Decode: 41 β†’ 2 β†’ 2 β†’ 2 β†’ 2 3. CUSTOM VLLM GILDED GNOSIS TP=2 WITH NATIVE DSPARK This was not normal vLLM. The stack used: β€’ vLLM 0.11.2.dev280+gilded.gnosis.v20 β€’ Gilded Gnosis v20 custom container β€’ True TP=2 across both GPUs β€’ Native DSpark at K=5 β€’ Blackwell-specific B12X kernels β€’ FP4/W4A8 MoE experts β€’ FP8 KV cache β€’ Official DeepSeek V4 Flash 0731 checkpoint Exact image: voipmonitor/vllm:gilded-gnosis-v20-vllm1e9c9c3-sieec30ff-fi801d57a-cu132-20260731-r16 HOW IT FIT INTO 192GB OF VRAM The cards have 192GB on the box. vLLM saw about 191 GB of usable VRAM. TP=2 does not turn that into one big memory pool. Each rank still has to fit on its own GPU. Per GPU: β€’ Usable VRAM: about 96 GB β€’ vLLM VRAM budget: about 94 GB β€’ Weights, DSpark, graphs, activations, and runtime: about 86 GB β€’ KV cache: about 8 GB β€’ Safety margin: about 1 GB Across both GPUs: β€’ Non-KV allocations: about 173 GB β€’ KV cache: about 15 GB β€’ Safety margin: about 3 GB β€’ Total usable VRAM: about 191 GB Each GPU got about 8 GB of KV cache. Together, that provided one logical pool of 137,603 tokens. It was not a 275,206-token pool. You cannot simply add the two TP ranks together. One resident 128K request fit. Multiple long requests did not. That forced vLLM to preempt and recompute during the concurrent long-context tests. The model advertises a 1M-token maximum context. This setup could not reach it. A 1M context would need about 59 GB of KV per GPU. Total usage would rise to about 145 GB per GPU, or 290 GB across TP=2. That is far beyond 192GB. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C32 2K context Prefill: 976 β†’ 4,317 β†’ 9,723 β†’ 12,406 β†’ 13,226 Decode: 208 β†’ 161 β†’ 360 β†’ 416 β†’ 581 32K context Prefill: 12,183 β†’ 14,056 β†’ 10,880 β†’ 10,975 β†’ 10,602 Decode: 376 β†’ 67 β†’ 47 β†’ 45 β†’ 42 64K context Prefill: 12,726 β†’ 13,578 β†’ 11,118 β†’ 10,996 β†’ 10,120 Decode: 201 β†’ 34 β†’ 24 β†’ 23 β†’ 20 100K context Prefill: 12,627 β†’ 9,803 β†’ 7,666 β†’ 6,883 β†’ 6,877 Decode: 252 β†’ 15 β†’ 10 β†’ 9 β†’ 9 4. CUSTOM VLLM GILDED GNOSIS TP=2 WITH NATIVE DSPARK AND CPU KV OFFLOAD This used the same custom vLLM stack, model, TP=2, DSpark K=5, FP8 KV cache, 128K maximum context, and 32-sequence limit. The change was KV_OFFLOADING_SIZE=40, which added one 40 GB shared mmap in system RAM. GPU KV capacity remained 141,047 tokens. CPU offload did not pool the two GPUs into a larger VRAM block. It stored evicted KV in system RAM and restored it over PCIe. Every scenario and stream got a unique tag in its first prompt block. That kept completed-request KV from turning later cells into cache-hit tests. Concurrency order: C1 β†’ C4 β†’ C8 β†’ C16 β†’ C32 2K context Prefill: 4,839 β†’ 6,013 β†’ 10,376 β†’ 12,400 β†’ 13,365 Decode: 215 β†’ 305 β†’ 385 β†’ 493 β†’ 615 32K context Prefill: 13,073 β†’ 13,977 β†’ 14,284 β†’ 12,726 β†’ 12,906 Decode: 297 β†’ 70 β†’ 61 β†’ 52 β†’ 52 64K context Prefill: 12,617 β†’ 13,553 β†’ 12,773 β†’ 12,124 β†’ 11,828 Decode: 228 β†’ 35 β†’ 28 β†’ 25 β†’ 24 100K context Prefill: 12,748 β†’ 12,663 β†’ 12,318 β†’ STALL β†’ STALL Decode: 256 β†’ 20 β†’ 17 β†’ STALL β†’ STALL The sequential sweep first stalled at 100K C8 after old offload state had accumulated. I cleared only the disposable CPU KV mmap and reran that cell. It then completed at 12,318 prefill and 17 aggregate decode tok/s. I also reran C16 and C32 from clean server starts. C16 completed 6 of 16 requests, then stopped. C32 completed 11 of 32, then stopped. Both ended with 0 running requests, 3 connector-deferred requests, and no KV-transfer or preemption progress for more than two minutes. STALL means no valid throughput number, not zero tok/s. WHAT STOOD OUT At 100K and C1, vLLM reached 12,627 prefill and 252 decode tok/s. That was about 10x faster in prefill and 6x faster in decode than llama.cpp. At 100K and C32, vLLM was still about 5x faster in both prefill and effective decode. CPU offload helped the completed 100K concurrency cells. At C4, prefill rose from 9,803 to 12,663 tok/s, while aggregate decode rose from 15 to 20 tok/s. At C8, prefill rose from 7,666 to 12,318, while aggregate decode rose from 10 to 17. But the failure mode matters more than those gains. The native 40 GiB CPU tier completed 100K C8 after a clean reset. Clean C16 and C32 runs still stalled inside the experimental offload connector. I would not deploy this exact offload setup for unrestricted long-context concurrency. DSpark did not help this llama.cpp setup. The worst case was 2K at C32. Prefill fell from 461 to 95 tok/s. Decode fell from 104 to 6 tok/s. The draft overhead was too high. Acceptance was modest, and multi-slot batching was weak. One detail matters here. At concurrency above one, decode means aggregate serving throughput. It is not per-stream speed. For example, vLLM’s 15 tok/s at 100K and C4 is the total across all four requests. It is not 15 tok/s for each stream. This is also not a pure engine comparison. llama.cpp used an 8-bit GGUF target with layer split. vLLM used a custom FP8/FP4 B12X representation with true TP=2. I was comparing practical deployment options, not identical model formats. QUALITY WITH HERMES AGENT Speed is only half the test. I also wanted to know if the model could do useful agent work without hand-holding. The task was to create an elegant 8-10 page editorial PDF about breakfast culture in: β€’ Paris β€’ Tokyo β€’ Istanbul β€’ Mexico City β€’ Hong Kong β€’ Copenhagen It had to use verified facts and unique, authentic images. Hermes Agent researched the morning rituals in each city. It sourced 18 different images from Wikimedia Commons. It built the journal in HTML and CSS, then rendered it with Chromium. The first draft was 17 pages. The agent refined it to exactly 10 pages. It also checked the content, layout, and images before finishing. The result was Breakfast_Around_the_World.pdf, a polished 10-page A4 journal with: β€’ A cover and introduction β€’ Six city stories β€’ A cross-city comparison table β€’ References and photo credits β€’ Different imagery for every city β€’ An editable HTML source CONCLUSION This was the result I cared about most. DeepSeek V4 Flash 0731 worked really well with Hermes Agent. It completed the whole editorial task in one shot. I did not send a follow-up request. I did not ask for corrections. The first completed output was the final output I wanted. Two RTX PRO 6000 GPUs can run and serve this model locally for a small team. The CPU tier can help with a few simultaneous long requests, but I would cap concurrency until the connector stall is fixed. For a larger team, I would rather move to four RTX PRO 6000 GPUs for more GPU-resident KV headroom. The four-GPU recommendation comes from the memory and concurrency behavior above. I did not benchmark the four-GPU setup.
1
3
2
13
8,196
Local AGI retweeted
I've run Unsloth's Qwen3.6-27B GGUF for months. It's my favorite agent model. Then NVFP4 dropped. I wanted it on vLLM, but didn't know how to tune MTP speculative decoding, how it'd behave across 2 RTX 5090s, or what long context would do. So I benchmarked it. 3 findings 🧡
1
6
1
8
6,221