
🦥Hearing things 💤 | Dev Explorer 🥷 | Crypto Optimist 💎 (hodling) | Node Runner (insomniac) | NFT Enthusiast, Let's modular! CoFounder @0xAJPanda
blockchain
Joined August 2021
- Tweets5.6K
- Following4K
- Followers654
- Likes4K
🐼 0xShibu (Ø,G) .ink retweeted
FLOP Devnet is live!!!!!
@flop_labs connects AI agents with GPU miners.
I made a simple guide:
... Set up Talisman
... Claim test FLOP
... Make a transfer
... Explore validator and miner paths
Devnet is for testing. Full guide 👇
github.com/0xAJPanda/flop-pa…
🐼 0xShibu (Ø,G) .ink retweeted
Qwen3.8-27B could soon become accessible on NVIDIA GPUs with just 16GB of VRAM.
An inference setup is currently in development to bring the 27B model to smaller GPUs while retaining features such as large context windows and Multi-Token Prediction (MTP) speculative decoding.
The planned features include:
Qwen3.8-27B support.
Large context windows.
MTP speculative decoding.
Linux and Windows compatibility.
Potentially supported hardware includes the RTX 5060 Ti 16GB, RTX 5070 Ti 16GB, RTX 5080 16GB and older NVIDIA GPUs with 16GB of VRAM.
If the implementation works as intended, developers could have another way to run a capable 27B model locally without immediately upgrading to a 24GB or 32GB graphics card.
But fitting a 27B model into 16GB is not a simple quantisation problem.
Why 16GB VRAM is such a difficult target
The model weights are only one part of the memory requirement. The entire inference pipeline must share the available VRAM.
Four components compete for that space.
1. Model weights
These contain the parameters responsible for the model’s learned capabilities.
Quantisation reduces the number of bits used to represent those parameters, allowing a larger model to occupy less memory.
However, more aggressive quantisation introduces greater numerical error, which can affect reasoning, coding and instruction-following quality.
The challenge is finding a format that reduces memory consumption without sacrificing too much capability.
2. KV cache
The KV cache stores attention information from previously processed tokens so it can be reused during generation.
Its memory requirements grow with context length.
A model might load comfortably into 16GB with a short prompt but run out of memory when processing a much longer conversation.
This makes large-context support one of the hardest parts of the project.
3. Multi-Token Prediction (MTP)
MTP uses speculative decoding to propose multiple future tokens before the main model verifies them.
When several proposals are accepted, the engine can generate more tokens with fewer sequential decoding steps.
The downside is additional memory and computational overhead.
Keeping MTP enabled alongside a large context window on a 16GB GPU requires careful memory management. The actual speed benefit will also depend on acceptance rates and implementation efficiency.
4. Runtime overhead
The inference engine needs memory beyond the model weights and KV cache.
Activations, temporary buffers, attention operations and other internal allocations all compete for the same VRAM.
This means a checkpoint that appears to fit within 16GB does not automatically guarantee that the complete inference setup will run successfully.
Which GPUs could benefit?
The proposed compatibility includes:
RTX 5060 Ti 16GB
RTX 5070 Ti 16GB
RTX 5080 16GB
Older NVIDIA GPUs with 16GB VRAM
This matters because many local AI developers already own 16GB graphics cards but struggle to run larger models while maintaining long contexts and additional inference features.
A successful implementation could make Qwen3.8-27B more accessible for local coding, reasoning and agentic workflows.
It could also give developers another option for experimenting with larger models without depending entirely on cloud APIs.
What I want to see from the first release
The project is still in development, so planned hardware compatibility should not be confused with confirmed performance.
Several questions remain:
Which quantisation formats will be supported?
How much VRAM will the complete setup consume?
What context lengths will remain practical?
Can MTP stay enabled at larger context sizes?
What sustained decode speeds can users expect?
How much does generation slow down as context grows?
How well does the model perform on coding, reasoning and agentic tasks?
I’d particularly like to see direct comparisons with MTP enabled and disabled.
A 12GB RTX 3060 is reportedly generating around 50 tokens/s with Qwen3.8-35B-A3B.
Prompt processing reaches approximately 550 tokens/s, using just 16GB of system RAM alongside the GPU.
No RTX 3090. No multi-GPU server. Just a consumer GPU, llama.cpp and a carefully configured inference stack.
The interesting part isn’t simply the speed. It’s how a 35B-parameter model can run on hardware that cannot hold the entire model in VRAM.
The reported setup
GPU: NVIDIA RTX 3060, 12GB VRAM
System RAM: 16GB
Model: Qwen3.8-35B-A3B
Quantisation: Q4_K_M
Inference engine: llama.cpp
Decode: ~50 tokens/s
Prefill: ~550 tokens/s
Configured context: 262K tokens
Attention: Flash Attention
KV cache: Q8
CPU offloading: 26 MoE layers
It’s important to distinguish the two speed figures.
The 550 tokens/s is prefill speed, measuring how quickly the model processes incoming text. The 50 tokens/s is decode speed, measuring how quickly it generates new tokens.
They describe different parts of inference, so they shouldn’t be compared as if they were the same measurement.
How does a 35B model fit into this setup?
The answer starts with its Mixture-of-Experts architecture.
Qwen3.8-35B-A3B has approximately 35 billion total parameters, but activates around 3 billion parameters per token.
Instead of sending every token through every expert, the model’s routing mechanism selects the experts needed for that particular computation.
This reduces the amount of computation required during generation.
However, 3B active parameters does not mean the model only needs memory for 3B parameters. The full collection of weights must remain accessible because different experts can be selected as the model processes new tokens.
This is where CPU/GPU hybrid inference becomes important.
Splitting the workload between GPU and CPU
The RTX 3060 contributes 12GB of VRAM, while the system provides another 16GB of RAM.
With llama.cpp, the developer distributes model execution across both memory pools rather than trying to fit everything into the GPU.
The reported configuration offloads 26 MoE layers to the CPU, allowing the system to access model components that cannot all reside in VRAM.
There is a cost to this approach.
Moving data between system RAM and GPU memory requires PCIe transfers. Excessive CPU offloading can introduce latency and reduce generation speed.
The goal is to find a balance between GPU residency, CPU execution and memory-transfer overhead.
Three optimisations help make it work
1. Q4_K_M quantisation
The model uses four-bit GGUF quantisation to reduce the amount of memory required for its weights.
This makes a much larger model accessible on limited hardware, although lower precision can introduce quality losses. The impact needs to be checked on actual reasoning, coding and instruction-following tasks.
2. Flash Attention
Attention calculations can become increasingly expensive as the context grows.
Flash Attention improves memory efficiency by reducing unnecessary memory transfers and avoiding the need to materialise large intermediate attention matrices in full.
This helps reduce memory pressure during inference.
3. Q8 KV cache
The KV cache stores information from previously processed tokens so the model can reuse it when generating subsequent tokens.
Its memory requirements increase with context length.
Using an 8-bit cache reduces its memory footprint compared with higher-precision formats, freeing up additional memory for inference.
These optimisations address different bottlenecks: model weights, attention computation and context storage.
The 262K context claim needs context
The developer also reports a configured context window of up to 262K tokens.
That is interesting, but a configured maximum does not mean the system can maintain 50 tokens/s throughout the entire context window.
🐼 0xShibu (Ø,G) .ink retweeted
Repost if you still hold your $DOG
🐼 0xShibu (Ø,G) .ink retweeted
Repost if you still hold your $DOG 🫡🧡🐕
🐼 0xShibu (Ø,G) .ink retweeted
We're giving away a Hogwarts Legacy custom wrapped NVIDIA GeForce RTX 5090 to celebrate its official release with DLSS 5 🪄
To enter:
🟢 Share this post
🟢 Comment #RTXON
Terms and Conditions: nvidia.com/en-us/geforce/con…
Encrypted NFTs are more than hidden metadata—they’re a step toward digital ownership with real privacy. Fungo Labs using FHE on Zama makes that idea tangible: seen by all, known by one. My Genesis application is sealed. 🔒🟨 @FungoLabs
1
33
🐼 0xShibu (Ø,G) .ink retweeted
We just hit 20,000. Thank you!
Whitelist reviews are ongoing, so keep an eye on the checker for your status.
🟨 We’ll have a lot more to share about Fungo Launchpad very soon.
We’re just getting started.
for me i bought this one : safetrade.com/exchange/QUANT…
⚠️ WARNING: You might be buying the wrong $QTC
– @QuantusNetwork ($QTC): post-quantum PoW blockchain, backed by Naval, Mert, Balajis, ...
– Qubitcoin (QTC): unknown project
📌 On SafeTrade:
– QTC/USDT ❌ is Qubitcoin
– QUANTUS/USDT ✅ is the real Quantus: safetrade.com/exchange/QUANT…
Because of the $QTC ticker clash, SafeTrade had to list @QuantusNetwork as QUANTUS (but its official ticker is QTC).
📌 On X:
The $QTC cashtag is showing Qubitcoin data, not Quantus. Don't trust the price you see on the cashtag.
✅ Check it yourself:
1. Go to coingecko.com/
2. Search for QTC or Quantus
3. Pick the project named Quantus with the ticker $QTC.
4. Open the Markets tab: it shows every pair and exchange where it's trading.
❗If you already bought the wrong one:
Your money isn't gone. You're holding Qubitcoin and can sell it back on the QTC/USDT pair on SafeTrade.
Share this so nobody else apes the wrong coin. 🙏
Would it be possible to work with @safetrade on a clearer solution? It would really help traders avoid buying the wrong coin: @QuantusNetwork @n13 @YuviLightman @JoeMattia
Thank you, everyone.
End of Untitled is the beginning of it.
The numbers: 414 payments from 66 people in 23 hours. 223 different topics. Proud to see how ideas brings us together.
The portion of the funds donated to @fndzionaversano to support art and artists.
Thank you @jackbutcher for inspiration.
Hey @commonsmade this is an interesting concept, i am now vouching for @ASPRO_22 and my friends which is kind of cool, let's see where this goes!
3
72
🐼 0xShibu (Ø,G) .ink retweeted
What would an airdrop look like if you could only give it to others?
commonsmade.com/vouch




















