Eric Henderson

Qwen3.8-27B reportedly went from 82 tokens/s to 381 tokens/s on a single RTX 3090 with 24GB VRAM. No GPU upgrade. No hardware replacement. The developer achieved roughly 4.6× the original generation speed by changing how the model handles decoding and reuses information from its input. The performance progression: • Initial setup: ~82 tok/s • Optimised MTP: ~114 tok/s • DFlash2 + context lookup: ~138 tok/s • Optimised lookup verification: ~381 tok/s The final jump is especially interesting because it combines speculative decoding with a technique that takes advantage of text already present in the prompt. 1. DFlash2 reduces the cost of sequential decoding A standard autoregressive model generates one token at a time. Each new token depends on the tokens generated before it. That sequential process limits how quickly output can be produced. Speculative decoding takes a different approach. A draft mechanism proposes several tokens ahead, then the main model verifies those proposals. When the proposals are accepted, multiple tokens can be produced without requiring a separate full decoding step for each one. DFlash2 uses this approach to reduce the number of sequential generation steps. 2. Context lookup avoids regenerating text that already exists This is where the developer found another opportunity to improve performance. Imagine giving Qwen3.8-27B a 25K-token document and asking it to reproduce a section of that document. The expected output may already exist somewhere in the input. Generating every token again would involve doing work that could potentially be avoided. Context lookup searches the existing prompt for matching sequences, proposes them as candidate output and lets the main model verify them. The developer optimised the verification process to handle blocks of up to 16 proposed tokens, compared with seven in the earlier setup. The reported acceptance rate was approximately 15 out of every 16 proposed tokens. That high acceptance rate makes a substantial difference when the output closely matches the source text. On the reported document-reproduction workload, the results were: • Previous DFlash2 setup: ~260 tok/s • Optimised context lookup: ~382 tok/s There is an important distinction here. 381 tok/s is not the model’s normal conversational speed. The developer reports approximately 133 tok/s for ordinary chat workloads. Context lookup is particularly effective when the expected output overlaps heavily with the input. It won’t provide the same acceleration when the model is writing original content or solving a problem that requires mostly new text. 3. Prefix caching cuts repeated prompt processing The developer also improved performance when asking multiple questions about the same large document. Normally, a new request containing that document may require the system to process the shared text again. Prefix caching allows it to reuse computations from the previous request when the prompt prefix remains unchanged. In the reported 25K-token document test, time to first token dropped from 22.4 seconds to just 0.56 seconds. That’s approximately a 40× reduction in waiting time for this repeated-prefix scenario. The system still processes the new question and generates a fresh answer. It simply avoids repeating much of the computation associated with the shared document. Where these techniques become useful The combination is particularly relevant to workloads involving: • Large-document processing and extraction • Retrieval-augmented generation • Repeated questions about the same source material • Code transformations and repository workflows • Tasks that reuse or reproduce existing text The three optimisations target different bottlenecks. DFlash2 reduces sequential decoding steps. Context lookup finds reusable text in the prompt. Prefix caching avoids recomputing an unchanged prompt prefix.