Professor at IST Austria
Vienna
Joined May 2022
- Tweets180
- Following311
- Followers1.9K
- Likes97
Dan Alistarh retweeted
We have noticed recent comparisons published by Unsloth that do not align with our results for DeepSeek-V4.1-Flash-1017. So just if it is not obvious: we did of course not modify any hyperparameters to favor our models. We used standard settings appropriate for a model of this scale (763B parameters) and its 1M context length capability: a 128K context window and a reasoning effort set to 50.
To the best of our knowledge, the discrepancy in performance stems from a difference in benchmark constraints - not in the model being worse. Unsloth's comparisons has a strict 32K token reasoning budget, whereas our benchmarks allow up to 128K output tokens. This significant difference in compute explains the variance in results; it is not due to any hidden adjustments on our part. Furthermore, imposing such a low reasoning budget (32K) appears inconsistent for a model designed with a 1M context window and 763B params.
Our primary goal is open collaboration and transparency. All of our code is available on GitHub, and our papers are on arXiv. We are genuinely happy that other quantizations are performing well, as this was a key motivation for publishing these models in the first place. We invite anyone with the compute resources to reproduce our results using our exact settings (128K output budget, reasoning effort 50) for independent verification. Our previous models were benchmarked by third parties like @bnjmn_marie, who do great work for the community. We welcome rigorous testing of our claims. Furthermore, we are currently running the comparisons on the models within our setup and will post results here soon.
Thanks and stay tuned!
Dan Alistarh retweeted
Disaggregated quantization for LLMs 🧵
LLM prefill and decode can be processed by separate weights in different precisions. Boosts speed AND accuracy. Offloading makes it zero-overhead on long sequences for dense models. Applicable to existing quantized checkpoints.
Paper: arxiv.org/abs/2609.26333
Code: github.com/IST-DASLab/disagg…
Qwen3.8 NVFP4 prefillers: huggingface.co/ISTA-DASLab/Q…
Dan Alistarh retweeted
We’re excited to share our #ICML2026 paper on optimizer robustness in Async Pipeline Parallel LLM pretraining
Main message: with the right PP schedule, optimizer choice, and Error Feedback correction, Async PP can match sync training quality even at 10B MoE scale
[1/10]
ALT Training loss on a 10B MoE model trained for 200B FineWeb-Edu tokens. Both asynchronous runs remain stable throughout training, with Async PP + Error Feedback closing the final gap to the synchronous baseline entirely.
Weight-only quantization powers local LLMs like llama.cpp or Ollama. But SOTA quantized accuracy requires complex kernels that are notoriously hard to implement.
Can we get SOTA accuracy and keep things simple? Our new GSQ (Gumbel-Softmax Quantization) method says yes. 🧵
🚀 Scalar structure means GSQ can use highly-optimized low-precision GEMM kernels:
Using vLLM + the excellent Humming kernels (github.com/inclusionAI/hummi…) on L40s GPUs, 2-bit GSQ-quantized Llama-3.1-70B hits up to 6.2× throughput vs BF16!
As always, our release is fully open:
💻 Paper and code: arxiv.org/pdf/2604.18556
@huggingface : huggingface.co/collections/I…
Credits: Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan, Michael Helcig, Eldar Kurtić, and Max Kleinegger
We thank @RedHat_AI and @verdacloud for the support!
Dan Alistarh retweeted
TurboQuant has drawn a lot of attention recently, but the accompanying evals didn't tell the full story.
So we ran what I believe is the first comprehensive study of TurboQuant: where it helps, where it falls short, and how it impacts accuracy, latency, and throughput.
Findings:
Dan Alistarh retweeted
🚀 Our #ICLR2026 paper: The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm
We show GPTQ is exactly Babai's nearest plane algorithm, giving a geometric view of LLM quantization and inspiring improved PTQ methods.
Efficient GPTQ Triton kernels included!
Speedrunning GPT-2 is now routine thanks to @karpathy.
But can we speedrun GPT3-175B?
We attempted to match accuracy on a <$10K budget; while we didn't quite reach it, our first results show that quality data, engineering, and native FP4 can get close.
Details in 🧵
Credit goes to Erik Schultheis, Matin Ansaripour @matin_asp, Andrei Panferov @black_samorez, and George Vlassis @gvlassis98. Thanks to @verdacloud (particularly Paul Chang) for compute support, and Jen Iofinova for safety testing. This work was supported by FWF BilAI and SwissAI.
As usual, our release is fully open:
- Full report: github.com/IST-DASLab/Clover…
- QuartetII paper: arxiv.org/abs/2601.22813
- Tokenized data: huggingface.co/datasets/dasl…
- Model: huggingface.co/daslab-testin…
- Codebase: github.com/IST-DASLab/Clover…
We’re releasing MatGPTQ (Matryoshka GPTQ) an accurate and efficient post-training quantization (PTQ) method that jointly optimizes a single model across multiple bit-widths, producing a sliceable checkpoint that can be deployed across diverse hardware and memory budgets. [1/4]
On the systems side, we provide custom CUDA kernels for Ampere GPUs and vLLM integration (see below for OSS code) to fully leverage nested models:
- 3x-5.6x kernel speedups when memory-bound
- 1.5x-3.5x end-to-end speedups in vLLM
Credits go to @maxkleinegger and Elvir Crnčević!
- Paper: arxiv.org/abs/2602.03537
- Models: huggingface.co/collections/I…
- Code: github.com/IST-DASLab/MatGPT…
