Jorge De Castro | EA1FWI

Microsoft trained a language model on 4 TRILLION tokens using just three possible weight values: -1, 0, and +1. And the GitHub repo behind it has crossed 41,000 stars. It's called BitNet, and the engineering is fascinating. Most language models rely on weights stored in formats like FP16 or BF16, requiring 16 bits per weight. BitNet b1.58 uses ternary weights, meaning each quantized weight has only three possible states. That's where the 1.58-bit name comes from. Three states contain approximately 1.58 bits of information. Microsoft didn't simply take an existing full-precision model and compress it after training. Its BitNet b1.58 2B4T model was trained from scratch using ternary weights during the forward pass, with 8-bit activations. The model has roughly 2.4 billion parameters and was pretrained on 4 trillion tokens. The results are impressive. In Microsoft's published evaluation, the non-embedding weights occupy approximately 0.4 GB. The model also achieves competitive results against several full-precision language models in the 1B to 2B class. And Microsoft open-sourced the inference engine needed to exploit this architecture. It's called bitnet.cpp. The implementation includes specialized CPU kernels, packed ternary-weight representations, lookup-table computation, and GPU inference support. In the team's original CPU experiments: → Up to 6.17× faster inference on x86 → Up to 5.07× faster inference on ARM → Up to 82.2% lower energy consumption on x86 Those results come from controlled tests using synthetic model configurations, rather than a guarantee for every deployed model. What makes BitNet particularly interesting is that the efficiency comes from the model architecture, the training method, and the inference implementation working together. The model weights, research papers, and inference framework are publicly available. Open source under MIT. GitHub: github.com/microsoft/BitNet