# 1.58-bit large language model

A 1.58-bit large language model is a type of large language model that uses ternary weights (−1, 0, +1) to reduce memory and computational costs, enabling efficient inference on low-power hardware while maintaining performance comparable to full-precision models.

A 1.58-bit large language model (also known as a ternary LLM) is a type of [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) designed for computational efficiency. Unlike conventional models that store weights as 16-bit floating-point numbers (such as FP16 or BF16), a 1.58-bit model restricts each weight to one of three values: −1, 0, or +1. This ternary representation reduces the memory footprint by roughly an order of magnitude and replaces expensive multiplication operations with simpler additions, speeding up both [inference](https://www.wikiprompt.org/wiki/inference) and training. The name derives from the information-theoretic fact that a three-state value carries approximately log₂3 ≈ 1.58 bits of information. These models are sometimes loosely called 1-bit LLMs, though that term can also refer to true binary models with only −1 and +1 weights.

Research has shown that for models up to several billion parameters, 1.58-bit LLMs achieve performance comparable to their full-precision counterparts on a range of tasks. This efficiency could allow powerful AI to run on less specialized, lower-power hardware, addressing energy demands in edge and mobile applications. The approach has attracted attention from both academic and industrial researchers, with Microsoft being a prominent contributor.

## BitNet and Native Training

In 2024, a team of researchers at [microsoft](https://www.wikiprompt.org/wiki/microsoft) led by Shuming Ma and Furu Wei introduced BitNet b1.58, a 1.58-bit LLM that demonstrated performance comparable to the 16-bit [llama-2](https://www.wikiprompt.org/wiki/llama-2) model at similar scale. The key innovation was not post-training [quantization](https://www.wikiprompt.org/wiki/quantization) but rather a new BitLinear transform that replaced the standard nn.Linear layer in the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture. This allowed the model to be trained natively in ternary precision, avoiding the accuracy loss typically associated with quantizing a pre-trained model. The paper, titled "The Era of 1-bit LLMs," argued that such models could open a new era of efficient AI.

In 2025, Microsoft researchers released an open-weights and open-inference-code model called BitNet b1.58 2B4T, with 2 billion parameters trained on 4 trillion tokens. This model showed competitive performance with full-precision models of similar size, further validating the approach. The release included code for efficient inference, making it accessible to the broader community.

## Post-Training Quantization and Fine-Tuning

While BitNet is trained natively in 1.58-bit precision, many existing models are trained in full precision, and converting them post hoc is desirable to save costs. In 2024, researchers at [hugging-face-transformers](https://www.wikiprompt.org/wiki/hugging-face-transformers) reported a method to gradually ramp up 1.58-bit quantization during [fine-tuning](https://www.wikiprompt.org/wiki/fine-tuning) of an existing model, eventually reaching ternary weights. This approach allows practitioners to adapt pre-trained models to low-bit precision without starting from scratch, though it still requires a fine-tuning step.

## Performance and Scaling Laws

Empirical studies indicate that for models up to several billion parameters, 1.58-bit LLMs perform comparably to full-precision models on benchmarks such as language modeling and downstream tasks. However, the picture is nuanced. Research on [scaling-laws](https://www.wikiprompt.org/wiki/scaling-laws) suggests that low-bit quantization is most favorable for undertrained models, i.e., those trained on fewer tokens. As the number of training tokens increases, the limitations of low-bit precision become more apparent, potentially leading to larger performance gaps. This has implications for the design of future models, as training data volumes continue to grow.

## Hardware and Energy Efficiency

The primary motivation for 1.58-bit LLMs is to reduce the computational and energy costs of running large models. By eliminating multiplications, these models can run on CPUs and low-power edge devices, potentially enabling on-device AI applications. The reduced memory footprint also allows larger models to fit in limited memory, which is critical for mobile and embedded systems. Reports from ieee-spectrum and other outlets have highlighted the potential to address AI's energy demands, though practical deployment depends on hardware support for ternary operations.

## Critique and Limitations

Some researchers have pointed out that the benefits of 1.58-bit quantization may not scale indefinitely. As training token counts increase, the deficiencies of low-bit weights surface, leading to degraded performance relative to full-precision models. This is particularly relevant for very large models trained on massive datasets. Additionally, the ternary representation can limit the model's capacity to represent fine-grained information, which may affect tasks requiring high precision. The debate over scaling laws for precision remains active, with studies such as "Scaling Laws for Precision" and "Low-Bit Quantization Favors Undertrained LLMs" providing conflicting or complementary insights.

## Future Directions

Research on 1.58-bit LLMs is ongoing, with efforts focused on improving training techniques, developing better quantization methods, and exploring hardware acceleration. The open release of BitNet models by Microsoft has spurred community experimentation. As of 2025, the approach is seen as a promising avenue for making large language models more accessible and sustainable, though it is not yet a universal replacement for full-precision models. Future work may address the scaling limitations and extend ternary models to even larger parameter counts.

## See Also

- [quantization-neural-networks](https://www.wikiprompt.org/wiki/quantization-neural-networks)
- [llama-leak](https://www.wikiprompt.org/wiki/llama-leak)
- [inference](https://www.wikiprompt.org/wiki/inference)
- [fine-tuning](https://www.wikiprompt.org/wiki/fine-tuning)

## References

1. Ma, Shuming; et al. (2024). "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits." arXiv:2402.17764.
2. Ma, Shuming; et al. (2025). "BitNet b1.58 2B4T Technical Report." arXiv:2504.12285.
3. Wang, Hongyu; et al. (2023). "BitNet: Scaling 1-bit Transformers for Large Language Models." arXiv:2310.11453.
4. Kumar, Tanishq; et al. (2024). "Scaling Laws for Precision." arXiv:2411.04330.
5. Ouyang, Xu; et al. (2024). "Low-Bit Quantization Favors Undertrained LLMs." arXiv:2411.17691.
6. Hutson, Matthew (2024). "1-bit LLMs Could Solve AI's Energy Demands." IEEE Spectrum.
7. Morales, Jowi (2025). "Microsoft researchers build 1-bit AI LLM with 2B parameters." Tom's Hardware.

## External links

- [Wikipedia: 1.58-bit large language model](https://en.wikipedia.org/wiki/1.58-bit_large_language_model)

---
Source: https://www.wikiprompt.org/wiki/1-58-bit-large-language-model
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-11-09T12:23:28.233432+00:00