DeepSeek V3 is a Large language model developed by the Chinese AI company DeepSeek, first released on December 26, 2024. It is a Transformer (architecture)-based architecture employing a Mixture-of-Experts (MoE) design, with a total of 671 billion parameters, of which 37 billion are activated per token. The model was trained on 14.8 trillion tokens and is notable for its cost-efficient training process, reportedly using only 2.788 million H800 GPU hours.
DeepSeek V3 appeared on public LLM and media leaderboards shortly after its release, with benchmark snapshots showing consistent performance across multiple variants. The model family includes at least five variants in these snapshots, reflecting different configuration or fine-tuning states, though specific details of each variant are not publicly documented. The base model and its instruction-tuned version have been evaluated on standard academic benchmarks.
Architecture and Training
DeepSeek V3 uses a Deep learning architecture with 61 layers of transformer blocks. It incorporates multi-head latent attention (MLA) for efficient inference and employs a DeepSeekMoE structure for feed-forward layers, where each token activates only a subset of experts. The model uses a vocabulary of 128,000 tokens and supports a context length of 128K tokens.
Training was conducted using a Learning Rate Scheduling with a peak learning rate of 2.4e-3, combined with Gradient Clipping and Batch Normalization techniques adapted for large-scale training. The dataset comprised 14.8 trillion tokens, primarily in English and Chinese, sourced from web text, books, and other curated corpora. The training process utilized TSMC-manufactured H800 GPUs, with a reported cost of approximately $5.6 million for the final training run.
Performance and Benchmarks
On public benchmarks, DeepSeek V3 achieved notable scores. On the MMLU (Massive Multitask Language Understanding) benchmark, it scored 88.5% in a 5-shot setting. On the MATH-500 benchmark, it achieved 90.2% in a zero-shot setting. The model also performed well on coding tasks, scoring 82.6% on HumanEval and 67.3% on the more challenging LiveCodeBench.
In comparison to contemporaneous models, DeepSeek V3's performance was competitive with leading proprietary systems from OpenAI and Anthropic, while being significantly cheaper to train. On the GPQA (Graduate-Level Google-Proof Q&A) benchmark, it scored 59.1% in a zero-shot setting, and on the DROP benchmark, it achieved 91.6% in a 3-shot setting.
Release and Variants
The initial release on December 26, 2024, included the base model and a chat-optimized version, DeepSeek-V3-Chat. Both were made available under the MIT license, allowing commercial use and modification. The model weights were distributed through Hugging Face and other platforms, enabling broad adoption in the Generative AI community.
Public benchmark snapshots from early 2025 show at least five distinct variants of DeepSeek V3, likely representing different fine-tuning runs or quantization levels. These variants have appeared on leaderboards such as the LMSYS Chatbot Arena and the Open LLM Leaderboard, though the exact configuration of each variant is not officially documented. As of early 2025, no additional official variants beyond the base and chat versions had been announced.
Impact and Reception
The release of DeepSeek V3 generated significant attention in the Artificial intelligence community due to its combination of high performance and low training cost. It demonstrated that efficient training techniques could rival the capabilities of models developed with substantially larger budgets. The model's open-source nature under the MIT license contributed to its rapid integration into various applications and research projects.
Critics and analysts noted that DeepSeek V3's training efficiency was partly due to its MoE architecture, which reduces computational cost during inference. However, the model's reliance on a large total parameter count still requires substantial memory for deployment. The model also sparked discussions about the competitive landscape of Machine learning, particularly regarding the role of Chinese AI companies in advancing open-source models.
Technical Details and Limitations
DeepSeek V3 employs a custom Positional Encoding scheme and uses Multi-Head Attention mechanisms. It does not use traditional Dropout in its attention layers, a design choice that improves training efficiency. The model's inference supports Top-P (Nucleus) Sampling and Temperature Scaling for controlled generation.
Despite its strengths, DeepSeek V3 has limitations. It may exhibit biases present in its training data, and its performance on non-English languages beyond Chinese is less thoroughly evaluated. The model's large memory footprint limits deployment on consumer hardware, requiring Model Pruning or quantization for practical use in some settings. As of early 2025, the model has not been updated with a successor, though ongoing research at DeepSeek continues.