# CPM-2

CPM-2 is a Chinese pretrained language model with 198 billion parameters, developed by the Beijing Academy of Artificial Intelligence (BAAI), released in 2021 as a successor to CPM-1, focusing on large-scale generative and understanding tasks.

CPM-2 is a large-scale pretrained language model developed by the Beijing Academy of Artificial Intelligence (BAAI). Released in 2021, it is the successor to CPM-1 and is notable for its 198 billion parameters, making it one of the largest Chinese-language models at the time. CPM-2 is designed to handle a wide range of natural language processing tasks, including text generation, comprehension, and dialogue, with a focus on the Chinese language.

Unlike many models that rely solely on a single architecture, CPM-2 employs a mixture of experts (MoE) approach within a [transformer](https://www.wikiprompt.org/wiki/transformer) framework. This design allows the model to activate only a subset of its parameters for each input, improving computational efficiency while maintaining high performance. The model was trained on a diverse corpus of Chinese text, including web pages, books, and other sources, and it demonstrated strong results on benchmarks such as CLUE (Chinese Language Understanding Evaluation).

## Architecture and Training

CPM-2 is built on a [transformer](https://www.wikiprompt.org/wiki/transformer) architecture, specifically using a decoder-only structure for generative tasks. Its 198 billion parameters are organized into multiple expert layers, where each token is routed to a small number of experts, reducing the effective computational cost per inference. The model uses a vocabulary of Chinese subword units and incorporates techniques like [positional-encoding](https://www.wikiprompt.org/wiki/positional-encoding) and [layer-normalization](https://www.wikiprompt.org/wiki/layer-normalization) to stabilize training.

The training process involved a two-stage approach: first, a dense pretraining phase on a large corpus, followed by a sparse fine-tuning phase using the MoE structure. This method allowed BAAI to scale the model beyond what would be feasible with a dense architecture alone. The training data comprised over 200 gigabytes of Chinese text, and the model was trained on a cluster of GPUs, though specific hardware details are not publicly disclosed.

## Capabilities and Applications

CPM-2 excels in various [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) tasks, including text completion, summarization, question answering, and dialogue generation. In evaluations, it achieved state-of-the-art results on several Chinese benchmarks, such as the CLUE leaderboard, outperforming previous models like GPT-3 in certain Chinese-specific tasks. The model's ability to handle long contexts and generate coherent, contextually relevant text made it suitable for applications in content creation, customer service, and educational tools.

One notable feature is its bilingual capability, as CPM-2 can also process English text, though its primary focus is Chinese. This versatility stems from the inclusion of English data in its training corpus, allowing cross-lingual transfer. The model is available in two versions: a full 198B parameter version and a smaller 11B parameter version, the latter being more accessible for research and deployment.

## Impact and Reception

CPM-2 was released as an open-source model, with its weights and code made available to the research community. This openness contributed to its adoption in academic and industrial settings, particularly in China, where it served as a foundation for further fine-tuning on domain-specific tasks. The model's release also sparked discussions about the environmental and computational costs of training large models, as well as the need for efficient inference methods.

Compared to contemporaneous models like [openai](https://www.wikiprompt.org/wiki/openai)'s GPT-3, CPM-2 demonstrated that competitive performance could be achieved with a focus on a specific language and with a more efficient MoE architecture. Its success encouraged further research into sparse models and contributed to the development of subsequent models in the CPM series, such as CPM-3.

## Limitations and Ethical Considerations

Like other large language models, CPM-2 has limitations, including potential biases in its training data, which can lead to biased outputs. It may also produce incorrect or nonsensical responses, especially in niche topics. The model's large size poses challenges for deployment, requiring significant computational resources for inference, which limits its use in resource-constrained environments.

BAAI has acknowledged these issues and recommends careful evaluation before deploying the model in sensitive applications. The organization also emphasizes responsible use, encouraging researchers to consider the ethical implications of generated content. As of 2024, CPM-2 remains a reference point in the development of Chinese-language AI, though it has been superseded by more advanced models with even larger parameter counts and improved capabilities.

## Legacy and Future Directions

The release of CPM-2 marked a milestone in the field of [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence), particularly for non-English language models. It demonstrated that large-scale pretraining could be effectively applied to languages other than English, paving the way for similar efforts in other languages. The MoE architecture used in CPM-2 has influenced subsequent model designs, including those from [alibaba-damiao-academy](https://www.wikiprompt.org/wiki/alibaba-damiao-academy) and other Chinese AI research institutions.

Future iterations of the CPM series have built upon CPM-2's foundation, incorporating newer techniques such as reinforcement learning from human feedback ([rlaif](https://www.wikiprompt.org/wiki/rlaif)) and more efficient training methods. CPM-2's legacy lies in its demonstration of scalable, open-source AI development outside the Western tech ecosystem, contributing to a more diverse global AI landscape.

---
Source: https://www.wikiprompt.org/wiki/cpm-2
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T16:28:35.462888+00:00