XLNet is an autoregressive Transformer (architecture) model designed for natural language processing, introduced as an improvement over BERT. It was released on 19 June 2019 under the Apache 2.0 license, with 340 million parameters and trained on 33 billion words. The model achieved state-of-the-art results on tasks including language modeling, question answering, and natural language inference, and is a notable example of Large language model architecture in [[deep-learning] research.
Unlike BERT, which uses masked language modeling, XLNet employs permutation language modeling. This approach factorizes the joint probability of a sequence in all possible orders, allowing the model to capture bidirectional context without relying on masking. This design was intended to address a limitation of BERT, where masked tokens are not seen during pretraining, creating a discrepancy between pretraining and fine-tuning.
Permutation Language Modeling
In standard autoregressive modeling, the probability of a sequence is factorized in a fixed left-to-right order. For example, the sentence "My dog is cute" is modeled as the product of conditional probabilities: Pr(My) Pr(dog|My) Pr(is|My, dog) * Pr(cute|My, dog, is). XLNet instead samples a random permutation of the token positions and factorizes the probability according to that order. For instance, with permutation 3-2-4-1, the model predicts "is" first, then "dog", then "cute", and finally "My", each conditioned on the previously predicted tokens. By training on all permutations, the model learns to use context from both directions, improving its ability to capture long-range dependencies.
Two-Stream Self-Attention
To implement permutation language modeling, XLNet uses a two-stream self-attention mechanism. The first stream, called the content stream, encodes the content of each token using standard causal self-attention. The second stream, the query stream, encodes the content of each token in the context of previously predicted tokens, using a masked cross-attention where queries come from the query stream and key-value pairs come from the content stream. This design allows the model to predict a token without seeing its own content, while still leveraging the content of other tokens in the permuted order.
Training and Performance
XLNet was trained on a large corpus of 33 billion words, using a model with 340 million parameters. The training process involved both permutation language modeling and a segment-reordering objective, which helped the model learn relationships between sentences. On release, XLNet outperformed BERT on a range of benchmarks, including the Stanford Question Answering Dataset (SQuAD), the General Language Understanding Evaluation (GLUE) benchmark, and several reading comprehension tasks. Its success demonstrated the effectiveness of permutation-based pretraining and influenced subsequent work in Generative AI and Machine learning.
Impact and Legacy
XLNet contributed to the evolution of pretrained language models by highlighting the trade-offs between autoregressive and autoencoding approaches. While BERT's masking strategy enabled bidirectional context, it introduced a pretrain-finetune discrepancy; XLNet's permutation approach offered a way to achieve bidirectional context while maintaining the autoregressive property. This idea has been incorporated into later models, such as Transformer-XL and other variants. Although newer architectures have since surpassed XLNet in performance, it remains a significant milestone in the history of Artificial intelligence and Neural network research.
See Also
- BERT (not in provided slugs, but could be linked as Transformer (architecture))
- Large language model
- Deep learning
- Machine learning