A bigram, also known as a digram, is a sequence of two adjacent elements drawn from a string of tokens, which are typically letters, syllables, or words. In formal terms, a bigram is an n-gram with n equal to 2. The frequency distribution of every bigram in a given string is commonly used for simple statistical analysis of text across many applications, including computational linguistics, cryptography, and speech recognition.
Bigrams serve as a foundational building block in understanding and processing natural language. By examining the co-occurrence of adjacent items, researchers and engineers can capture local relationships within text, which are useful for tasks ranging from predicting the next word in a sequence to identifying the language of a document. They represent a middle ground between single-character or single-word analysis and longer n-gram models, balancing computational efficiency with descriptive power.
Applications
Bigrams are employed in a variety of domains where the statistical properties of text are relevant. In computational linguistics and natural language processing, they are integral to many language models, particularly those used in automatic speech recognition. These models rely on the probability of a word given the preceding word, a direct application of bigram frequency, to disambiguate between possible phonetic interpretations and improve transcription accuracy.
In cryptography, bigram frequency attacks are a classical technique used to solve cryptograms, including simple substitution ciphers. By comparing the frequency of letter pairs in encrypted text against known distributions from the suspected language, cryptanalysts can deduce likely letter mappings. This method falls under broader frequency analysis, which exploits the non-uniform distribution of characters and their combinations in natural languages.
For statistical language identification, bigram frequency provides a straightforward approach. By building profiles of characteristic bigram distributions for different languages, systems can classify an unknown text sample by measuring how closely its bigram frequencies match each profile. This technique is effective even with relatively short text snippets.
In recreational linguistics, or logology, bigrams are a subject of playful exploration. Enthusiasts seek English words that begin with every possible bigram, or attempt to discover words containing a string of repeated bigrams, such as the word 'logogogue'. These activities highlight the curious patterns and limitations of word formation in the English language.
Additionally, gappy bigrams, also called skipping bigrams, are word pairs that allow gaps between the constituent words. This variation permits the modeling of broader dependencies, such as avoiding connecting words or simulating syntactic relationships akin to those in a dependency grammar. Such bigrams extend the basic concept to capture more distant interactions within a sentence.
Bigram Frequency in the English Language
The frequency of the most common letter bigrams in a large English corpus follows a distinct distribution. The most frequent bigram is 'th', appearing at 3.56% of all bigram occurrences, followed by 'he' at 3.07%, 'in' at 2.43%, 'er' at 2.05%, and 'an' at 1.99%. Other frequent bigrams include 're' at 1.85%, 'on' at 1.76%, and 'at' at 1.49%.
The distribution continues with 'en' at 1.45%, 'nd' at 1.35%, 'ti' at 1.34%, 'es' at 1.34%, 'or' at 1.28%, and 'te' at 1.20%. The bigram 'of' appears at 1.17%, with 'ed' also at 1.17%. Lower but still notable frequencies include 'is' at 1.13%, 'it' at 1.12%, 'al' at 1.09%, 'ar' at 1.07%, 'st' at 1.05%, and 'to' at 1.05%.
Further down the list, 'nt' has a frequency of 1.04%, 'ng' at 0.95%, 'se' at 0.93%, 'ha' at 0.93%, 'as' at 0.87%, 'ou' at 0.87%, 'io' at 0.83%, 'le' at 0.83%, and 've' at 0.83%. The bigrams 'co' and 'me' each occur at 0.79%, followed by 'de' and 'hi' at 0.76% each. Other small percentages include 'ri' at 0.73%, 'ro' at 0.73%, 'ic' at 0.70%, 'ne' at 0.69%, 'ea' at 0.69%, 'ra' at 0.69%, and 'ce' at 0.65%.
This frequency data is crucial for applications such as cryptography, where identifying the most common pairs helps in cracking ciphers, and for optimizing keyboard layouts or text compression algorithms.
Relationship to Other N-grams
Bigrams are a specific case of the general n-gram technique, where n denotes the number of elements in the sequence. Unigrams (n=1) consider individual tokens, while trigrams (n=3) and higher-order n-grams capture longer contexts. The choice of n involves a trade-off: larger n values can model more complex dependencies but require exponentially more data to estimate reliably due to data sparsity.
In modern Machine learning applications, particularly within Natural language processing pipelines, n-gram statistics have largely been superseded by Neural network models such as Transformer (architecture) architectures. However, bigrams remain relevant in foundational language model research and in scenarios where computational resources are limited or the need for interpretability is high.
Computational Aspects
The computation of bigram frequencies is straightforward and efficient. For a given string of tokens, one can iterate through the sequence, counting each pair of adjacent tokens. The resulting counts can then be normalized to produce probabilities. This process forms the basis of Markov models, where the probability of the next token is conditioned on the current token.
Bigrams are also used in Data Augmentation techniques and in Loss Functions for certain tasks, though these uses are less common than in traditional statistical methods. The concept of Beam Search in sequence generation often relies on probability estimates from n-gram models, though modern systems typically use more advanced Neural network-based scoring.
See Also
- Digraph (orthography)
- Letter frequency
- N-gram
- Frequency analysis