An n-gram is a contiguous sequence of n items from a given sample of text or speech. The items can be characters, syllables, words, phonemes, or even biological sequences such as base pairs in a genome. N-grams are collected from a text corpus or speech corpus and are fundamental to statistical natural language processing, enabling models to capture local patterns and dependencies in sequential data.
The term derives from Latin numerical prefixes: a unigram is a sequence of one item, a bigram (or digram) of two, a trigram of three, and so on. For larger sizes, English cardinal numbers are used, such as four-gram or five-gram. In computational biology, similar sequences are called k-mers, using Greek prefixes like monomer, dimer, trimer, or English forms like one-mer, two-mer. When the items are words, n-grams are sometimes called shingles.
Historical Development
The concept of n-gram models in language dates to 1951, when Claude Shannon discussed them in the context of information theory. Shannon demonstrated how character-level and word-level n-gram models could generate plausible English text by sampling from probability distributions of observed sequences. For example, a 3-gram character model might produce text like "in no ist lat whey cratict froure birs grocid pondenome," while a 2-gram word model could generate "the head and in frontal attack on an english writer." These early experiments established n-grams as a practical tool for modeling language statistically.
Applications in Natural Language Processing
In natural language processing, n-grams allow bag-of-words models to capture word order information that would otherwise be lost. A traditional bag-of-words representation treats a document as an unordered set of words, but n-gram features preserve local context. This capability is essential for tasks such as language modeling, spelling correction, machine translation, and text classification. N-gram language models estimate the probability of a word given the preceding n-1 words, forming the basis for many statistical approaches before the rise of neural networks.
Modern large language models built on transformer architectures have largely superseded explicit n-gram models for many tasks, but n-grams remain relevant in areas like authorship attribution, stylometry, and bioinformatics. The Google Books Ngram Viewer, which charts the frequency of n-grams across millions of digitized books, demonstrates their continued utility for cultural and linguistic analysis.
Computational Biology
In genomics, k-mers (the biological equivalent of n-grams) are used to analyze DNA and RNA sequences. A k-mer is a contiguous subsequence of length k, and k-mer frequency analysis is a standard technique for genome assembly, sequence alignment, and quality control. For instance, algorithms for DNA sequencing projects often use k-mer distributions to detect errors or estimate genome size. The choice of k affects sensitivity and specificity: smaller k-mers are more abundant but less specific, while larger k-mers provide more unique matches but may be absent due to sequencing errors.
Statistical Properties and Variants
N-gram models can be smoothed to handle unseen sequences, using techniques such as add-one smoothing or more advanced methods like Kneser-Ney smoothing. They can also be weighted by frequency, as seen in the Google n-gram corpus, which provides counts for phrases like "serve as the independent" (794 occurrences) or "ceramics collectables fine" (130 occurrences). These frequency distributions enable probabilistic predictions and are used in applications ranging from predictive text input to speech recognition.
Limitations and Extensions
A key limitation of n-grams is their inability to capture long-range dependencies beyond the fixed window size. Increasing n improves context but leads to data sparsity, as many possible sequences never appear in training data. To address this, researchers have developed backoff models, interpolation, and neural language models that learn distributed representations. Nevertheless, n-grams remain a simple, interpretable baseline and are often used in hybrid systems alongside machine learning methods.
In addition to text and speech, n-grams have been applied to music analysis, where sequences of notes or chords are treated as items, and to network security for detecting anomalies in packet sequences. Their generality makes them a versatile tool across disciplines that deal with sequential data.
See Also
- Artificial intelligence
- Machine learning
- Natural language processing
- statistical-model
- google-books-ngram-viewer