# Large language model

A large language model is a machine learning model, typically a neural network, trained on vast text to generate, summarize, translate, and analyze text, forming the basis of modern chatbots.

A large language model (LLM) is a [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) model (typically a [neural-network](https://www.wikiprompt.org/wiki/neural-network)) trained on a vast amount of text for [natural-language-processing](https://www.wikiprompt.org/wiki/natural-language-processing) tasks, especially language generation. LLMs can typically generate, summarize, translate, and analyze text in many contexts. They are the basis for many modern chatbots, such as [chatgpt](https://www.wikiprompt.org/wiki/chatgpt), [claude](https://www.wikiprompt.org/wiki/claude), [gemini](https://www.wikiprompt.org/wiki/gemini), [grok](https://www.wikiprompt.org/wiki/grok), [qwen](https://www.wikiprompt.org/wiki/qwen), and [deepseek](https://www.wikiprompt.org/wiki/deepseek). LLMs are typically based on the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture. Generative pre-trained transformers (GPTs) are a type of LLM that is pre-trained to predict the next word, often fine-tuned to follow instructions and behave as assistants. Biased or inaccurate training data can make an LLM's output less reliable. A software harness can provide LLMs with memory and access to tools such as web search. Benchmark evaluations for LLMs attempt to measure model reasoning, factual accuracy, alignment, and safety.

## History

Before the emergence of transformer-based models in 2017, some language models were considered large relative to the computational and data constraints of their time. In the early 1990s, IBM's statistical models pioneered word alignment techniques for machine translation, laying the groundwork for corpus-based language modeling. In 2001, a smoothed [n-gram](https://www.wikiprompt.org/wiki/n-gram) model, such as those employing Kneser–Ney smoothing, trained on 300 million words, achieved state-of-the-art perplexity on benchmark tests. During the 2000s, with the rise of widespread internet access, researchers began compiling massive text datasets from the web ("web as corpus") to train statistical language models.

Moving beyond n-gram models, researchers started in 2000 to use neural networks as language models. Following the breakthrough of deep neural networks in image classification around 2012, similar architectures were adapted for language tasks. This shift was marked by the development of word embeddings (e.g., [word2vec](https://www.wikiprompt.org/wiki/word2vec) by Mikolov in 2013) and [seq2seq](https://www.wikiprompt.org/wiki/sequence-to-sequence) models using [lstm](https://www.wikiprompt.org/wiki/lstm). In 2016, Google transitioned its translation service to [neural machine translation](https://www.wikiprompt.org/wiki/neural-machine-translation) (NMT), replacing statistical phrase-based models with deep recurrent neural networks. These early NMT systems used LSTM-based encoder-decoder architectures, as they preceded the invention of transformers.

At the 2017 NeurIPS conference, Google researchers introduced the transformer architecture in their landmark paper "[Attention Is All You Need](https://www.wikiprompt.org/wiki/attention-is-all-you-need)". This paper's goal was to improve upon 2014 seq2seq technology, and was based mainly on the [attention mechanism](https://www.wikiprompt.org/wiki/attention-mechanism) developed by Bahdanau et al. in 2014. The following year in 2018, [BERT](https://www.wikiprompt.org/wiki/bert) was introduced and quickly became "ubiquitous". Though the original transformer has both encoder and decoder blocks, BERT is an encoder-only model. Academic and research usage of BERT began to decline in 2023, following rapid improvements in the abilities of decoder-only models (such as GPT) to solve tasks via prompting.

Although decoder-only [GPT-1](https://www.wikiprompt.org/wiki/gpt-1) was introduced in 2018, it was [GPT-2](https://www.wikiprompt.org/wiki/gpt-2) in 2019 that caught widespread attention because OpenAI claimed to have initially deemed it too powerful to release publicly, out of fear of malicious use. [GPT-3](https://www.wikiprompt.org/wiki/gpt-3) in 2020 went a step further and as of 2025 is available only via API with no offering of downloading the model to execute locally. But it was the consumer-facing chatbot [ChatGPT](https://www.wikiprompt.org/wiki/chatgpt) in late 2022 that received extensive media coverage and public attention by 2023. The 2023 [GPT-4](https://www.wikiprompt.org/wiki/gpt-4) was praised for its increased accuracy and as a "holy grail" for its multimodal capabilities. OpenAI did not reveal the high-level architecture and the number of parameters of GPT-4. The release of ChatGPT led to an uptick in LLM usage across several research subfields of computer science, including robotics, software engineering, and societal impact work. In 2024, OpenAI released the reasoning model [OpenAI o1](https://www.wikiprompt.org/wiki/openai-o1), which generates long chains of thought before returning a final answer. Many LLMs with parameter counts comparable to those of OpenAI's GPT series have been developed.

Since 2022, open-source models (those with their source code and weights made publicly available) have been gaining popularity, especially at first with [BLOOM](https://www.wikiprompt.org/wiki/bloom) and [LLaMA](https://www.wikiprompt.org/wiki/llama), though both have restrictions on the field of use. [Mistral AI](https://www.wikiprompt.org/wiki/mistral-ai)'s open-weight models [Mistral 7B](https://www.wikiprompt.org/wiki/mistral-7b) and [Mixtral 8x7B](https://www.wikiprompt.org/wiki/mixtral-8x7b) have a more permissive Apache License. In January 2025, [DeepSeek](https://www.wikiprompt.org/wiki/deepseek) released DeepSeek R1, a 671-billion-parameter open-weight model that performs comparably to OpenAI o1 but at a much lower price per token for users.

Since 2023, many LLMs have been trained to be multimodal, having the ability to also process or generate other types of data, such as images, audio, or 3D meshes. Open-weight LLMs have become more influential since 2023. Per Vake et al. (2025), community-driven contributions to open-weight models improve their efficiency and performance via collaborative platforms such as [Hugging Face](https://www.wikiprompt.org/wiki/hugging-face).

## How they work

LLMs are trained in a [pretraining](https://www.wikiprompt.org/wiki/pretraining) phase on vast, largely uncurated text corpora using a self-supervised objective, learning to predict the next token given prior context. This produces an autoregressive model capable of generating fluent text one token at a time. Pretrained "base" models are typically adapted further through [fine-tuning](https://www.wikiprompt.org/wiki/fine-tuning) and reinforcement learning from human feedback to follow instructions and converse safely, a process central to models like ChatGPT and Claude. The amount of text a model can process in one interaction is bounded by its [context-window](https://www.wikiprompt.org/wiki/context-window), which has grown from roughly a few thousand tokens in early models to over a million in some 2025-era systems.

## Scale and capability

Empirical relationships known as [scaling-laws](https://www.wikiprompt.org/wiki/scaling-laws) describe how a model's loss improves predictably as its parameter count, training data, and compute increase together. Some capabilities, such as multi-step arithmetic or following complex instructions, appear to emerge relatively abruptly past certain scale thresholds, a phenomenon called [emergent-abilities](https://www.wikiprompt.org/wiki/emergent-abilities), though researchers dispute how much of this is a real property of the models versus an artifact of how capabilities are measured.

## Limitations

LLMs are prone to [hallucination](https://www.wikiprompt.org/wiki/hallucination), producing fluent but factually incorrect statements with no signal to the user that the output is unreliable. Because they generate text based on learned statistical patterns rather than verified knowledge, they can also reproduce biases present in training data and remain vulnerable to adversarial prompting techniques such as jailbreaks. Mitigations include grounding outputs in retrieved external documents, a technique known as retrieval-augmented generation, and increasingly, "reasoning models" that use additional test-time computation to check their own work before answering.

## Recent developments

Since 2023, the field has seen a rapid expansion of open-weight models, with releases such as Mistral 7B, Mixtral 8x7B, and DeepSeek R1 (January 2025) offering competitive performance at lower costs. The introduction of reasoning models like OpenAI o1 in 2024 marked a shift toward models that generate extended chains of thought before answering. Multimodal capabilities have become standard, and community-driven contributions via platforms like Hugging Face have accelerated improvements in open-weight models.

---
Source: https://www.wikiprompt.org/wiki/large-language-model
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-10-23T02:24:22.19189+00:00