Inference, in artificial intelligence, is the process of running a trained model on new input to produce an output, as distinct from training, the earlier phase in which the model's parameters are learned from data. A large language model answering a user's question, an image classifier labeling a photo, and a Diffusion model generating an image from a text prompt are all performing inference. Once training finishes and a model's weights are fixed, every subsequent use of that model is an inference call.
Cost and hardware
Inference has different computational demands than training. Training a large model requires processing enormous datasets over many passes and is typically run on large clusters of GPUs or TPUs over weeks or months. Inference instead runs continuously, often serving millions of individual requests, and its cost structure depends more on latency, the time to produce a single response, and throughput, the number of requests a system can serve per unit of hardware, than on total compute consumed by a single job. Because language models generate text one token at a time, with each new token depending on all the previous ones, inference for autoregressive models is comparatively slow and memory-bound rather than compute-bound, which has driven a wave of specialized inference engineering distinct from training-focused research. NVIDIA and other chipmakers have released hardware specifically optimized for inference workloads, and cloud providers price inference and training capacity differently.
Optimization techniques
Because inference cost scales directly with usage, it is a major target for optimization. Quantization reduces the numerical precision of a model's weights, cutting memory use and often speeding up computation with a modest accuracy tradeoff. Knowledge distillation trains a smaller model to imitate a larger one, producing a model that is cheaper to run at inference time. Speculative decoding uses a small, fast draft model to propose several tokens at once, which a larger model then verifies in parallel, reducing the number of slow sequential steps needed to generate long outputs. Batching multiple users' requests together, caching previously computed intermediate results such as the attention key-value cache, and routing simpler queries to smaller models are additional common techniques for lowering the cost of serving a model at scale.
Test-time compute
Inference has traditionally been treated as a fixed, lightweight step compared to training, but the rise of reasoning models that generate extended internal reasoning before answering introduced a new dimension: spending more compute at inference time, an approach known as Test-time compute, can substantially improve output quality on difficult problems. This blurred the older assumption that only training-time compute mattered for capability, since a model can now trade inference cost for accuracy on a given problem by "thinking longer," a tradeoff that did not exist in the same way for earlier generations of models that produced a fixed amount of output regardless of task difficulty.
Economics
Because widely used AI products serve inference requests continuously, inference cost, not training cost, dominates the long-run economics of running a deployed model, and industry discussion of AI infrastructure spending increasingly centers on inference capacity as usage scales, alongside ongoing investment in Pretraining ever larger models.