A loss function, also called a cost function or objective function, is a mathematical function that measures the difference between a model's predictions and the true target values it is trying to match. Training a Machine learning model is, in almost all cases, the process of adjusting its parameters to make this function's output as small as possible, typically using Gradient descent guided by gradients computed through Backpropagation.
Common forms
For regression tasks, where the target is a continuous number, mean squared error, the average of the squared differences between predictions and targets, is the most widely used loss, penalizing large errors disproportionately more than small ones. For classification tasks, cross-entropy loss, sometimes called log loss, measures the difference between the predicted probability distribution over classes and the true distribution, and is the standard choice for problems from image classification to next-token prediction. Large language models are trained almost exclusively with a cross-entropy loss applied at every token position, effectively asking the model, at each step, to assign as high a probability as possible to the actual next token in the training text. Other losses serve specialized purposes: hinge loss for margin-based classifiers such as support vector machines, contrastive and triplet losses for learning embeddings where similar items should end up close together in Latent space, and adversarial losses, where a second network's ability to distinguish real from generated data becomes the loss for a Generative adversarial network's generator.
Loss versus what we actually want
A recurring theme in machine learning is the gap between the loss function, which is what a model is mathematically optimized to minimize, and the actual downstream goal, which is often harder to specify precisely. A Large language model trained purely to minimize next-token cross-entropy loss will fluently continue text but has no explicit incentive to be honest, helpful, or safe; those properties are added afterward through additional objectives such as RLHF, where a learned reward model provides a further loss signal shaped by human preferences. When a model finds a way to score well on its loss function or a related proxy metric without genuinely achieving the intended goal, the outcome is called Reward hacking, a failure mode that becomes more consequential as models are given more autonomy and better optimization.
Loss and generalization
A loss function is normally evaluated on both a training set, which the model learns from directly, and a held-out validation set, which estimates how well the model will perform on new data. A large and persistent gap between training loss and validation loss is the classic signature of Overfitting, where a model has effectively memorized details of the training data rather than learning patterns that generalize. Techniques grouped under Regularization, such as weight penalties added directly into the loss function, dropout, and early stopping based on validation loss, are the standard tools for keeping this gap under control. Because the loss landscape of a deep network is extremely high-dimensional, the specific shape of the loss function, and not just its final value, materially affects how easy a model is to train, motivating research into loss landscape smoothness and its connection to architectural choices such as normalization layers and residual connections.