Knowledge distillation is a technique in Machine learning for transferring the learned behavior of a large, computationally expensive model, called the teacher, into a smaller model, called the student, so that the student approximates the teacher's performance at a fraction of the size and cost. Rather than training the student only on hard labels, distillation trains it to match the teacher's output distribution, which carries additional information about how confident the teacher is across all possible answers, not just the single correct one.
History
The core idea traces to work on model compression in the early 2000s, but the modern formulation was introduced in 2015 by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in the paper "Distilling the Knowledge in a Neural Network." They proposed training the student on the teacher's softened output probabilities, produced by raising the "temperature" of the softmax function, arguing that these soft targets expose relative similarities between classes that hard labels discard. The technique remained a niche compression method through the 2010s, applied mostly to Convolutional neural networks for vision tasks, before becoming central to the Large language model era.
Method
In the standard setup, the student is trained with a loss that combines two terms: a conventional Loss function against the ground-truth labels, and a divergence term, typically Kullback-Leibler divergence, between the student's and teacher's softened output distributions. The temperature parameter controls how much the soft targets are smoothed; higher temperatures reveal more of the teacher's uncertainty structure. Variants extend the idea beyond output matching: feature-based distillation aligns intermediate layer activations, and relational distillation matches how the teacher represents relationships between examples. Distillation is distinct from Fine-tuning and LoRA, which adapt an existing model's own weights rather than training a separate, smaller network from a teacher's signal, though the techniques are often combined in practice.
Applications
Distillation became commercially significant with BERT-family compression, most notably DistilBERT (2019), which retained about 97% of BERT's language understanding performance with 40% fewer parameters and roughly 60% faster inference, making it practical for Edge AI and latency-sensitive deployment. The technique gained renewed attention in 2025 after DeepSeek-R1 released a family of smaller open-weight models distilled from its own reasoning traces, showing that distillation could transfer step-by-step reasoning ability, not just classification accuracy, into compact models that ran on modest hardware. Major labs also use distillation internally to produce cheaper, faster variants of flagship models for high-volume Inference (AI), and to compress ensembles into single deployable networks.
Limitations and criticism
Distilled students generally underperform their teachers on the hardest examples and can inherit or amplify the teacher's biases and blind spots, since they are optimized to mimic the teacher rather than to independently model the underlying task. There is ongoing debate about how much of a model's capability can survive distillation versus how much depends on scale itself; some evidence suggests certain Emergent abilities degrade disproportionately in smaller distilled models even when benchmark scores look similar. Distillation has also become commercially contentious: several AI labs have accused competitors of distilling proprietary models by training on their outputs at scale, a practice that violates most providers' terms of service but is difficult to detect or prove, and became a flashpoint in coverage of the DeepSeek market shock of January 2025.