Transfer learning is the practice of reusing a model, or the representations it has learned, from one task as the starting point for a different but related task, rather than training a new model from randomly initialized weights. It rests on the observation that many tasks within a domain, such as recognizing edges and shapes in images or understanding grammar and word relationships in text, share underlying structure, so a model that has already learned that structure needs comparatively little additional data to specialize further.
History and motivation
Transfer learning has roots in cognitive science and early machine learning research from the 1990s exploring "learning to learn," but it became a dominant practical paradigm following the 2012 ImageNet moment, when AlexNet demonstrated that a Convolutional neural network trained on a million labeled images learned visual features, edges, textures, and object parts, that transferred remarkably well to entirely different image classification problems with only a small amount of task-specific data and retraining. This made transfer learning the default strategy in computer vision for years: rather than training from scratch, practitioners would take an ImageNet-pretrained network and adapt it.
In natural language processing, an analogous shift arrived later, catalyzed by work such as Jeremy Howard and Sebastian Ruder's ULMFiT approach and then decisively by BERT in 2018, which showed that a single model pretrained on a generic language-modeling objective could be fine-tuned to state-of-the-art performance across a wide range of NLP tasks. This established "pretrain then transfer" as the standard recipe that continues to underlie modern large language models.
Relationship to pretraining and fine-tuning
Transfer learning is the general principle; Pretraining and Fine-tuning are the specific mechanism by which it is usually carried out in deep learning today. A model is pretrained on a large, general dataset, often using Self-supervised learning so that no manual labels are required, and this pretrained model is then fine-tuned on a smaller labeled dataset for the target task. The success of transfer learning depends on how related the source and target tasks or domains are: transferring from natural images to medical X-rays works less well than transferring between two natural-image tasks, because the low-level statistical structure of the data differs more.
Techniques
Several strategies exist for how much of a pretrained model to adapt. Feature extraction freezes the pretrained network entirely and trains only a new final layer on top of its outputs, useful when the target dataset is very small. Full fine-tuning updates all parameters and typically achieves the best performance when enough target-domain data is available. Parameter-efficient methods such as LoRA sit between these extremes, adapting a small number of additional parameters while leaving most of the pretrained weights untouched, which has become especially important as base models have grown to hundreds of billions of parameters.
Significance
Transfer learning is a major reason deep learning became practical outside of a handful of well-resourced labs: a research team or company without the budget to train a large model from scratch can still build a capable, specialized system by adapting an existing pretrained model. It also underlies the broader shift toward foundation models, where a single large pretrained model serves as the shared substrate for many downstream applications rather than each application requiring its own model trained from the ground up. The technique's central limitation, sometimes called negative transfer, is that adapting a model to a domain too dissimilar from its pretraining data can perform worse than training from scratch, making the choice of a well-matched base model an important practical decision.