Deep learning is a branch of Machine learning that uses artificial Neural networks with many stacked layers to learn hierarchical representations directly from raw data, such as pixels, audio waveforms, or text, without requiring hand-engineered features. The word "deep" refers to the number of layers between input and output, with systems typically having a credit assignment path (CAP) depth greater than two. Deep learning can be supervised, semi-supervised, or unsupervised, and draws inspiration from but does not aim to model biological brain function.
History
The mathematical foundations of deep learning date to the 1980s, when Backpropagation was popularized as a way to train multi-layer networks. Progress stalled through the 1990s and 2000s due to limited data, limited compute, and competition from other statistical methods, a period sometimes described as part of a broader AI winter for connectionist approaches specifically. The field's modern era is usually dated to 2012, when AlexNet, a deep Convolutional neural network trained on graphics processing units, won the ImageNet competition by a wide margin over methods based on hand-crafted features. Researchers Geoffrey Hinton, Yann LeCun, and Yoshua Bengio, later called the field's founding trio, received the 2018 Turing Award for laying its groundwork; Hinton and physicist John Hopfield shared the 2024 Nobel Prize in Physics for related foundational work.
The term "deep learning" was introduced to the machine learning community by Rina Dechter in 1986, and to artificial neural networks by Igor Aizenberg and colleagues in 2000 in the context of Boolean threshold neurons.
Architecture
Early deep learning success came from convolutional networks for images and Recurrent neural networks, including the LSTM variant, for sequential data such as text and speech. Common architectures also include fully connected networks, deep belief networks, generative adversarial networks (Generative Adversarial Networks), transformers, and Neural Radiance Fields. The 2017 introduction of the Transformer (architecture) architecture, which processes sequences in parallel using Attention mechanisms rather than step-by-step recurrence, largely displaced recurrent networks for language tasks and became the basis of modern Large language models. Networks are trained by minimizing a loss function through variants of Gradient descent, with backpropagation used to compute how each parameter should change. Deep learning models can be constructed with a greedy layer-by-layer method, and can be applied to unsupervised learning tasks.
Why it works at scale
Deep networks benefit disproportionately from more data and more compute compared with earlier statistical methods, a relationship later formalized as scaling laws. This scaling behavior, combined with the parallel processing capability of GPU (in AI) hardware originally built for graphics rendering, allowed practitioners to train ever-larger models through the 2010s and 2020s. Nvidia's dominance in AI-capable GPUs and its CUDA software platform became a central piece of industry infrastructure as a result.
Interpretations
Deep neural networks are generally interpreted in terms of the universal approximation theorem or probabilistic inference. The classic universal approximation theorem states that feedforward networks with a single hidden layer of finite size can approximate continuous functions; the first proof was published by George Cybenko in 1989 for sigmoid activations and generalized to multi-layer architectures by Kurt Hornik in 1991. Recent work extended universal approximation to unbounded activations such as the rectified linear unit. The probabilistic interpretation, which considers activation nonlinearities as cumulative distribution functions, led to the introduction of dropout as a regularizer.
Impact and criticism
Deep learning drove rapid progress in Computer vision, Speech recognition, Machine translation, Natural language processing, and, most visibly to the public, text and image generation. The field has also been applied to bioinformatics, drug design, medical image analysis, climate science, material inspection, and board game programs, where results have matched or surpassed human expert performance in some cases. Critics note that deep networks remain largely opaque, function as statistical pattern matchers whose failures can be difficult to predict, and require enormous energy and data resources, raising both environmental and copyright concerns. Proponents counter that the same scaling recipe has repeatedly produced capabilities that were not explicitly designed for, a pattern central to ongoing debates about the field's trajectory toward more general intelligence.
Recent developments
In the 2020s, Large language models based on the transformer architecture, such as GPT and Transformers (Hugging Face) libraries, became mainstream products. Deep learning also expanded into generative models like GANs and diffusion models for image synthesis, and neural radiance fields for 3D scene reconstruction. Advances in Representation learning and Unsupervised learning continue to reduce the dependence on labeled data. The field remains a focus of intense research and public debate, with ongoing efforts to improve interpretability, efficiency, and alignment with human values.