Sepp Hochreiter is a German computer scientist known for two influential contributions to Deep learning made two decades apart: identifying the vanishing gradient problem in the early 1990s, and, more recently, proposing xLSTM as a competitor to the Transformer (architecture) architecture that displaced his earlier work as the dominant sequence model.
Vanishing gradients and LSTM
In his 1991 diploma thesis, supervised by Jürgen Schmidhuber at the Technical University of Munich, Hochreiter showed mathematically why standard Recurrent neural network training with Backpropagation through time tends to fail on long sequences: error gradients either shrink toward zero or grow uncontrollably as they are propagated backward through many time steps, making it very hard for early Neural network architectures to learn dependencies spanning more than a few steps. This diagnosis, largely overlooked outside German-language AI circles at the time, directly motivated the design of long short-term memory (LSTM), which Hochreiter and Schmidhuber published in 1997. LSTM introduced gated memory cells that could preserve information over long spans, and became the default architecture for tasks such as machine translation, speech recognition and text generation throughout the 2000s and 2010s, before the 2017 Attention Is All You Need paper introduced transformers.
Later career
Hochreiter is a professor at Johannes Kepler University Linz, where he leads the Institute for Machine Learning and heads a unit of the European Laboratory for Learning and Intelligent Systems (ELLIS). His lab has also worked on drug discovery and chemistry applications of Machine learning, using deep networks to predict molecular properties and screen candidate compounds long before such applications became a widely funded subfield in their own right. In 2024 his group introduced xLSTM, an updated version of the original architecture with exponential gating and revised memory structures, positioned as a more computationally efficient alternative to transformers for certain long-sequence tasks, part of a broader wave of interest in transformer alternatives alongside approaches like the State space model family. Hochreiter co-founded the company NXAI to commercialize this line of research, arguing that Europe should invest in home-grown foundation model architectures rather than relying solely on transformer-based Large language model designs from major US labs, framing xLSTM as evidence that older, well-understood architectures still have room for meaningful improvement.