Jascha Sohl-Dickstein is a computer scientist and researcher in Machine learning and Artificial intelligence. He is best known as the lead author of the 2015 paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics," which introduced diffusion probabilistic models - a class of Generative AI models that later became central to image and video generation systems. His work bridges statistical physics, optimization, and Deep learning.
Sohl-Dickstein received his PhD from the University of California, Berkeley, where he worked on machine learning and computational neuroscience. He subsequently held positions at Stanford University and the RIKEN Brain Science Institute before joining Google Brain in 2015. At Google Brain, he contributed to research on optimization, generalization, and generative modeling. In 2021, he moved to Google DeepMind as a research scientist, and in 2023 he joined Anthropic, where he has focused on interpretability and safety of large-scale AI systems.
Diffusion Models and Early Generative Work
Sohl-Dickstein's 2015 paper, co-authored with Eric Weiss, Niru Maheswaranathan, and Surya Ganguli, proposed a framework for generative modeling based on gradually adding noise to data and then learning to reverse that process. The approach drew on nonequilibrium thermodynamics and was initially overshadowed by other generative methods such as generative adversarial networks and variational autoencoders. However, it later became the foundation for high-profile systems like DALL-E and Stable Diffusion, which use diffusion models to produce realistic images from text prompts. The paper is now widely cited as a key precursor to modern Generative AI systems.
Research Contributions
Beyond diffusion models, Sohl-Dickstein has published on a range of topics in Machine learning. His work includes studies on the geometry of loss landscapes, the dynamics of Neural network training, and the role of noise in optimization. He has also investigated the connection between deep learning and neuroscience, particularly how biological circuits might implement learning rules similar to those used in artificial networks. His research often emphasizes theoretical understanding alongside empirical results.
At OpenAI - where he was a research scientist from 2017 to 2018 - he worked on large-scale generative models and contributed to early efforts in Large language model research. His time there overlapped with the development of the Transformer architecture, though his primary focus remained on generative modeling and optimization.
Later Career and Interpretability
After returning to Google Brain and then moving to Google DeepMind, Sohl-Dickstein continued to explore the theoretical foundations of deep learning. He co-authored papers on the implicit bias of gradient descent, the effect of batch size on generalization, and the scaling behavior of neural networks. His work on the relationship between model size and performance has informed practical guidelines for training large models.
In 2023, Sohl-Dickstein joined Anthropic, where he has worked on mechanistic interpretability - the effort to understand the internal computations of neural networks. This includes analyzing how transformers represent concepts and how to identify circuits within them. His background in physics and optimization has been valuable in developing methods to reverse-engineer the behavior of complex models.
Impact and Recognition
The 2015 diffusion paper has been recognized as one of the most influential works in modern AI. It laid the groundwork for a family of models that now power widely used tools for image synthesis, audio generation, and scientific applications. Sohl-Dickstein's broader contributions to optimization and learning theory have also been cited extensively. He has spoken at major conferences and workshops, and his work has been featured in both academic venues and popular media.
Sohl-Dickstein's career illustrates the interdisciplinary nature of AI research, combining insights from physics, statistics, and computer science. His trajectory - from academia to industry research labs - mirrors the growth of the field itself, as foundational ideas from the 2010s became the basis for commercial products in the 2020s.
Selected Publications
- Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. International Conference on Machine Learning (ICML).
- Sohl-Dickstein, J., Poole, B., & Ganguli, S. (2014). Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. ICML.
- Sohl-Dickstein, J., et al. (2020). On the relationship between batch size and generalization. (Preprint).
These works have been influential in shaping how researchers think about generative modeling and optimization in Deep learning.