Jonathan Frankle is a computer scientist specializing in Machine learning and Deep learning. He is best known for his research on neural network sparsity, particularly the lottery ticket hypothesis, which he co-developed with Michael Carbin at the MIT CSAIL. As of 2024, he serves as chief scientist at MosaicML, a company focused on efficient training and deployment of large language models.
Frankle's work bridges theoretical insights in neural networks with practical systems for generative AI. His contributions have influenced how researchers and engineers approach model compression and resource-efficient AI, a topic of growing importance across the industry.
Early Life and Education
Frankle completed his undergraduate studies in computer science at Carnegie Mellon University, graduating in 2013. He then pursued a PhD at the MIT CSAIL, where he was advised by Michael Carbin. His doctoral research focused on understanding the internal structure of trained neural networks, leading to the formulation of the lottery ticket hypothesis in 2018. He received his PhD in 2021.
Lottery Ticket Hypothesis
The lottery ticket hypothesis, introduced in a 2019 paper co-authored with Carbin, posits that a dense neural network contains a subnetwork - a 'winning ticket' - that, when trained in isolation, can achieve test accuracy comparable to the original network. This subnetwork is identified through an iterative pruning process: train the network, prune the smallest-magnitude weights, reset the remaining weights to their initial values, and retrain. The hypothesis challenged conventional wisdom that pruning primarily serves inference-time efficiency, showing that sparse networks could also be trained from scratch effectively.
The work gained significant attention in the Artificial intelligence community for its implications for both understanding and optimizing neural networks. It suggested that overparameterization in deep learning might be less about capacity and more about the presence of many viable subnetworks. Subsequent research by Frankle and collaborators extended the idea to more complex architectures, including transformers, and explored the role of learning rate schedules and weight initialization in finding these tickets.
Career at MosaicML
After completing his PhD, Frankle joined MosaicML, a startup founded in 2021 by former Intel and NVIDIA engineers, as chief scientist. At MosaicML, he led research on efficient training methods, including the development of the MosaicML Streaming dataset library and the Composer training framework. His team focused on reducing the cost and time required to train large models, making them accessible to smaller organizations.
In 2023, MosaicML was acquired by Databricks, a data analytics company, for approximately $1.3 billion. Frankle continued in his role as chief scientist post-acquisition, overseeing research on model optimization and the release of open-source models such as MPT-7B and MPT-30B. These models were notable for their competitive performance relative to OpenAI's GPT-3 while requiring fewer computational resources.
Research Contributions
Beyond the lottery ticket hypothesis, Frankle has contributed to several areas of machine learning research. His work on 'linear mode connectivity' explored how independently trained models can be interpolated in weight space, providing insights into the loss landscape of deep networks. He also investigated the impact of data ordering and batch normalization on training dynamics.
Frankle has been an advocate for reproducibility in AI research, publishing detailed experimental protocols and open-sourcing code for his studies. His papers have appeared at major conferences including NeurIPS, ICML, and ICLR, and he has served as a reviewer for these venues.
Impact and Recognition
Frankle's research has been cited extensively in the deep learning literature, with the lottery ticket hypothesis paper receiving thousands of citations. His ideas have influenced practical tools for model compression, such as pruning libraries in frameworks like PyTorch and TensorFlow. Industry practitioners at companies like Google DeepMind and Anthropic have referenced his work in discussions of efficient model design.
In the broader context of Generative AI, Frankle's focus on efficiency aligns with efforts to democratize access to large models. His work at MosaicML contributed to the trend of open-weight models, which contrast with the proprietary approaches of major labs. As of 2024, he remains active in the field, speaking at conferences and contributing to public discourse on sustainable AI development.
Selected Publications
- 'The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks' (with Michael Carbin, 2019)
- 'Linear Mode Connectivity and the Lottery Ticket Hypothesis' (with Gundappa Sahu et al., 2020)
- 'Stabilizing the Lottery Ticket Hypothesis' (with Gundappa Sahu et al., 2020)
- 'MosaicML: Efficient Training of Large Language Models' (technical report, 2023)
References
His work is documented in academic papers and technical reports available through arxiv and conference proceedings. Interviews and profiles have appeared in technology media outlets, discussing his contributions to efficient AI.