Naman Goyal is a computer scientist and researcher in artificial intelligence. He is best known for co-authoring BART, a denoising autoencoder for pretraining sequence-to-sequence models, and for his contributions to Fairseq, a widely used open-source deep learning library. His work has influenced the development of modern Large language models.
Goyal's research is situated within the field of Machine learning, particularly in areas related to Sequence-to-Sequence (Seq2Seq) learning and pretraining. His contributions bridge foundational architectures and practical tooling for training large models.
BART and Pretraining
Goyal was part of the team that released BART in 2019. The BART architecture combines a bidirectional encoder and an autoregressive decoder, using a denoising objective to pretrain on corrupted text. This approach allowed the model to generate high-quality output for tasks such as summarization and translation. The paper introduced the model with up to 400 million parameters, and it achieved state-of-the-art results on several benchmarks, including Stanford Question Answering Dataset and GLUE.
BART was a significant step in pretraining because it unified and extended ideas from earlier models like BERT and GPT. Rather than using only a standard language model objective, BART corrupted text with various noising functions, including token masking and order permutation. Goyal's role contributed to the empirical success of the approach, which demonstrated that sequence-to-sequence pretraining could be as effective as pure encoder or decoder models.
Fairseq Toolkit
Simultaneously, Goyal contributed to Fairseq, a software library developed at Meta AI (formerly Facebook AI Research). Fairseq supports training models for translation, summarization, and other sequence tasks. It uses PyTorch and provides efficient implementations for Transformer (architecture) and other architectures. The library was open-sourced and has been widely adopted in academia and in other labs.
Fairseq introduced techniques like Weight Initialization, Layer Normalization, and Learning Rate Scheduling that made training deep networks more stable. Goyal's involvement in Fairseq helped in the engineering and experiment, leading it to become a standard tool for research.
Contributions to Model Scaling
In related work, Goyal explored issues such as Gradient Clipping and Top-P (Nucleus) Sampling. He also participated in research on scaling and large batch training which affected how practitioners approach pretraining. For example, early 2020 work on model-scaling-laws for neural language models demonstrated that compute, data, and parameters need to be in balance; this line of research influenced the design of large models.
Goyal was also among the researchers who released the Natural Questions data set through the community, although other researchers managed its attribution. However, many of his publications have later become citations in the deep learning literature.
Impact
His work on BART and Fairseq helped establish a baseline for conditional generation by translating. The models also informed the design of text-to-speech and research in more efficient training.
The legacy continues in subsequent releases and untold trains. As of 2024, BART has become a standard model in several libraries. Goyal's contributions have been recognized through citation counts and participation in top conferences. He has also collaborated with researchers such as Yann Lecun and Mikel Artetxe.