PyTorch is an open-source machine learning library originally developed by Facebook's AI Research lab and first released in 2016. It is the successor to the Torch framework and provides a high-level application programming interface built on optimized low-level implementations of deep learning algorithms. As of 2025, PyTorch remains one of the most widely used libraries for artificial intelligence research and production, alongside frameworks such as TensorFlow and Keras. The library is governed by the PyTorch Foundation, a subsidiary of the Linux Foundation, since September 2022.
History
The origins of PyTorch trace back to Torch, a machine learning library written in C and Lua that was released under a GNU General Public License by the Idiap Research Institute in 2001. Around 2010, researchers including Ronan Collobert, Clement Farabet, and Koray Kavukcuoglu rewrote the library as Torch7, separating the C backend from the Lua frontend. In mid-2016, developers at Facebook refactored Torch7 to decouple the frontend and backend further, drawing influence from the autograd library and the Chainer framework. This work led to the creation of PyTorch, which was released as an open-source project in 2016. Development on Torch7 ceased in 2018, and its functionality was subsumed into PyTorch.
In September 2017, Meta and Microsoft launched the Open Neural Network Exchange (ONNX) project to enable interoperability between deep learning frameworks. ONNX allows models to be converted between frameworks and optimized for various execution providers, such as NVIDIA's TensorRT. In March 2018, Caffe2 was merged into PyTorch, consolidating Meta's deep learning efforts. The PyTorch 2.0 release on 15 March 2023 introduced TorchDynamo, a Python-level compiler that accelerates code execution by up to two times, along with significant improvements in training and inference performance across major cloud platforms.
Core features
PyTorch is built around the tensor data structure, which is a homogeneous multidimensional array similar to NumPy arrays. Tensors can be operated on by central processing units (CPUs) or graphics processing units (GPUs), with support for CUDA on NVIDIA hardware, as well as AMD's ROCm and Apple's Metal framework. The library's automatic differentiation system, called Autograd, records operations on tensors to construct a directed acyclic graph (DAG) during the forward pass. When a loss is computed, backpropagation traverses this graph to compute gradients, which are used to update model parameters.
The torch.nn module provides a comprehensive collection of building blocks for neural networks, including various layer types, activation functions, and loss functions. Models are typically defined by subclassing torch.nn.Module and implementing the forward method. This modular design allows for flexible construction of complex architectures, from simple feedforward networks to transformer-based models.
Model serialization
PyTorch models can be saved and loaded using its native serialization format, which is a ZIP64 archive containing the model weights in a Python pickle file, along with metadata such as byte order. The file extensions .pt and .pth are commonly used for these files. This format enables easy sharing and deployment of trained models across different environments.
Example
The following code demonstrates the low-level functionality of PyTorch. It defines a simple neural network with linear layers using the torch.nn module and performs a forward pass.
import torch
import torch.nn as nn
class SimpleNet(nn.Module):
def __init__(self):
super(SimpleNet, self).__init__()
self.fc1 = nn.Linear(10, 5)
self.fc2 = nn.Linear(5, 1)
def forward(self, x):
x = torch.relu(self.fc1(x))
x = self.fc2(x)
return x
model = SimpleNet()
input_tensor = torch.randn(3, 10)
output = model(input_tensor)
print(output)See also
- Comparison of deep learning software
- Comparison of machine learning software
- Differentiable programming
- DeepSpeed
- Open-source artificial intelligence
- PyTorch Lightning
References
- PyTorch documentation. Available at: https://pytorch.org/docs/stable/index.html
- PyTorch GitHub repository. Available at: https://github.com/pytorch/pytorch
- Paszke, A. et al. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32.